Master Note: Troubleshooting Oracle Background Processes_1509616.1

rongshiyuan發表於2014-02-18
Master Note:
Troubleshooting Oracle Background Processes (Doc ID 1509616.1)


In this Document





Purpose





Troubleshooting
Steps





  Concepts





  Issues
Involving Oracle Background Processes





  ORA-
errors related to Background Processes





  ORA-443
background process did not start





  ORA-444
background process failed while starting





  ORA-445
background process did not start after seconds





  ORA-446
background process started when not expected





  ORA-447
fatal error in background process





  ORA-448
normal completion of background process





  ORA-449
background process unexpectedly terminated with
error





  ORA-450
background process '%s' did not start





  Generic
issues not related to a specific background
process





  CJQ0
/ Jnnn ( Job Queue Coordinator Process / Job Queue
Slave Processes )





  Issues
Specific to CJQ0 / Jnnn





  CKPT
( Checkpoint Process )





  Issues
Specific to CKPT





  DBWn
( Database Writer Process )





  Issues
Specific to DBWn





  EMNC
/ Ennn ( EMON Coordinator Process / EMON Slave Process
)





  Issues
Specific to EMNC / Ennn





  LGWR
( Log Writer Process )





  Issues
Specific to LGWR





  MMON
/ Mnnn ( Manageability Monitor Process / MMON Slave
Process )





  Issues
Specific to MMON / Mnnn





  PMON
( Process Monitor )





  Issues
Specific to PMON





  QMNC
/ QMNn ( AQ Coordinator Process / AQ Server Class
Process )





  Issues
Specific to QMNC / QMNn





  RECO
( Recoverer Process )





  Issues
Specific to RECO





  SMON
( System Monitor Process )





  Issues
Specific to SMON





  VKTM
( Virtual Keeper of Time Process )





  Issues
Specific to VKTM





  Wnnn
( Space Management Slave Process )





  Issues
Specific to Wnnn





  Helpful
Articles on Background Processes





  Further
Diagnostics





References





Applies to:

Oracle Database - Enterprise Edition - Version
9.2.0.1 and later
Information in this document applies to any platform.

Purpose


This document is intended to assist Database Administrators resolve issues
encountered involving Oracle Background Processes. It mainly applies to a single
instance environment and looks at a basic typical database configuration.


Troubleshooting Steps


Concepts


To maximize performance and accommodate many users, a multiprocess Oracle
Database system uses background processes. Background processes consolidate
functions that would otherwise be handled by multiple database programs running
for each user process. Background processes asynchronously perform I/O and
monitor other Oracle Database processes to provide increased parallelism for
better performance and reliability.


Issues Involving Oracle Background Processes


Oracle Database creates background processes automatically when a database
instance starts. An instance can have many background processes, not all of
which always exist in every database configuration.


The following information helps to outline some of the known issues involving
common Oracle background processes, and has been grouped by background process
where possible.


 


ORA- errors related to Background Processes


ORA-443 background process did not start


          See Note 18442.1 - OERR: ORA 443 background process
did not start


ORA-444 background process failed while starting


          See Note 18443.1 - OERR: ORA 444 background process
failed while starting


ORA-445 background process did not start after seconds


          See Note 18444.1 - OERR: ORA 445 background process
did not start after seconds


ORA-446 background process started when not expected


          See Note 18445.1 - OERR: ORA 446 background process started
when not expected


ORA-447 fatal error in background process


          See Note 18446.1 - OERR: ORA 447 fatal error in background
process


ORA-448 normal completion of background process


          See Note 18447.1 - OERR: ORA 448 normal completion of
background process


ORA-449 background process unexpectedly terminated with error


          See Note 18448.1 - OERR: ORA 449 background process
unexpectedly terminated with error


ORA-450 background process '%s' did not start


          See Note 287652.1 - OERR: ORA-450 background process '%s'
did not start


 


Generic issues not related to a specific background
process



Note 1379200.1 - Troubleshooting Guide (TSG) -
ksvcreate: Process(xxxx) creation failed / ORA-00445: background process "xxxx"
did not start after n seconds


Note 1349496.1 - Background Process is Killed with
ORA-00028 and Instance is Terminated


Note 1345364.1 - ORA-00445: Background Process "xxxx"
Did Not Start After 120 Seconds


Note 954245.1 - BACKGROUND PROCESSES START VERY SLOWLY
AND SOMETIMES ORA-445 IS RAISED


Note 730287.1 - Database Crashes With ORA-00449 and
ORA-00601


Note 375517.1 - Background Process Terminated, e.g.
ORA-471, Accompanied By Alert Log Warning "Low open file descriptor limit"[


Note 605280.1 - Database Fails With Message User
Terminating Instance Due To Error 472


Note 779552.1 - ORA-00494 Or ORA-600 [2103] During High
Load After 10.2.0.4


Note 1029808.6 - BACKGROUND PROCESS DIES, NO TRACE FILE
GENERATED


Note 166409.1 - Successfully Connect to Database Even if
Background Processes are Killed


 


CJQ0 / Jnnn ( Job Queue
Coordinator Process / Job Queue Slave Processes )


CJQ0 is automatically started and stopped as needed by Oracle Scheduler.


The JOB_QUEUE_PROCESSES initialization parameter specifies the
maximum number of processes that can be created for the execution of jobs. CJQ0
starts only as many job queue processes as required by the number of jobs to run
and available resources.


Job slave processes are created or awakened by the job coordinator when it is
time for a job to be executed.


Job slaves gather all the metadata required to run the job from the data
dictionary. The slave processes start a database session as the owner of the
job, execute triggers, and then execute the job. After the job is complete, the
slave processes commit and then execute appropriate triggers and close the
session. The slave can repeat this operation in case additional jobs need to be
run.


Issues Specific to CJQ0 / Jnnn



Note 1467179.1 - ORA-00450, ORA-00443, background
process "CJQ0" did not start


Note 1449639.1 - ORA-00450: background process 'CJQ0'
did not start


Note 979217.1 - Messages In Alert Log -
Starting/Stopping Background Process CJQ0


Note 413399.1 - CJQ Process Starts Even When
job_queue_processes=0


Note 854214.1 - CJQ Process Fails With ORA-7445
[kkjqawi+353]


Note 1292151.1 - ORA-7445 [Kggibr] Crashes of CJQ
Process


Note 8285404.8 - Bug 8285404 - CJQ errors with ORA-1000
during window switches


Note 9167430.8 - Bug 9167430 - CJQ job queue process
SQL uses FULL TABLE SCAN of OBJ$


Note 344275.1 - Kkjcre1p: Unable To Spawn Jobq Slave
Process, Error 1089


Note 564957.1 - DBMS_SCHEDULER NOT RUNNING THE JOBS, CJQ
WAITING FOR 'ENQ: JS - WDW OP'


Note 416244.1 - ORA-00610 And/Or "unable to spawn jobq
slave process " And/Or "Process(<>) creation failed" In The Alert Log
And/Or TNS-12518/ TNS-12500 In Listener Log


Note 436509.1 - ORA-610 INTERNAL ERROR CODE, KKJCRE1P:
UNABLE TO SPAWN JOBQ


Note 756389.1 - TNS-12514,TNS-12523,TNS-12528,J000 died
kkjcre1p: unable to spawn jobq slave process


Note 790397.1 - PROCESS J000 And M000 Die


 


CKPT ( Checkpoint Process )


At specific times CKPT starts a checkpoint request by messaging DBWn to begin writing dirty buffers. On completion of
individual checkpoint requests, CKPT updates data file headers and control files
to record most recent checkpoint.


Issues Specific to CKPT



Note 1151601.1 - CKPT: terminating instance due to
error 472 in Linux on System Z


Note 7385253.8 - Bug 7385253 - Slow Truncate / DBWR
uses high CPU / CKPT blocks on RO enqueue


 


DBWn ( Database Writer Process )


The primary responsibility of DBWn is to
write data blocks to disk. DBWn also handles
checkpoints, file open synchronization, and logging of Block Written
records.


In many cases the blocks that DBWn writes
are scattered throughout the disk. Thus, the writes tend to be slower than the
sequential writes performed by LGWR. DBWn
performs multiblock writes when possible to improve efficiency. The number of
blocks written in a multiblock write varies by operating system.


The DB_WRITER_PROCESSES initialization parameter specifies the
number of DBWn processes (DBW0-DBW9 and
DBWa-DBWz). The database selects an appropriate default setting for this
parameter or adjusts a user-specified setting based on the number of CPUs and
processor groups.


Issues Specific to DBWn



Note 243248.1 - Database Startup fails with ORA-443
Background Process "DBW0" did not Start


Note 1065267.6 - ORA-445 Background Process DBW0 Did
Not Start After 120 Seconds


Note 1461704.1 - ORA-00600 [kcbbxsv_2] Errors And DBW0:
Terminating Instance Due To Error 471 (


Note 6005113.8 - Bug 6005113 - OERI[kclrwrite_7] in
DBWn process followed by instance crash in RAC


 


EMNC / Ennn ( EMON
Coordinator Process / EMON Slave Process )


EMNC coordinates event management and notification activity in the database,
including Streams Event Notifications, Continuous Query Notifications, and Fast
Application Notifications.


The database event management and notification load is distributed among the
EMON slave processes. These processes work on the system notifications in
parallel, offering a capability to process a larger volume of notifications, a
faster response time, and a lower shared memory use for staging
notifications.


Issues Specific to EMNC / Ennn



Note 1302050.1 - Shut Down Immediate Ora-00443
Background Process Emno Didn't Start


Note 329863.1 - The Event Monitor Process reports an
ORA-00600 [15239] when using OCI AQ Notification


Note 1076346.1 - EMN process crash with ORA-07445
[nserrbd()+129]


Note 105067.1 - Event Monitor Process: Architecture and
Known Issues


 


LGWR ( Log Writer Process )


Redo log entries are generated in the redo log buffer of the system global
area (SGA). LGWR writes the redo log entries sequentially into a redo log file.
If the database has a multiplexed redo log, then LGWR writes the redo log
entries to a group of redo log files.


Issues Specific to LGWR



Note 470.1 - Error ORA-00470: LGWR process terminated with
error


Note 431246.1 - Pmon terminated instance due to LGWR
termination ORA-470


Note 1089733.1 - ORA-00494: enqueue [CF] held for too
long (more than 900 seconds); LGWR:Terminating Instance Due to Error 2103


Note 9556189.8 - Bug 9556189 - LGWR hangs for long
periods using DNFS - CF waits likely


Note 6193945.8 - Bug 6193945 - High LGWR CPU use and
long 'log file sync' latency in RAC


Note 6074620.8 - Bug 6074620 - LGWR unconditionally
writes to trace file


Note 5896963.8 - Bug 5896963 - High LGWR CPU and longer
"log file sync" with fix for bug 5065930


Note 5118272.8 - Bug 5118272 - Instance crash with
ORA-600 [1433], [60] due to lgwr filling up msg blocks (Messages could be for
LGWR or ARCH)


Note 813473.1 - LGWR Uses Async I/O Inspite Of Setting
_lgwr_async_io=False


Note 1278149.1 - Intermittent Long 'log file sync'
Waits, LGWR Posting Long Write Times, I/O Portion of Wait Minimal


 


MMON / Mnnn (
Manageability Monitor Process / MMON Slave Process )


MMON performs many tasks related to manageability, including taking Automatic
Workload Repository snapshots and performing Automatic Database Diagnostic
Monitor analysis.


Mnnn performs manageability tasks dispatched to them by MMON. Tasks
performed include taking Automatic Workload Repository snapshots and Automatic
Database Diagnostic Monitor analysis.


Issues Specific to MMON / Mnnn



Note 1233079.1 - Ksvcreate: Process(M001) Creation
Failed , Database hang


Note 846217.1 - Getting Ora-00600 [12826] In MMON
Background Slave (m000) Process


Note 359950.1 - Trace Files Reporting ORA-1476 For m000
Background Process


Note 1174758.1 - M000 tracefiles containing:
ket_aba_validate : Error 15649 After 11.1 Upgrade


Note 555124.1 - High Cpu Usage By M000 Process And
Kewrafc Code 5 In Mmon Trace File


Note 13833511.8 - Bug 13833511 - M000 process spins
after a tablespace is renamed and new tablespace is created with original
name


Note 13358531.8 - Bug 13358531 - MMON slave (M000) may
hold "TH" enqueue too long


 


PMON ( Process Monitor )


PMON periodically performs cleanup of all the following:




  • Processes that died abnormally



  • Sessions that were killed



  • Detached transactions that have exceeded their idle
    timeout



  • Detached network connections which have exceeded their
    idle timeout


In addition, PMON monitors, spawns, and stops the following as needed:




  • Dispatcher and shared server processes



  • Job queue processes



  • Pooled server processes for database resident connection
    pooling



  • Restartable background processes


PMON is also responsible for registering information about the instance and
dispatcher processes with the network listener.


Issues Specific to PMON



Note 465770.1 - Instance Terminated By PMON Due To Error
ORA-00443


Note 789919.1 - Instance crash by PMON due to ORA-00484
; Dump shows 'failed with errno=4'


Note 744619.1 - PMON crashing with: ORA-07445: exception
encountered: CORE Dump [kfiofib_detach()+84]


Note 1130713.1 - Pmon Spins While Cleaning Dead
Process


Note 468740.1 - "Pmon Failed To Acquire Latch" Messages
in Alert Log -Database Hung


Note 976714.1 - 'PMON failed to acquire latch, see PMON
dump' in Alert Log - How To Diagnose


Note 1138755.1 - PMON Processes Utilizing Excessive CPU
[100%] in 10gR2


 


QMNC / QMNn ( AQ
Coordinator Process / AQ Server Class Process )


QMNC is responsible for facilitating various background activities required
by AQ and Oracle Streams: time management of messages, management of
nonpersistent queues, cleanup of resources, and so on. QMNC dynamically spawns
Qnnn processes as needed for performing these
tasks.


Note that if the AQ_TM_PROCESSES initialization parameter is set
to 0, this process will not start. The database writes the following message to
the alert log: WARNING: AQ_TM_PROCESSES is set to 0. System might be
adversely affected.


Qnnn acts as a slave process for QMNC and
carry out tasks assigned by QMNC. The number of these processes is dynamically
managed by QMNC based on load.


Issues Specific to QMNC / QMNn



Note 793632.1 - Restarting Dead Queue Monitor Process
upgrade from 9.2 to 10.2


Note 839527.1 - Continuous Warning 'Restarting Dead
Background Process Qmn0' in Alert.log


Note 395137.1 - Repeated : Restarting dead background
process QMNC recorded in the alert.log file


 


RECO ( Recoverer Process )


RECO uses the information in the pending transaction table to finalize the
status of in-doubt transactions. At timed intervals, the local RECO attempts to
connect to remote databases and automatically complete the commit or rollback of
the local portion of any pending distributed transactions. All transactions
automatically resolved by RECO are removed from the pending transaction
table.


Issues Specific to RECO



Note 761852.1 - RECO Process Fails with ORA-476 or
ORA-444


Note 1447285.1 - RECO Process Crashes Instance After
ORA-2068 ORA-7445 [kksfft()+109] ORA-476


 


SMON ( System Monitor Process )


SMON performs many database maintenance tasks, including the following:




  • Creates and manages the temporary tablespace metadata



  • Reclaims space used by orphaned temporary segments



  • Maintains the undo tablespace by onlining, offlining, and shrinking the undo
    segments based on undo space usage statistics



  • Cleans up the data dictionary when it is in a transient and inconsistent
    state



  • Maintains the SCN to time mapping table used to support Oracle Flashback
    features


Issues Specific to SMON



Note 155413.1 - SMON Background Process is Running for
Long Time


Note 761094.1 - SMON: Failed to acquire SEG2, skipping
transaction


Note 1073704.1 - SMON Consumes High CPU With Excessive
Redo Log Generation


Note 821743.1 - SMON Trace File Reports Messages "SMON:
about to recover undo segment X" "SMON: mark undo segment X as available"


Note 7529174.8 - Bug 7529174 - Deadlock / hang between
SMON and foreground process


Note 1076161.6 - Shutdown Normal or Shutdown Immediate
Hangs. SMON disabling TX Recovery


Note 414242.1 - Database Hangs Because SMON Is Taking
100% CPU Doing Transaction Recovery


Note 549000.1 - ORA-600 [6006] ORA-600 [6856] During
Startup Instance, Followed by Termination by SMON


 


VKTM ( Virtual Keeper of Time Process )


VKTM acts as a time publisher for an Oracle instance. VKTM publishes two sets
of time: a wall clock time using a seconds interval and a higher resolution time
(which is not wall clock time) for interval measurements. The VKTM timer service
centralizes time tracking and offloads multiple timer calls from other
clients.


Issues Specific to VKTM



Note 558374.1 - Database Startup Fails With ORA-00443:
Background Process "VKTM" Did Not Start


Note 837105.1 - The Database Server Is Creating A Lot Of
Trace Files From the VKTM Process


Note 1347586.1 - Time Drift Detected. Please Check Vktm
Trace File For More Details


Note 1185093.1 - VKTM Trace Files Generated With
KSTMCHKDRIFT "BACKWARD DRIFT ENDED AT"


 


Wnnn ( Space Management Slave Process
)


Wnnn processes are slave processes dynamically spawned by SMCO to
perform space management tasks in the background. These tasks include
preallocating space into locally managed tablespace and SecureFiles segments
based on space usage growth analysis, and reclaiming space from dropped
segments. At most 10 Wnnn slaves can run on one database instance.
After being started, the slave acts as an autonomous agent. After it finishes
task execution, it automatically picks up another task from the queue. The
process terminates itself after being idle for a long time.


Issues Specific to Wnnn



Note 1492880.1 - Wnnn processes consuming high
CPU


Note 13871316.8 - Bug 13871316 - Wnnn space
background process may be periodically CPU bound


Note 7661251.8 - Bug 7661251 - A process may spin
causing ORA-445 background process "wnnn" did not start .." with fix
for bug 6858062


Note 10237773.8 - Bug 10237773 - ORA-600
[kcbz_check_objd_typ] / ORA-600 [ktecgsc:kcbz_objdchk]


 


Helpful Articles on Background Processes



Note 91062.1 - Database Writer and Buffer Management


Note 305662.1 - Master Note for AQ Queue Monitor Process
(QMON)


Note 743773.1 - SMCO (Space Management Coordinator) For
Autoextend On Datafiles And How To Disable/Enable


Note 268197.1 - New Background Processes In 10g


Note 444149.1 - New Background Processes In 11g


Note 331000.1 - What are the 'PSP0' and 'LNS1'
Background Processes Running In 10g Release 2?


Note 1038976.6 - HOW TO FIND BACKGROUND PROCESS RUNNING
ON NT


Note 564989.1 - How To Truncate a Background Trace File
Without Bouncing the Database


Note 164768.1 - Troubleshooting: High CPU
Utilization


Note 250262.1 - RDA 4 - Health Check / Validation Engine
Guide


 


Further Diagnostics


If you were not able to resolve the issue with the details provided in this
document, please raise a Service Request for further assistance from Oracle
Support.


References

NOTE:1503146.1
- Master Note: Overview of Oracle Background Processes
NOTE:250262.1
- RDA - Health Check / Validation Engine Guide

來自 “ ITPUB部落格 ” ,連結:http://blog.itpub.net/17252115/viewspace-1083326/,如需轉載,請註明出處,否則將追究法律責任。

相關文章