Master Note: Troubleshooting Oracle Background Processes_1509616.1
Troubleshooting Oracle Background Processes (Doc ID 1509616.1)
In this Document
Purpose |
Troubleshooting Steps |
Concepts |
Issues Involving Oracle Background Processes |
ORA- errors related to Background Processes |
ORA-443 background process did not start |
ORA-444 background process failed while starting |
ORA-445 background process did not start after seconds |
ORA-446 background process started when not expected |
ORA-447 fatal error in background process |
ORA-448 normal completion of background process |
ORA-449 background process unexpectedly terminated with error |
ORA-450 background process '%s' did not start |
Generic issues not related to a specific background process |
CJQ0 / Jnnn ( Job Queue Coordinator Process / Job Queue Slave Processes ) |
Issues Specific to CJQ0 / Jnnn |
CKPT ( Checkpoint Process ) |
Issues Specific to CKPT |
DBWn ( Database Writer Process ) |
Issues Specific to DBWn |
EMNC / Ennn ( EMON Coordinator Process / EMON Slave Process ) |
Issues Specific to EMNC / Ennn |
LGWR ( Log Writer Process ) |
Issues Specific to LGWR |
MMON / Mnnn ( Manageability Monitor Process / MMON Slave Process ) |
Issues Specific to MMON / Mnnn |
PMON ( Process Monitor ) |
Issues Specific to PMON |
QMNC / QMNn ( AQ Coordinator Process / AQ Server Class Process ) |
Issues Specific to QMNC / QMNn |
RECO ( Recoverer Process ) |
Issues Specific to RECO |
SMON ( System Monitor Process ) |
Issues Specific to SMON |
VKTM ( Virtual Keeper of Time Process ) |
Issues Specific to VKTM |
Wnnn ( Space Management Slave Process ) |
Issues Specific to Wnnn |
Helpful Articles on Background Processes |
Further Diagnostics |
References |
Applies to:
Oracle Database - Enterprise Edition - Version9.2.0.1 and later
Information in this document applies to any platform.
Purpose
This document is intended to assist Database Administrators resolve issues
encountered involving Oracle Background Processes. It mainly applies to a single
instance environment and looks at a basic typical database configuration.
Troubleshooting Steps
Concepts
To maximize performance and accommodate many users, a multiprocess Oracle
Database system uses background processes. Background processes consolidate
functions that would otherwise be handled by multiple database programs running
for each user process. Background processes asynchronously perform I/O and
monitor other Oracle Database processes to provide increased parallelism for
better performance and reliability.
Issues Involving Oracle Background Processes
Oracle Database creates background processes automatically when a database
instance starts. An instance can have many background processes, not all of
which always exist in every database configuration.
The following information helps to outline some of the known issues involving
common Oracle background processes, and has been grouped by background process
where possible.
ORA- errors related to Background Processes
ORA-443 background process did not start
See Note 18442.1 - OERR: ORA 443 background process
ORA-444 background process failed while starting
See Note 18443.1 - OERR: ORA 444 background process
ORA-445 background process did not start after seconds
See Note 18444.1 - OERR: ORA 445 background process
ORA-446 background process started when not expected
See Note 18445.1 - OERR: ORA 446 background process started
when not expected
ORA-447 fatal error in background process
See Note 18446.1 - OERR: ORA 447 fatal error in background
process
ORA-448 normal completion of background process
See Note 18447.1 - OERR: ORA 448 normal completion of
background process
ORA-449 background process unexpectedly terminated with error
See Note 18448.1 - OERR: ORA 449 background process
ORA-450 background process '%s' did not start
See Note 287652.1 - OERR: ORA-450 background process '%s'
did not start
Generic issues not related to a specific background
process
Note 1379200.1 - Troubleshooting Guide (TSG) -
ksvcreate: Process(xxxx) creation failed / ORA-00445: background process "xxxx"
did not start after n seconds
Note 1349496.1 - Background Process is Killed with
ORA-00028 and Instance is Terminated
Note 1345364.1 - ORA-00445: Background Process "xxxx"
Did Not Start After 120 Seconds
Note 954245.1 - BACKGROUND PROCESSES START VERY SLOWLY
AND SOMETIMES ORA-445 IS RAISED
Note 730287.1 - Database Crashes With ORA-00449 and
ORA-00601
Note 375517.1 - Background Process Terminated, e.g.
ORA-471, Accompanied By Alert Log Warning "Low open file descriptor limit"[
Note 605280.1 - Database Fails With Message User
Terminating Instance Due To Error 472
Note 779552.1 - ORA-00494 Or ORA-600 [2103] During High
Load After 10.2.0.4
Note 1029808.6 - BACKGROUND PROCESS DIES, NO TRACE FILE
GENERATED
Note 166409.1 - Successfully Connect to Database Even if
Background Processes are Killed
CJQ0 / Jnnn ( Job Queue
Coordinator Process / Job Queue Slave Processes )
CJQ0 is automatically started and stopped as needed by Oracle Scheduler.
The JOB_QUEUE_PROCESSES initialization parameter specifies the
maximum number of processes that can be created for the execution of jobs. CJQ0
starts only as many job queue processes as required by the number of jobs to run
and available resources.
Job slave processes are created or awakened by the job coordinator when it is
time for a job to be executed.
Job slaves gather all the metadata required to run the job from the data
dictionary. The slave processes start a database session as the owner of the
job, execute triggers, and then execute the job. After the job is complete, the
slave processes commit and then execute appropriate triggers and close the
session. The slave can repeat this operation in case additional jobs need to be
run.
Issues Specific to CJQ0 / Jnnn
Note 1467179.1 - ORA-00450, ORA-00443, background
process "CJQ0" did not start
Note 1449639.1 - ORA-00450: background process 'CJQ0'
did not start
Note 979217.1 - Messages In Alert Log -
Starting/Stopping Background Process CJQ0
Note 413399.1 - CJQ Process Starts Even When
job_queue_processes=0
Note 854214.1 - CJQ Process Fails With ORA-7445
[kkjqawi+353]
Note 1292151.1 - ORA-7445 [Kggibr] Crashes of CJQ
Process
Note 8285404.8 - Bug 8285404 - CJQ errors with ORA-1000
during window switches
Note 9167430.8 - Bug 9167430 - CJQ job queue process
SQL uses FULL TABLE SCAN of OBJ$
Note 344275.1 - Kkjcre1p: Unable To Spawn Jobq Slave
Process, Error 1089
Note 564957.1 - DBMS_SCHEDULER NOT RUNNING THE JOBS, CJQ
WAITING FOR 'ENQ: JS - WDW OP'
Note 416244.1 - ORA-00610 And/Or "unable to spawn jobq
slave process " And/Or "Process(<>) creation failed" In The Alert Log
And/Or TNS-12518/ TNS-12500 In Listener Log
Note 436509.1 - ORA-610 INTERNAL ERROR CODE, KKJCRE1P:
UNABLE TO SPAWN JOBQ
Note 756389.1 - TNS-12514,TNS-12523,TNS-12528,J000 died
kkjcre1p: unable to spawn jobq slave process
Note 790397.1 - PROCESS J000 And M000 Die
CKPT ( Checkpoint Process )
At specific times CKPT starts a checkpoint request by messaging DBWn to begin writing dirty buffers. On completion of
individual checkpoint requests, CKPT updates data file headers and control files
to record most recent checkpoint.
Issues Specific to CKPT
Note 1151601.1 - CKPT: terminating instance due to
error 472 in Linux on System Z
Note 7385253.8 - Bug 7385253 - Slow Truncate / DBWR
uses high CPU / CKPT blocks on RO enqueue
DBWn ( Database Writer Process )
The primary responsibility of DBWn is to
write data blocks to disk. DBWn also handles
checkpoints, file open synchronization, and logging of Block Written
records.
In many cases the blocks that DBWn writes
are scattered throughout the disk. Thus, the writes tend to be slower than the
sequential writes performed by LGWR. DBWn
performs multiblock writes when possible to improve efficiency. The number of
blocks written in a multiblock write varies by operating system.
The DB_WRITER_PROCESSES initialization parameter specifies the
number of DBWn processes (DBW0-DBW9 and
DBWa-DBWz). The database selects an appropriate default setting for this
parameter or adjusts a user-specified setting based on the number of CPUs and
processor groups.
Issues Specific to DBWn
Note 243248.1 - Database Startup fails with ORA-443
Background Process "DBW0" did not Start
Note 1065267.6 - ORA-445 Background Process DBW0 Did
Not Start After 120 Seconds
Note 1461704.1 - ORA-00600 [kcbbxsv_2] Errors And DBW0:
Terminating Instance Due To Error 471 (
Note 6005113.8 - Bug 6005113 - OERI[kclrwrite_7] in
DBWn process followed by instance crash in RAC
EMNC / Ennn ( EMON
Coordinator Process / EMON Slave Process )
EMNC coordinates event management and notification activity in the database,
including Streams Event Notifications, Continuous Query Notifications, and Fast
Application Notifications.
The database event management and notification load is distributed among the
EMON slave processes. These processes work on the system notifications in
parallel, offering a capability to process a larger volume of notifications, a
faster response time, and a lower shared memory use for staging
notifications.
Issues Specific to EMNC / Ennn
Note 1302050.1 - Shut Down Immediate Ora-00443
Background Process Emno Didn't Start
Note 329863.1 - The Event Monitor Process reports an
ORA-00600 [15239] when using OCI AQ Notification
Note 1076346.1 - EMN process crash with ORA-07445
[nserrbd()+129]
Note 105067.1 - Event Monitor Process: Architecture and
Known Issues
LGWR ( Log Writer Process )
Redo log entries are generated in the redo log buffer of the system global
area (SGA). LGWR writes the redo log entries sequentially into a redo log file.
If the database has a multiplexed redo log, then LGWR writes the redo log
entries to a group of redo log files.
Issues Specific to LGWR
Note 470.1 - Error ORA-00470: LGWR process terminated with
error
Note 431246.1 - Pmon terminated instance due to LGWR
termination ORA-470
Note 1089733.1 - ORA-00494: enqueue [CF] held for too
long (more than 900 seconds); LGWR:Terminating Instance Due to Error 2103
Note 9556189.8 - Bug 9556189 - LGWR hangs for long
periods using DNFS - CF waits likely
Note 6193945.8 - Bug 6193945 - High LGWR CPU use and
long 'log file sync' latency in RAC
Note 6074620.8 - Bug 6074620 - LGWR unconditionally
writes to trace file
Note 5896963.8 - Bug 5896963 - High LGWR CPU and longer
"log file sync" with fix for bug 5065930
Note 5118272.8 - Bug 5118272 - Instance crash with
ORA-600 [1433], [60] due to lgwr filling up msg blocks (Messages could be for
LGWR or ARCH)
Note 813473.1 - LGWR Uses Async I/O Inspite Of Setting
_lgwr_async_io=False
Note 1278149.1 - Intermittent Long 'log file sync'
Waits, LGWR Posting Long Write Times, I/O Portion of Wait Minimal
MMON / Mnnn (
Manageability Monitor Process / MMON Slave Process )
MMON performs many tasks related to manageability, including taking Automatic
Workload Repository snapshots and performing Automatic Database Diagnostic
Monitor analysis.
Mnnn performs manageability tasks dispatched to them by MMON. Tasks
performed include taking Automatic Workload Repository snapshots and Automatic
Database Diagnostic Monitor analysis.
Issues Specific to MMON / Mnnn
Note 1233079.1 - Ksvcreate: Process(M001) Creation
Failed , Database hang
Note 846217.1 - Getting Ora-00600 [12826] In MMON
Background Slave (m000) Process
Note 359950.1 - Trace Files Reporting ORA-1476 For m000
Background Process
Note 1174758.1 - M000 tracefiles containing:
ket_aba_validate : Error 15649 After 11.1 Upgrade
Note 555124.1 - High Cpu Usage By M000 Process And
Kewrafc Code 5 In Mmon Trace File
Note 13833511.8 - Bug 13833511 - M000 process spins
after a tablespace is renamed and new tablespace is created with original
name
Note 13358531.8 - Bug 13358531 - MMON slave (M000) may
hold "TH" enqueue too long
PMON ( Process Monitor )
PMON periodically performs cleanup of all the following:
-
Processes that died abnormally
-
Sessions that were killed
-
Detached transactions that have exceeded their idle
timeout -
Detached network connections which have exceeded their
idle timeout
In addition, PMON monitors, spawns, and stops the following as needed:
-
Dispatcher and shared server processes
-
Job queue processes
-
Pooled server processes for database resident connection
pooling -
Restartable background processes
PMON is also responsible for registering information about the instance and
dispatcher processes with the network listener.
Issues Specific to PMON
Note 465770.1 - Instance Terminated By PMON Due To Error
ORA-00443
Note 789919.1 - Instance crash by PMON due to ORA-00484
; Dump shows 'failed with errno=4'
Note 744619.1 - PMON crashing with: ORA-07445: exception
encountered: CORE Dump [kfiofib_detach()+84]
Note 1130713.1 - Pmon Spins While Cleaning Dead
Process
Note 468740.1 - "Pmon Failed To Acquire Latch" Messages
in Alert Log -Database Hung
Note 976714.1 - 'PMON failed to acquire latch, see PMON
dump' in Alert Log - How To Diagnose
Note 1138755.1 - PMON Processes Utilizing Excessive CPU
[100%] in 10gR2
QMNC / QMNn ( AQ
Coordinator Process / AQ Server Class Process )
QMNC is responsible for facilitating various background activities required
by AQ and Oracle Streams: time management of messages, management of
nonpersistent queues, cleanup of resources, and so on. QMNC dynamically spawns
Qnnn processes as needed for performing these
tasks.
Note that if the AQ_TM_PROCESSES initialization parameter is set
to 0, this process will not start. The database writes the following message to
the alert log: WARNING: AQ_TM_PROCESSES is set to 0. System might be
adversely affected.
Qnnn acts as a slave process for QMNC and
carry out tasks assigned by QMNC. The number of these processes is dynamically
managed by QMNC based on load.
Issues Specific to QMNC / QMNn
Note 793632.1 - Restarting Dead Queue Monitor Process
upgrade from 9.2 to 10.2
Note 839527.1 - Continuous Warning 'Restarting Dead
Background Process Qmn0' in Alert.log
Note 395137.1 - Repeated : Restarting dead background
process QMNC recorded in the alert.log file
RECO ( Recoverer Process )
RECO uses the information in the pending transaction table to finalize the
status of in-doubt transactions. At timed intervals, the local RECO attempts to
connect to remote databases and automatically complete the commit or rollback of
the local portion of any pending distributed transactions. All transactions
automatically resolved by RECO are removed from the pending transaction
table.
Issues Specific to RECO
Note 761852.1 - RECO Process Fails with ORA-476 or
ORA-444
Note 1447285.1 - RECO Process Crashes Instance After
ORA-2068 ORA-7445 [kksfft()+109] ORA-476
SMON ( System Monitor Process )
SMON performs many database maintenance tasks, including the following:
-
Creates and manages the temporary tablespace metadata
-
Reclaims space used by orphaned temporary segments
-
Maintains the undo tablespace by onlining, offlining, and shrinking the undo
segments based on undo space usage statistics -
Cleans up the data dictionary when it is in a transient and inconsistent
state -
Maintains the SCN to time mapping table used to support Oracle Flashback
features
Issues Specific to SMON
Note 155413.1 - SMON Background Process is Running for
Long Time
Note 761094.1 - SMON: Failed to acquire SEG2, skipping
transaction
Note 1073704.1 - SMON Consumes High CPU With Excessive
Redo Log Generation
Note 821743.1 - SMON Trace File Reports Messages "SMON:
about to recover undo segment X" "SMON: mark undo segment X as available"
Note 7529174.8 - Bug 7529174 - Deadlock / hang between
SMON and foreground process
Note 1076161.6 - Shutdown Normal or Shutdown Immediate
Hangs. SMON disabling TX Recovery
Note 414242.1 - Database Hangs Because SMON Is Taking
100% CPU Doing Transaction Recovery
Note 549000.1 - ORA-600 [6006] ORA-600 [6856] During
Startup Instance, Followed by Termination by SMON
VKTM ( Virtual Keeper of Time Process )
VKTM acts as a time publisher for an Oracle instance. VKTM publishes two sets
of time: a wall clock time using a seconds interval and a higher resolution time
(which is not wall clock time) for interval measurements. The VKTM timer service
centralizes time tracking and offloads multiple timer calls from other
clients.
Issues Specific to VKTM
Note 558374.1 - Database Startup Fails With ORA-00443:
Background Process "VKTM" Did Not Start
Note 837105.1 - The Database Server Is Creating A Lot Of
Trace Files From the VKTM Process
Note 1347586.1 - Time Drift Detected. Please Check Vktm
Trace File For More Details
Note 1185093.1 - VKTM Trace Files Generated With
KSTMCHKDRIFT "BACKWARD DRIFT ENDED AT"
Wnnn ( Space Management Slave Process
)
Wnnn processes are slave processes dynamically spawned by SMCO to
perform space management tasks in the background. These tasks include
preallocating space into locally managed tablespace and SecureFiles segments
based on space usage growth analysis, and reclaiming space from dropped
segments. At most 10 Wnnn slaves can run on one database instance.
After being started, the slave acts as an autonomous agent. After it finishes
task execution, it automatically picks up another task from the queue. The
process terminates itself after being idle for a long time.
Issues Specific to Wnnn
Note 1492880.1 - Wnnn processes consuming high
CPU
Note 13871316.8 - Bug 13871316 - Wnnn space
background process may be periodically CPU bound
Note 7661251.8 - Bug 7661251 - A process may spin
causing ORA-445 background process "wnnn" did not start .." with fix
for bug 6858062
Note 10237773.8 - Bug 10237773 - ORA-600
[kcbz_check_objd_typ] / ORA-600 [ktecgsc:kcbz_objdchk]
Helpful Articles on Background Processes
Note 91062.1 - Database Writer and Buffer Management
Note 305662.1 - Master Note for AQ Queue Monitor Process
(QMON)
Note 743773.1 - SMCO (Space Management Coordinator) For
Autoextend On Datafiles And How To Disable/Enable
Note 268197.1 - New Background Processes In 10g
Note 444149.1 - New Background Processes In 11g
Note 331000.1 - What are the 'PSP0' and 'LNS1'
Background Processes Running In 10g Release 2?
Note 1038976.6 - HOW TO FIND BACKGROUND PROCESS RUNNING
ON NT
Note 564989.1 - How To Truncate a Background Trace File
Without Bouncing the Database
Note 164768.1 - Troubleshooting: High CPU
Utilization
Note 250262.1 - RDA 4 - Health Check / Validation Engine
Guide
Further Diagnostics
If you were not able to resolve the issue with the details provided in this
document, please raise a Service Request for further assistance from Oracle
Support.
References
NOTE:1503146.1- Master Note: Overview of Oracle Background Processes
NOTE:250262.1
- RDA - Health Check / Validation Engine Guide
來自 “ ITPUB部落格 ” ,連結:http://blog.itpub.net/17252115/viewspace-1083326/,如需轉載,請註明出處,否則將追究法律責任。
相關文章
- Master Note: Troubleshooting Oracle Tablespace Management (文件 ID 1522807.1)ASTOracle
- Master Note - Troubleshooting DBCA Issues (文件 ID 1269459.1)AST
- Master Note: Troubleshooting Redo Logs and Archiving (文件 ID 1507157.1)AST
- 【Oracle】Master Note for Parallel Execution Wait EventsOracleASTParallelAI
- Master Note for Tablespace IssuesAST
- Master Note - Oracle GoldenGate (Doc ID 1298817.1)ASTOracleGo
- Master Note for RAC Oracle Clusterware and Oracle Grid Infrastructure 1096952.ASTOracleStruct
- Master Note for Oracle Disk Manager (Doc ID 1226653.1)ASTOracle
- Master Note: Overview of Oracle Tablespace Management (Doc ID 1493350.1)ASTViewOracle
- Master Note Oracle GoldenGate Management Pack - Director [ID 1307305.1]ASTOracleGo
- Master Note: Oracle Transaction Management (Local) Overview (Doc ID 1506115.1)ASTOracleView
- Oracle RAC Background processesOracle
- zt_oracle troubleshooting案例Oracle
- * Master Note: Database Performance Overview (Doc ID 402983.1)ASTDatabaseORMView
- Master Note For ASM For OS Administrators [ID 1345562.1]ASTASM
- The Oracle Background Processes for Sybase ProsOracleROS
- Master Note for Handling Oracle Database Corruption Issues [ID 1088018.1]ASTOracleDatabase
- Systematic Latch Contention Troubleshooting in OracleOracle
- Troubleshooting Oracle ClusterwareThis appendix introducesOracleAPP
- Master Note of Linux OS Requirements for Database Server [ID 851598.1]ASTLinuxUIREMDatabaseServer
- Master Note: Locks, Enqueues and Deadlocks [ID 1392319.1]ASTENQ
- oracle 817 ora_snp background processOracle
- Master Note: Overview of Database ControlFiles (文件 ID 1493674.1)ASTViewDatabase
- Master Note of Linux OS Requirements for Database Server (Doc ID 851598.1)ASTLinuxUIREMDatabaseServer
- Master Note for Streams Recommended Configuration [ID 418755.1]AST
- Master Note:Online Redefinition of Tables (DBMS_REDEFINITION)_1357825.1AST
- Master Note for Transportable Tablespaces (TTS) -- Common Questions_1166564.1ASTTTS
- Master Note Parallel Execution Wait Events [ID 1097154.1]ASTParallelAI
- Master Note: Overview of Redo Logs and Archiving_1503091.1ASTView
- Master Note for Automatic Storage Management (ASM) [ID 1187723.1]ASTASM
- oracle 11.2 所有Background Processes 的功能描述Oracle
- Master Note For PL/SQL UTL_FILE Package (Doc ID 1155024.1)ASTSQLPackage
- Oracle診斷工具 - ORA-4030 Troubleshooting ToolOracle
- MySQL Master/Slave Master/MasterMySqlAST
- How to find Master Node in Oracle RAC(zt)ASTOracle
- New Background Processes Oracle 10g/11gOracle 10g
- background-clip 和 background-origin
- oracle troubleshooting waits for locks/Enqueues other than 'TM','TX' and 'UL'OracleAIENQ