Troubleshooting Database Control Startup Issues
使用em時經常遇到問題，最近就遇到了port被佔用的問題，發現了一篇有關em的好文章，記錄一下 。[@more@]
Information in this document applies to any platform.
ORACLE_HOME/sysman
-> Files that are unique to each instance of the database are stored in following directory of the Oracle Database 10g home:
ORACLE_HOME/hostname_sid/sysman/ (for a single instance database)
ORACLE_HOME/nodename_sid/sysman/ (for a cluster database)
-> Log files for diagnosing issues are located under:
ORACLE_HOME/hostname_sid/sysman/log/
ORACLE_HOME/nodename_sid/sysman/log/
Check if the Database Control service is running by executing:
ORACLE_HOME/bin/emctl status dbconsole
the output should be:
Oracle Enterprise Manager 10g Database Control Release <5 digit version>
Copyright (c) 1996, 2005 Oracle Corporation. All rights reserved.
qualified domain name>:5500/em/console/aboutApplication
Oracle Enterprise Manager 10g is running.
------------------------------------------------------------------
Logs are generated in directory /d01/oracle/product/10.2.0/db_1/hostname_sid/sysman/log
If the Database Control service is not running start it with:
ORACLE_HOME/bin/emctl start dbconsole
To stop the Database Control service run:
ORACLE_HOME/bin/emctl stop dbconsole
A complete list of available emctl commands can be obtained by just typing:
ORACLE_HOME/bin/emctl
- ORACLE_HOME/hostname_sid/sysman/log/emagent.log and emagent.trc - main agent log & trace files
- ORACLE_HOME/hostname_sid/sysman/log/emoms.log and emoms.trc - main management service log & trace files
- ORACLE_HOME/hostname_SID/sysman/log/emdctl.log and emdctl.trc - emctl log & trace files
- ORACLE_HOME/hostname_SID/sysman/log/emagentfetchlet.trc and emagentfetchlet.log - log and trace file for Java Fetchlets
Additional information related to log files can be found under , .
1. Check if environment variables are set correctly. Mainly, check, ORACLE_HOME, PATH, LD_LIBRARY_PATH, LANG.
2. Check if the SYSMAN/DBSNMP account is open. To check, connect to database as SYS and run:
emagent.trc errors
2008-01-19 11:20:21,231 [HttpRequestHandler-28730188] ERROR conn.ConnectionService verifyRepositoryEx.433 - Invalid Connection Pool. ERROR = ORA-28000: the account is locked
3. Check the timezone set in the environment. If the timezone does not match one of the values in ORACLE_HOME/sysman/admin/supportedtzs.lst the dbcontrol agent will not start. When checking dbcontrol status returns: "EM Deamon is not running".emdb.nohup errors----- Wed Jul 25 22:31:53 2007::property 'agentTZregion' in '/usr/pkg/oracle/product/10.2.0/db//sysman/config/emd.properties' contains an invalid value of 'TZ set to '.Agent start up can not proceed.This value might have been manually modified to be an incorrect value.This value needs to be set to one of the values listed in '/usr/pkg/oracle/product/10.2.0/db/sysman/admin/supportedtzs.lst'. Execute 'emctl config agent getTZ' and see if this is an appropriate value. ----- 4. Check the OS network configuration:
ORA-2005-07-04 12:23:08,120 [XMLLoader0] ERROR conn.ConnectionService verifyRepositoryEx.418 - Invalid Connection Pool. ERROR = Listener refused the connection with the following error: ORA-12514, TNS:listener does not currently know of service requested in connect descriptor The Connection descriptor used by the client was: (DESCRIPTION=(ADDRESS_LIST=(ADDRESS=(PROTOCOL=TCP)(HOST=)(PORT=)))(CONNECT_DATA=(SERVIC_NAME=)))
6. Check the configuration files are correct.
If none of the above helps check the Known Issues list below.
1. EM Deamon is not running. Database Control starts successfully, however checking the status shows "EM Deamon is not running". Also checking the agent status shows that agent is not running.
//sysman/log/emdctl.trc shows:
2005-11-06 18:04:40 Thread-3840 ERROR main: nmectl.c: nmectl_validateTZRegion, agentTZoffset=120, and testTZoffset for GMT:0 do not matchSolution:
a) Set the desired time zone at the OS level:
Windows: Control Panel->Date&Time->Time Zone
Linux/Unix: export TZ=
the timezone select must correspond to one of the timzones in ORACLE_HOME/sysman/admin/supportedtzs.lst
b) Stop the dbconsole
ORACLE_HOME/bin/emctl stop dbconsole
c) Run:
ORACLE_HOMEbinemctl config agent getTZ
This may return a diffrent timezone than the one set in step 1.
ORACLE_HOMEbinemctl config agent updateTZ
This will update the ORACLE_HOMEsysmanconfigemd.properties file with the correct timezone.
d) Start dbconsole
ORACLE_HOME/bin/emctl start dbconsole
2. GIM-00104: Health check failed to connect to instance.
ORACLE_HOME//sysman/log/emagent.trc shows:
2006-05-04 13:17:29 Thread-2206875655 ERROR fetchlets.healthCheck: GIM-00104:Health check failed to connect to instance.
GIM-00090: OS-dependent operation:open failed with status: 24
GIM-00091: OS failure message: Too many open files
GIM-00092: OS failure occurred at: sskgmsmr_7
2006-05-04 13:17:29 Thread-2206875655 ERROR engine: [oracle_database,tmprod_tmprod2,health_check] : nmeegd_GetMetricData failed : Instance HealthCheck initialization failed due to one of the following causes: the owner of the EM agent process is not same as the owner of the Oracle instance processes; the owner of the EM agent process is not part of the dba group; or the database version is not 10g (10.1.0.2) and above.
2006-05-04 13:17:30 Thread-2206892039 ERROR http: snmehl_connect: failed to create socket: Too many open files (error = 24)
Solution:Check
3. Error starting ORMI-Server. Unable to bind socket: Address already in use. Trying to start db control fails without an obvious reason.
ORACLE_HOME//sysman/log/emdb.nohup shows:
----- Mon Nov 6 10:34:13 2006::Console Launched with PID 3441 at time Mon Nov 6 10:34:13 2006
06/11/06 10:34:16 Error starting ORMI-Server. Unable to bind socket: Address already in use
Solution:
Check , , ,
4. Unable to determine local host from URL.
emctl start dbconsole shows:
EMD_URL= qualified domain name>:/emd/main
Solution:
Check A HREF="/metalink/plsql/showdoc?db=NOT&id=266027.1&blackframe=1" >Note 266027.1,
5. OC4J Configuration Issue: $ORACLE_HOME/oc4j/j2ee/OC4J_DBConsole_host_sid not found
emctl start dbconsole fails with the following error:
OC4J Configuration Issue: $ORACLE_HOME/oc4j/j2ee/OC4J_DBConsole_host_sid not found
There are three cases for this issue:
a) ORACLE_HOME variable needed to run emctl is set to the wrong database home. Set the right value and retry the process.
b) Network changes. If the hostname where Database Control was created in the first place is not resolvable anymore startup will fail with above error.
Example:
Database configuration folders:
ORACLE_HOME/oc4j/j2ee/OC4J_DBConsole_myhost_sid
ORACLE_HOME/myhost_sid
network changes made hostname "myhost" to be unresolvable. Instead of "myhost" the hostname resolves to "myhost.mydomain.com" emctl will not find:
ORACLE_HOME/oc4j/j2ee/OC4J_DBConsole_myhost.mydomain.com_sid
ORACLE_HOME/myhost.mydomain.com_sid
To resolve the issue database control needs to be recreated using the correct hostname.
Check for steps to create db control.
c) Database Control was not configured. Check for steps to create db control.
6. Perl errors. Getting the following errors when starting the dbcontrol:
perl: warning: Setting locale failed.
perl: warning: Please check that your locale settings:
LC_ALL = (unset),
LC__FASTMSG = "true",
LANG = "En_US"
are supported and installed on your system.
perl: warning: Falling back to the standard locale ("C")
SolutionThis is a label view environment problem. In the local environment do the following:
Unset the LANG variable.
Stop and restart the database.
7. Start Dbconsole Shows Errors With Wrong Ps Option on Unix AIX, HP, SOLARIS
> emctl start dbconsole
Oracle Enterprise Manager 10g Database Control Release 10.2.0.1.0
Copyright (c) 1996, 2005 Oracle Corporation. All rights reserved.
ps: unknown output format: -o cmd
ps: illegal option -- -
ps: unknown output format: -o ls
usage: ps [ -aAdeflcjLPy ] [ -o format ] [ -t termlist ]
[ -u userlist ] [ -U userlist ] [ -G grouplist ]
[ -p proclist ] [ -g pgrplist ] [ -s sidlist ]
'format' is one or more of:
user ruser group rgroup uid ruid gid rgid pid ppid pgid sid taskid pri opri pcpu pmem vsz rss osz nice class time etime stime f s c lwp nlwp psr tty addr wchan fname comm args projid project pset
Starting Oracle Enterprise Manager 10g Database Control ..................... started.
SolutionCheck
8. Starting dbcontrol fails. emdb.nohup shows:
-----::Console Launched with PID 12031 at time -----
Exception in thread "main" java.util.zip.ZipException: No such file or directory
at java.util.zip.ZipFile.open(Native Method)
at java.util.zip.ZipFile.(ZipFile.java:112)
at java.util.jar.JarFile.(JarFile.java:127)
SolutionCheck
9. Startup dbcontrol is making too many "__JDBC__" entries in LISTENER log:
26-SEP-2005 12:09:00 * (CONNECT_DATA=(CID=(PROGRAM=)(HOST=__jdbc__)(USER=))(SERVICE_NAME=beta.query))* (ADDRESS=(PROTO =tcp)(HOST=hostname)(PORT=52163)) * establish * beta.query * 0
26-SEP-2005 12:09:00 * (CONNECT_DATA=(CID=(PROGRAM=)(HOST=__jdbc__)(USER=))(SERVICE_NAME=beta.query))* (ADDRESS=(PROTO =tcp)(HOST=hostname)(PORT=52164)) * establish * beta.query * 0
SolutionCheck
10. 'emctl start dbconsole' takes ages to start. Emagent process is actually running, although one is unable to access dbcontrol from the browser. emagent.trc shows:
2007-09-16 10:48:16 Thread-1290 WARN vpxoci: OCI Error -- ErrorCode(6550): ORA-06550: line 1, column 65: PLS-00201: identifier 'DBMS_AQADM' must be declared ORA-06550: line 1, column 65:
PL/SQL: Statement ignored
SQL = "/* OracleOEM */ BEGN :succ_sub := 0; dbms_aqadm.creat"...
LOGIN = dbsnmp/@(DESCRIPTION=(ADDRESS=(PROTOCOL=TCP)(HOST=)(PORT=1521))(CONNECT_DATA=(SID=FSSYS)))
SolutionCheck
11. dbconsole having hostname starting with letter "u" fails. This is because of DBCONSOLE DOES NOT WORK HAVING A HOSTNAME STARTING WITH "U". Bug is fixed in 10.2.0.2 patchset.
Apply patchset 10.2.0.2 or if not possible:
a) Save the file $ORACLE_HOMEhost_SIDsysmanconfigemd.properties to emd.properties.orig
b) Update the file $ORACLE_HOMEhost_SIDsysmanconfigemd.properties, replacing with / in the
following line:
For example change:
omsRecvDir=d:oracleproduct10.2.0db_1ukp001_db0sysmanrecv
to
omsRecvDir=d:/oracle/product/10.2.0/db_1/ukp001_db0l/sysman/recv
c) Bounce DB Control
12. SEVERE: Cannot start Database Control. The following ports are already in use: [EM agent port:3938]
a) Check which ports are available for use for this dbcontrol
b) Run the creation of the dbcontrol with using this port numbers:
ORACLE_HOMEbinemca -config all db -repos recreate -AGENT_PORT -DBCONTROL_HTTP_PORT -RMI_PORT -JMS_PORT
The default port numbers are:
AGENT_PORT: 3938
DBCONTROL_HTTP_PORT: 5500 or 1158
RMI_PORT: 5520
JMS_PORT: 5540
13. The dbconsole cannot be initialized correctly and the logfile $ORACLE_HOME/_/sysman/log/emoms.trc shows the following error:
ORA-12516, TNS:listener could not find available handler with matching protocol stack
The Connection descriptor used by the client was:
< oracle.sysman.eml.mntr.emdRepConnectDescriptor defined in $ORACLE_HOME/_/sysman/config/emoms.properties>
Solution Check
14. Generic Time zone issues.
Check , , , .
Platform specific
AIX5L
1. dbcontrol start fails with:./emctl start dbconsolebin/emctl[336]: unlimited: 0403-009 The specified number is not valid for this command.
This happens because AIX 5L O/S itself as documented in AIX Version 4.3 to 5L Migration Guide:
on Page 195.
When the emctl script try to see the value of ulimit. If it's set to unlimited, it throws this exception and continue starting the DBConsole.
Solutiona) You can simply ignore the warning
b) Set the value of the ulimit to a value either than unlimited.
2. unable to star dbcontrol. Main error in emdb.nohup:
+12139 [ Unable to alloc heap of requested size, perhaps the maxdata value is too small - see
README.HTML for more information. ]
+12140 [ **Out of memory, aborting** ]
+12141 [ ]
+12142 [ ]
+12142 [ *** panic: JVMST017: Cannot allocate memory in initializeMarkAndAllocBits(markbits1) ]
+12143 /u01/app/oracle/CPSS/10.2.0/jdk/bin/java[3]: 1253762 IOT/Abort trap(coredump)
+12144 ----- Thu Jul 19 18:27:17 2007::DBConsole exited at Thu Jul 19 18:27:17 2007 with return
value 134. -----
+12145 ----- Thu Jul 19 18:27:17 2007::DBConsole has exited due to an internal error -----
+12146 ----- Thu Jul 19 18:27:17 2007:: - checking for corefile at
/u01/app/oracle/CPSS/10.2.0/abc.xyz.com_sid/sysman/emd -----
+12147 ----- Thu Jul 19 18:27:17 2007::Restarting DBConsole. -----
+12148 ----- Thu Jul 19 18:27:17 2007::Console Launched with PID 1015900 at time Thu Jul 19
18:27:17 2007 -----
+12149 [ Unable to alloc heap of requested size, perhaps the maxdata value is too small - see
README.HTML for more information. ]
+12150 [ **Out of memory, aborting** ]
SolutionIncrease ulimit resources for the user starting the dbcontrol.
3. dbcontrol fails to start. emagent.trc shows:
2006-06-14 14:06:01 Thread-1872 ERROR engine: [oracle_database,,health_check] : nmeegd_GetMetricData failed :
2006-06-14 14:06:11 Thread-1562 ERROR pingManager: nmepm_pingReposURL: Error in request response. code = 400. text = 2006-06-14 14:06:16 Thread-1634 ERROR fetchlets: Could not load library '/u01/app/oracle/product/10.2.0/db/lib32/libnmcfhc.so' for reason 'rtld: 0712-001 Symbol main was referenced from module /u01/app/oracle/product/10.2.0/db/lib32/libnmcfhc.so(), but a runtime definition of the symbol was not found.
rtld: 0712-001 Symbol nmeusb_StringBuffer_new was referenced from module /u01/app/oracle/product/10.2.0/db/lib32/libnmcfhc.so(), but a runtime definition of the symbol was not found.
SolutionCheck
4. The Refresh time on the database control home page shows two hours behind the standard time for
Europe/Copenhagen on AIX platform. How to change the time to correct value? (this w/a can be implemented for other timezones regions as long as the corresponding AIX timezone is known).
Check
Windows Server 2003
1. Starting the dbcontrol fails with:
The OracleDBConsoleCIMISYU service terminated with service-specific error 1 (0x1)
An error occured while trying to initialize the service.
Solutiona) Apply the latest Patch 6012744 - 10.2.0.3.0 Patch6 for Microsoft Windows (x64).
b) All bugs included in Patch 5846378 are also included in Patch 6012744, since these are cumulative patches.
2. dbcontrol fails to start. emagent.trc shows:
2005-08-26 11:53:56 Thread-544 ERROR pingManager: nmepm_pingReposURL: Cannot connect to
retStatus=-1
2005-08-26 11:53:57 Thread-544 WARN http: snmehl_connect: connect failed to (:5501): No connection could be made because the target machine actively refused it.
Solution This is an installation issue. During the installation there are some files missing.
The files
oc4jj2eeoc4j_applicationsapplicationsememWEB-INFlibuix2.jar
oc4jj2eeoc4j_applicationsapplicationsememWEB-INFlibohw.jar
oc4jj2eeoc4j_applicationsapplicationsememWEB-INFlibshare.jar
are missing. In the most cases you needs to create the directory WEB-INFlib manually.
As a workaround, copy the above 3 files to '...WEB-INFlib' and restarted dbconsole. The files are located in $ORACLE_HOMEjlib.
- How To Drop, Create And Recreate DB Control In A 10g Database
- Problem: Startup: Error starting Database Control, dbconsole - Unable to determine local host from URL
- Problem: Startup OMS: Oms Startup Fails With Integration Class Not Found
- Problem: Startup: DB Control Agent Crashes: Gim-00091 OS failure Message: Too Many Open Files
- Problem: Startup: EM Database Control Has Stopped Working and Unable to Start Again
- How to cycle the DB Control emdb.nohup file in $ORACLE_HOME/host_sid/sysman/log
- Problem: Startup: Cannot Start dbconsole and log Shows 'ORMI-Server address is already being used'
- EMCA or DB Control (DBConsole) Fails with Error starting ORMI-Server
- How to manage DB Console Log and Trace files
em配置之後不能正常啟動提示port被佔用，搜了一下找到下面文章參考之後重建em問題得以解決。
[oracle@node2 ~]$ emctl status dbconsole
TZ set to US/Eastern
Oracle Enterprise Manager 10g Database Control Release 10.2.0.1.0
Copyright (c) 1996, 2005 Oracle Corporation. All rights reserved.
Oracle Enterprise Manager 10g is not running.
------------------------------------------------------------------
Logs are generated in directory /app/oracle/product/10.2.0/db_1/node2_test/sysman/log
--下面配置em時手動指定了em成功啟動。
[oracle@node2 ~]$ emca -config all db -repos recreate -AGENT_PORT 3939 -DBCONTROL_HTTP_PORT 1159 -RMI_PORT 5521 -JMS_PORT 5541
STARTED EMCA at Dec 19, 2011 7:40:19 PM
EM Configuration Assistant, Version 10.2.0.1.0 Production
Copyright (c) 2003, 2005, Oracle. All rights reserved.
Enter the following information:
Database SID: test
Central agent home:
Central agent home: /app/oracle/product/10.2.0/db_1
Database Control is already configured for the database test
You have chosen to configure Database Control for managing the database test
You have chosen the central agent /app/oracle/product/10.2.0/db_1 for monitoring the database test
This will remove the existing configuration and the default settings and perform a fresh configuration
Do you wish to continue? [yes(Y)/no(N)]: yes
Listener port number: 1521
Password for SYS user:
Password for DBSNMP user:
Password for SYSMAN user:
Password for SYSMAN user: Email address for notifications (optional):
Outgoing Mail (SMTP) server for notifications (optional):
-----------------------------------------------------------------
You have specified the following settings
Database ORACLE_HOME ................ /app/oracle/product/10.2.0/db_1
Database hostname ................ node2
Listener port number ................ 1521
Database SID ................ test
Central agent home ................ /app/oracle/product/10.2.0/db_1
Email address for notifications ...............
Outgoing Mail (SMTP) server for notifications ...............
-----------------------------------------------------------------
Do you wish to continue? [yes(Y)/no(N)]: yes
Dec 19, 2011 7:41:05 PM oracle.sysman.emcp.EMConfig perform
INFO: This operation is being logged at /app/oracle/product/10.2.0/db_1/cfgtoollogs/emca/test/emca_2011-12-19_07-40-18-PM.log.
Dec 19, 2011 7:41:08 PM oracle.sysman.emcp.util.DBControlUtil stopOMS
INFO: Stopping Database Control (this may take a while) ...
Dec 19, 2011 7:41:24 PM oracle.sysman.emcp.EMReposConfig dropRepository
INFO: Dropping the EM repository (this may take a while) ...
Dec 19, 2011 7:42:45 PM oracle.sysman.emcp.EMReposConfig invoke
INFO: Repository successfully dropped
Dec 19, 2011 7:42:45 PM oracle.sysman.emcp.EMReposConfig createRepository
INFO: Creating the EM repository (this may take a while) ...
Dec 19, 2011 7:44:55 PM oracle.sysman.emcp.EMReposConfig invoke
INFO: Repository successfully created
Dec 19, 2011 7:45:03 PM oracle.sysman.emcp.util.CentralAgentUtil registerWithCentralAgents
INFO: Registering target(s) with central agent(s) (this may take a while)...
Dec 19, 2011 7:45:16 PM oracle.sysman.emcp.util.DBControlUtil startOMS
INFO: Starting Database Control (this may take a while) ...
Dec 19, 2011 7:46:57 PM oracle.sysman.emcp.EMDBPostConfig performConfiguration
INFO: Database Control started successfully
Dec 19, 2011 7:46:58 PM oracle.sysman.emcp.EMDBPostConfig performConfiguration
INFO: >>>>>>>>>>> The Database Control URL is <<<<<<<<<<<
Enterprise Manager configuration completed successfully
FINISHED EMCA at Dec 19, 2011 7:46:58 PM
[oracle@node2 ~]$ emctl status dbconsole
TZ set to US/Eastern
Oracle Enterprise Manager 10g Database Control Release 10.2.0.1.0
Copyright (c) 1996, 2005 Oracle Corporation. All rights reserved.
Oracle Enterprise Manager 10g is running.
------------------------------------------------------------------
Logs are generated in directory /app/oracle/product/10.2.0/db_1/node2_test/sysman/log
[oracle@node2 ~]$
參考doc如下：
In this Document
Applies to:
Enterprise Manager Grid Control - Version: 10.1.0.2 to 10.2.0.1Information in this document applies to any platform.
Purpose
Provide troubleshooting steps for Database Control startup issues.Last Review Date
January 15, 2008Instructions for the Reader
A Troubleshooting Guide is provided to assist in debugging a specific issue. When possible, diagnostic tools are included in the document to assist in troubleshooting.
Troubleshooting Details
What is Database Control?
The Web-based Database Control serves as the primary tool for managing an Oracle database. From the Oracle Enterprise Manager, one can perform administrative tasks such as creating schema objects (tablespaces, tables, and indexes), managing user security, backing up and recovering the database, and importing and exporting data. One can also view performance and status information about the managed database.Installation
Database Control is by default installed if “Use Database Control for Database Management” option is used on the “Select Database Management Option” page when installing the Oracle Database Server. If this option is not initially selected the Database Control can be created afterwards using emca tool. Steps to create/drop/recreate can be found inAdditional information regarding emca can be found at:
Oracle® Enterprise Manager Advanced Configuration
10g Release 3 (10.2.0.3.0)
Part Number B40002-02
Oracle® Enterprise Manager Advanced Configuration
10g Release 3 (10.2.0.3.0)
Part Number B40002-02
Directory structure
-> Files that are common and shared among all instances of the database are stored in the following directory of the Oracle Database 10g home:ORACLE_HOME/sysman
-> Files that are unique to each instance of the database are stored in following directory of the Oracle Database 10g home:
ORACLE_HOME/hostname_sid/sysman/ (for a single instance database)
ORACLE_HOME/nodename_sid/sysman/ (for a cluster database)
If a hostname_sid directory does not exist in the Oracle Database 10g home directory, then Oracle Enterprise Manager 10g Database Control was never configured for the database instance.
-> Log files for diagnosing issues are located under:
ORACLE_HOME/hostname_sid/sysman/log/
ORACLE_HOME/nodename_sid/sysman/log/
How to use?
At the end of a successful database creation, Database Control is automatically launched in the web browser and the database is started. To use the Database Control any other time the service needs to be running.Check if the Database Control service is running by executing:
ORACLE_HOME/bin/emctl status dbconsole
the output should be:
Oracle Enterprise Manager 10g Database Control Release <5 digit version>
Copyright (c) 1996, 2005 Oracle Corporation. All rights reserved.
qualified domain name>:5500/em/console/aboutApplication
Oracle Enterprise Manager 10g is running.
------------------------------------------------------------------
Logs are generated in directory /d01/oracle/product/10.2.0/db_1/hostname_sid/sysman/log
If the Database Control service is not running start it with:
ORACLE_HOME/bin/emctl start dbconsole
To stop the Database Control service run:
ORACLE_HOME/bin/emctl stop dbconsole
A complete list of available emctl commands can be obtained by just typing:
ORACLE_HOME/bin/emctl
Log files that aid in troubleshooting
- ORACLE_HOME/hostname_sid/sysman/log/emdb.nohup - contains the standard output from the Console main Java process at each start/exit (dbconsole and agent). It is appended with each new output of the Console main Java process.- ORACLE_HOME/hostname_sid/sysman/log/emagent.log and emagent.trc - main agent log & trace files
- ORACLE_HOME/hostname_sid/sysman/log/emoms.log and emoms.trc - main management service log & trace files
- ORACLE_HOME/hostname_SID/sysman/log/emdctl.log and emdctl.trc - emctl log & trace files
- ORACLE_HOME/hostname_SID/sysman/log/emagentfetchlet.trc and emagentfetchlet.log - log and trace file for Java Fetchlets
Additional information related to log files can be found under , .
Steps to troubleshoot Database Control startup issues
Common startup issues:- Environment variables;
- SYSMAN/DBSNMP issues
- Timezone
- Network
- Configuration (wrong port assignment, wrong connection string)
1. Check if environment variables are set correctly. Mainly, check, ORACLE_HOME, PATH, LD_LIBRARY_PATH, LANG.
2. Check if the SYSMAN/DBSNMP account is open. To check, connect to database as SYS and run:
SQL> select username, account_status from dba_users where username in ('SYSMAN','DBSNMP');
and the output:
USERNAME ACCOUNT_STATUS
-------------- ------------------------
DBSNMP OPEN
SYSMAN OPENemagent.trc errors
2008-01-19 11:20:21,231 [HttpRequestHandler-28730188] ERROR conn.ConnectionService verifyRepositoryEx.433 - Invalid Connection Pool. ERROR = ORA-28000: the account is locked
3. Check the timezone set in the environment. If the timezone does not match one of the values in ORACLE_HOME/sysman/admin/supportedtzs.lst the dbcontrol agent will not start. When checking dbcontrol status returns: "EM Deamon is not running".emdb.nohup errors----- Wed Jul 25 22:31:53 2007::property 'agentTZregion' in '/usr/pkg/oracle/product/10.2.0/db/
- static IP (no DHCP generated IP);
- hostname must not contain "_" (underscore character);
- nslookup, ping must resolve the fully qualified name;
- "hosts" file entries pattern:
- lookup and reverse lookup must work;
- IPv6 is not supported;
- check "lsnrctl status" if shows same listener details as ORACLE_HOME/network/admin/listener.ora
- check TNS status with tnsping utility
ORA-2005-07-04 12:23:08,120 [XMLLoader0] ERROR conn.ConnectionService verifyRepositoryEx.418 - Invalid Connection Pool. ERROR = Listener refused the connection with the following error: ORA-12514, TNS:listener does not currently know of service requested in connect descriptor The Connection descriptor used by the client was: (DESCRIPTION=(ADDRESS_LIST=(ADDRESS=(PROTOCOL=TCP)(HOST=
6. Check the configuration files are correct.
- ORACLE_HOME/
/sysman/emd.properties. Check properties:
REPOSITORY_URL= fully qualified domain name>:/em/upload/
EMD_URL= fully qualified domain name>:/emd/main - ORACLE_HOME/
/sysman/emoms.properties. Check properties:
oracle.sysman.eml.mntr.emdRepConnectDescriptor - must have a valid connection string
oracle.sysman.eml.mntr.emdRepPort=
oracle.sysman.eml.mntr.emdRepDBName=
oracle.sysman.emSDK.svlt.ConsoleServerPort=
oracle.sysman.emSDK.svlt.ConsoleServerHost=
oracle.sysman.emSDK.svlt.ConsoleServerHTTPSPort=If none of the above helps check the Known Issues list below.
Known issues
Generic platform1. EM Deamon is not running. Database Control starts successfully, however checking the status shows "EM Deamon is not running". Also checking the agent status shows that agent is not running.
2005-11-06 18:04:40 Thread-3840 ERROR main: nmectl.c: nmectl_validateTZRegion, agentTZoffset=120, and testTZoffset for GMT:0 do not matchSolution:
a) Set the desired time zone at the OS level:
Windows: Control Panel->Date&Time->Time Zone
Linux/Unix: export TZ=
the timezone select must correspond to one of the timzones in ORACLE_HOME/sysman/admin/supportedtzs.lst
b) Stop the dbconsole
ORACLE_HOME/bin/emctl stop dbconsole
c) Run:
ORACLE_HOMEbinemctl config agent getTZ
This may return a diffrent timezone than the one set in step 1.
ORACLE_HOMEbinemctl config agent updateTZ
This will update the ORACLE_HOME
d) Start dbconsole
ORACLE_HOME/bin/emctl start dbconsole
2. GIM-00104: Health check failed to connect to instance.
ORACLE_HOME/
2006-05-04 13:17:29 Thread-2206875655 ERROR fetchlets.healthCheck: GIM-00104:Health check failed to connect to instance.
GIM-00090: OS-dependent operation:open failed with status: 24
GIM-00091: OS failure message: Too many open files
GIM-00092: OS failure occurred at: sskgmsmr_7
2006-05-04 13:17:29 Thread-2206875655 ERROR engine: [oracle_database,tmprod_tmprod2,health_check] : nmeegd_GetMetricData failed : Instance HealthCheck initialization failed due to one of the following causes: the owner of the EM agent process is not same as the owner of the Oracle instance processes; the owner of the EM agent process is not part of the dba group; or the database version is not 10g (10.1.0.2) and above.
2006-05-04 13:17:30 Thread-2206892039 ERROR http: snmehl_connect: failed to create socket: Too many open files (error = 24)
Solution:Check
3. Error starting ORMI-Server. Unable to bind socket: Address already in use. Trying to start db control fails without an obvious reason.
ORACLE_HOME/
----- Mon Nov 6 10:34:13 2006::Console Launched with PID 3441 at time Mon Nov 6 10:34:13 2006
06/11/06 10:34:16 Error starting ORMI-Server. Unable to bind socket: Address already in use
Solution:
Check , , ,
4. Unable to determine local host from URL.
emctl start dbconsole shows:
EMD_URL= qualified domain name>:/emd/main
Solution:
Check A HREF="/metalink/plsql/showdoc?db=NOT&id=266027.1&blackframe=1" >Note 266027.1,
5. OC4J Configuration Issue: $ORACLE_HOME/oc4j/j2ee/OC4J_DBConsole_host_sid not found
emctl start dbconsole fails with the following error:
OC4J Configuration Issue: $ORACLE_HOME/oc4j/j2ee/OC4J_DBConsole_host_sid not found
There are three cases for this issue:
a) ORACLE_HOME variable needed to run emctl is set to the wrong database home. Set the right value and retry the process.
b) Network changes. If the hostname where Database Control was created in the first place is not resolvable anymore startup will fail with above error.
Example:
Database configuration folders:
ORACLE_HOME/oc4j/j2ee/OC4J_DBConsole_myhost_sid
ORACLE_HOME/myhost_sid
network changes made hostname "myhost" to be unresolvable. Instead of "myhost" the hostname resolves to "myhost.mydomain.com" emctl will not find:
ORACLE_HOME/oc4j/j2ee/OC4J_DBConsole_myhost.mydomain.com_sid
ORACLE_HOME/myhost.mydomain.com_sid
To resolve the issue database control needs to be recreated using the correct hostname.
Check for steps to create db control.
c) Database Control was not configured. Check for steps to create db control.
6. Perl errors. Getting the following errors when starting the dbcontrol:
perl: warning: Setting locale failed.
perl: warning: Please check that your locale settings:
LC_ALL = (unset),
LC__FASTMSG = "true",
LANG = "En_US"
are supported and installed on your system.
perl: warning: Falling back to the standard locale ("C")
SolutionThis is a label view environment problem. In the local environment do the following:
Unset the LANG variable.
Stop and restart the database.
7. Start Dbconsole Shows Errors With Wrong Ps Option on Unix AIX, HP, SOLARIS
> emctl start dbconsole
Oracle Enterprise Manager 10g Database Control Release 10.2.0.1.0
Copyright (c) 1996, 2005 Oracle Corporation. All rights reserved.
ps: unknown output format: -o cmd
ps: illegal option -- -
ps: unknown output format: -o ls
usage: ps [ -aAdeflcjLPy ] [ -o format ] [ -t termlist ]
[ -u userlist ] [ -U userlist ] [ -G grouplist ]
[ -p proclist ] [ -g pgrplist ] [ -s sidlist ]
'format' is one or more of:
user ruser group rgroup uid ruid gid rgid pid ppid pgid sid taskid pri opri pcpu pmem vsz rss osz nice class time etime stime f s c lwp nlwp psr tty addr wchan fname comm args projid project pset
Starting Oracle Enterprise Manager 10g Database Control ..................... started.
SolutionCheck
8. Starting dbcontrol fails. emdb.nohup shows:
-----
Exception in thread "main" java.util.zip.ZipException: No such file or directory
at java.util.zip.ZipFile.open(Native Method)
at java.util.zip.ZipFile.
at java.util.jar.JarFile.
SolutionCheck
9. Startup dbcontrol is making too many "__JDBC__" entries in LISTENER log:
26-SEP-2005 12:09:00 * (CONNECT_DATA=(CID=(PROGRAM=)(HOST=__jdbc__)(USER=))(SERVICE_NAME=beta.query))* (ADDRESS=(PROTO =tcp)(HOST=hostname)(PORT=52163)) * establish * beta.query * 0
26-SEP-2005 12:09:00 * (CONNECT_DATA=(CID=(PROGRAM=)(HOST=__jdbc__)(USER=))(SERVICE_NAME=beta.query))* (ADDRESS=(PROTO =tcp)(HOST=hostname)(PORT=52164)) * establish * beta.query * 0
SolutionCheck
10. 'emctl start dbconsole' takes ages to start. Emagent process is actually running, although one is unable to access dbcontrol from the browser. emagent.trc shows:
2007-09-16 10:48:16 Thread-1290 WARN vpxoci: OCI Error -- ErrorCode(6550): ORA-06550: line 1, column 65: PLS-00201: identifier 'DBMS_AQADM' must be declared ORA-06550: line 1, column 65:
PL/SQL: Statement ignored
SQL = "/* OracleOEM */ BEGN :succ_sub := 0; dbms_aqadm.creat"...
LOGIN = dbsnmp/
SolutionCheck
11. dbconsole having hostname starting with letter "u" fails. This is because of
Apply patchset 10.2.0.2 or if not possible:
a) Save the file $ORACLE_HOMEhost_SIDsysmanconfigemd.properties to emd.properties.orig
b) Update the file $ORACLE_HOMEhost_SIDsysmanconfigemd.properties, replacing with / in the
following line:
For example change:
omsRecvDir=d:oracleproduct10.2.0db_1ukp001_db0sysmanrecv
to
omsRecvDir=d:/oracle/product/10.2.0/db_1/ukp001_db0l/sysman/recv
c) Bounce DB Control
12. SEVERE: Cannot start Database Control. The following ports are already in use: [EM agent port:3938]
a) Check which ports are available for use for this dbcontrol
b) Run the creation of the dbcontrol with using this port numbers:
ORACLE_HOMEbinemca -config all db -repos recreate -AGENT_PORT
The default port numbers are:
AGENT_PORT: 3938
DBCONTROL_HTTP_PORT: 5500 or 1158
RMI_PORT: 5520
JMS_PORT: 5540
13. The dbconsole cannot be initialized correctly and the logfile $ORACLE_HOME/
ORA-12516, TNS:listener could not find available handler with matching protocol stack
The Connection descriptor used by the client was:
< oracle.sysman.eml.mntr.emdRepConnectDescriptor defined in $ORACLE_HOME/
Solution
14. Generic Time zone issues.
Check , , , .
Platform specific
AIX5L
1. dbcontrol start fails with:./emctl start dbconsole
This happens because AIX 5L O/S itself as documented in AIX Version 4.3 to 5L Migration Guide:
on Page 195.
When the emctl script try to see the value of ulimit. If it's set to unlimited, it throws this exception and continue starting the DBConsole.
Solutiona) You can simply ignore the warning
b) Set the value of the ulimit to a value either than unlimited.
2. unable to star dbcontrol. Main error in emdb.nohup:
+12139 [ Unable to alloc heap of requested size, perhaps the maxdata value is too small - see
README.HTML for more information. ]
+12140 [ **Out of memory, aborting** ]
+12141 [ ]
+12142 [ ]
+12142 [ *** panic: JVMST017: Cannot allocate memory in initializeMarkAndAllocBits(markbits1) ]
+12143 /u01/app/oracle/CPSS/10.2.0/jdk/bin/java[3]: 1253762 IOT/Abort trap(coredump)
+12144 ----- Thu Jul 19 18:27:17 2007::DBConsole exited at Thu Jul 19 18:27:17 2007 with return
value 134. -----
+12145 ----- Thu Jul 19 18:27:17 2007::DBConsole has exited due to an internal error -----
+12146 ----- Thu Jul 19 18:27:17 2007:: - checking for corefile at
/u01/app/oracle/CPSS/10.2.0/abc.xyz.com_sid/sysman/emd -----
+12147 ----- Thu Jul 19 18:27:17 2007::Restarting DBConsole. -----
+12148 ----- Thu Jul 19 18:27:17 2007::Console Launched with PID 1015900 at time Thu Jul 19
18:27:17 2007 -----
+12149 [ Unable to alloc heap of requested size, perhaps the maxdata value is too small - see
README.HTML for more information. ]
+12150 [ **Out of memory, aborting** ]
SolutionIncrease ulimit resources for the user starting the dbcontrol.
3. dbcontrol fails to start. emagent.trc shows:
2006-06-14 14:06:01 Thread-1872 ERROR engine: [oracle_database,
2006-06-14 14:06:11 Thread-1562 ERROR pingManager: nmepm_pingReposURL: Error in request response. code = 400. text = 2006-06-14 14:06:16 Thread-1634 ERROR fetchlets: Could not load library '/u01/app/oracle/product/10.2.0/db/lib32/libnmcfhc.so' for reason 'rtld: 0712-001 Symbol main was referenced from module /u01/app/oracle/product/10.2.0/db/lib32/libnmcfhc.so(), but a runtime definition of the symbol was not found.
rtld: 0712-001 Symbol nmeusb_StringBuffer_new was referenced from module /u01/app/oracle/product/10.2.0/db/lib32/libnmcfhc.so(), but a runtime definition of the symbol was not found.
SolutionCheck
4. The Refresh time on the database control home page shows two hours behind the standard time for
Europe/Copenhagen on AIX platform. How to change the time to correct value? (this w/a can be implemented for other timezones regions as long as the corresponding AIX timezone is known).
Check
Windows Server 2003
1. Starting the dbcontrol fails with:
The OracleDBConsoleCIMISYU service terminated with service-specific error 1 (0x1)
An error occured while trying to initialize the service.
Solutiona) Apply the latest Patch 6012744 - 10.2.0.3.0 Patch6 for Microsoft Windows (x64).
b) All bugs included in Patch 5846378 are also included in Patch 6012744, since these are cumulative patches.
2. dbcontrol fails to start. emagent.trc shows:
2005-08-26 11:53:56 Thread-544 ERROR pingManager: nmepm_pingReposURL: Cannot connect to
retStatus=-1
2005-08-26 11:53:57 Thread-544 WARN http: snmehl_connect: connect failed to (
Solution
The files
oc4jj2eeoc4j_applicationsapplicationsememWEB-INFlibuix2.jar
oc4jj2eeoc4j_applicationsapplicationsememWEB-INFlibohw.jar
oc4jj2eeoc4j_applicationsapplicationsememWEB-INFlibshare.jar
are missing. In the most cases you needs to create the directory WEB-INFlib manually.
As a workaround, copy the above 3 files to '...WEB-INFlib' and restarted dbconsole. The files are located in $ORACLE_HOMEjlib.
References
- Problem: Startup: Emctl Start Dbconsole Fails with Agent port missing in EMD_URL- How To Drop, Create And Recreate DB Control In A 10g Database
- Problem: Startup: Error starting Database Control, dbconsole - Unable to determine local host from URL
- Problem: Startup OMS: Oms Startup Fails With Integration Class Not Found
- Problem: Startup: DB Control Agent Crashes: Gim-00091 OS failure Message: Too Many Open Files
- Problem: Startup: EM Database Control Has Stopped Working and Unable to Start Again
- How to cycle the DB Control emdb.nohup file in $ORACLE_HOME/host_sid/sysman/log
- Problem: Startup: Cannot Start dbconsole and log Shows 'ORMI-Server address is already being used'
- EMCA or DB Control (DBConsole) Fails with Error starting ORMI-Server
- How to manage DB Console Log and Trace files
來自 “ ITPUB部落格 ” ，連結：http://blog.itpub.net/19602/viewspace-1056884/，如需轉載，請註明出處，否則將追究法律責任。
相關文章
- How to Collect Diagnostics for Database Hanging Issues (Doc ID 452358.1)Database
- use database 切換提示You can turn off this feature to get a quicker startupDatabaseUI
- startup,startup mount,startup nomount之間的區別
- postgreSQL troubleshooting 故障分析SQL
- Systematic Latch Contention Troubleshooting in OracleOracle
- startup nomount
- 個人用_kubernetes_troubleshooting_reference
- 【Spark篇】---Spark故障解決（troubleshooting）Spark
- MySQL Server Startup ScriptMySqlServer
- Linux Troubleshooting 超實用系列 - Disk AnalysisLinux
- 基於GitHub Issues的評論系統--gitmentGithub
- [20200302]startup dba.txt
- 4.1.2 關於Startup部署
- Jetpack之Startup快速掌握Jetpack
- VMware vSphere:Troubleshooting V4.x培訓
- ORA-8103 Troubleshooting, Diagnostic and Solution (文件 ID 8103.1)
- 自動部署基於issues的靜態部落格
- The database owner SID recorded in the master database differs from the database owner SID recorded in database 'DB_NAME'DatabaseAST
- 閱讀nopcommerce startup原始碼原始碼
- Oracle Database Cloud - Database as a Service Quick StartOracleDatabaseCloudUI
- Error querying database. XXXXXXXXXXXXX, No database selected。ErrorDatabase
- Convert a Physical Standby Database into a Snapshot Standby DatabaseDatabase
- WPF Customize control
- Cache-Control
- User Account Control
- WebRTC 通話質量調優：Troubleshooting 小工具Web
- 安裝S_S相關報錯的troubleshooting
- Database TimeoutDatabase
- Database OverallDatabase
- database no shardingDatabase
- .NET Core開發日誌——Startup
- asp.net core 系列之StartupASP.NET
- 理解ASP.NET Core - [01] StartupASP.NET
- Cache-Control for Civilians
- HTML 24 - Form ControlHTMLORM
- Cache-Control指令
- Statistical Process Control in SAP
- Control charts in SAP QM
- Oracle 12.2 使用Database Link優化Standby Database WorkloadOracleDatabase優化