Troubleshooting The 'oracle.sysman.emcp.oms.OmsPlugIn has failed' Error [ID 436605.1]

In this Document

Applies to:

Enterprise Manager Base Platform. - Version 10.1.0.2 to 10.2.0.4 [Release 10.1 to 10.2]
Information in this document applies to any platform.
***Checked for relevance on 13-Mar-2013***

Purpose

The purpose of this document is to fast track the debugging of the "oracle.sysman.emcp.oms.OmsPlugIn has failed" error.

Troubleshooting Steps

Symptoms:

Grid Control is failing at the Agent installation with the following error:
The installActions.log files shows:

INFO: Command = oracle.sysman.emcp.oms.OmsPlugIn -configureOms 

Operation Stopping OPMN Processes is in progress. 
Operation EM Deploying is in progress. 

Operation Unlocking Passwords and Out of Box setup is in progress. 

Command = oracle.sysman.emcp.oms.OmsPlugIn has failed 

INFO: Configuration assistant "OMS Configuration" failed 

SEVERE: OUI-25031:Some of the configuration assistants failed. 
It is strongly recommended that you retry the configuration assistants at this time. 
Not successfully running any "Recommended" assistants means your system will not be correctly configured.

LOG FILES:

The best log files to look at to get to the root cause of this failure are:

/cfgtoollogs/cfgfw/oracle.sysman.top.oms_log
/cfgtoollogs/cfgfw/CfmLogger_.log
/cfgtoollogs/oui/installActions.log

The following are the most common reasons for this failure:

1. System Pre-requisites for installing the OMS are not met. In order to check whether you meet the required system pre-requisites, please refer to the "System Requirements" section in the following documents:

Enterprise Manager Grid Control Quick Installation Guide for Linux x86 
URL: http://download-west.oracle.com/docs/cd/B16240_01/doc/install.102/b28091/toc.htm 
Enterprise Manager Grid Control Quick Installation Guide for Solaris (SPARC) 
URL: http://download-west.oracle.com/docs/cd/B16240_01/doc/install.102/b31198/toc.htm 
Enterprise Manager Grid Control Quick Installation Guide for HP-UX PA-RISC (64-bit) 
URL: http://download-west.oracle.com/docs/cd/B16240_01/doc/install.102/b28745/toc.htm 
Enterprise Manager Grid Control Quick Installation Guide for Windows 
URL: http://download-west.oracle.com/docs/cd/B16240_01/doc/install.102/b31288/toc.htm 
Enterprise Manager Grid Control Quick Installation Guide for AIX 5L Based Systems (64-bit) 
URL: http://download-west.oracle.com/docs/cd/B16240_01/doc/install.102/b28033/toc.htm 
Enterprise Manager Grid Control Quick Installation Guide for Linux x86-64 
URL: http://download-west.oracle.com/docs/cd/B16240_01/doc/install.102/e10054/toc.htm 
Enterprise Manager Grid Control Quick Installation Guide for HP-UX Itanium 
URL: http://download-west.oracle.com/docs/cd/B16240_01/doc/install.102/e10081/toc.htm

2. The OMS connection to the Repository Database fails (Refer Note <>).

Solution: 
A. Keep the OMS Configuration Assistant running 
B. Shutdown the listener 
C. Change the listener port to 1521 
D. Start the listener 
E. Click to retry the OMS Configuration Assistant

3. After experiencing the 'oracle.sysman.emcp.oms.OmsPlugIn has failed' error during the OMS Configuration & pressing retry, you will get this next error (Refer Note <>):

INFO: Starting to execute configuration assistants 
INFO: Command = oracle.sysman.emcp.oms.OmsPlugIn -configureOms 

Command = oracle.sysman.emcp.oms.OmsPlugIn has failed 
Exception : java.lang.Exception: Error processing emkey command config emkey -repos -force from repository = 

INFO: Configuration assistant "OMS Configuration" failed

Solution:

The work around is to Edit the listener.ora file add the parameter below:

SUBSCRIBE_FOR_NODE_DOWN_EVENT_=OFF

where would be replaced with the actual listener name configured in the
listener.ora file.

For example, if the listener name is LISTENER (default), the parameter would be:

SUBSCRIBE_FOR_NODE_DOWN_EVENT_LISTENER=OFF

- Save the listener.ora file
Bounce the listener
Hit the retry button in the OUI
If you hit the retry button and get the second error listed above you will need to edit the
listener.ora file again and this time remove the following lines:

(ADDRESS_LIST =
(ADDRESS = (PROTOCOL = IPC)(KEY = EXTPROC))
)

Then save the file again, bounce the listener, and hit retry again. This time is should complete.

NOTE: If these steps still fail then the problem could be to do with an invalid locale specification
of a language, territory, code set, and other features. In this case then please refer to Note <>

4. A New installation of Grid Control into an existing database running on an Active/Passive High Availability Cluster environment causes the following errors to occur at the OMS Configuration Stage (Refer Note <>):

OMS Configuration Assistant fails
==installActions file shows==
INFO: Starting to execute configuration assistants
INFO: Command = oracle.sysman.emcp.oms.OmsPlugIn -configureOms

Operation EM Deploying is in progress.
Operation Creating OMS Respository is in progress.
Operation Configuring OMS is in progress.

Command = oracle.sysman.emcp.oms.OmsPlugIn has failed
Exception : java.lang.Exception: Error processing emkey command config emkey
-repos -force from repository =
==CfmLogger log shows==
FINE: oracle.sysman.emCfg.logger.CfmLogger: log: oracle.sysman.top.oms:PerformSecureCommand:

runCmd:/home/oracle/gridcontrol/oms10g/bin/emctl config emkey -repos -force have completed with exitCode=1 
FINE: oracle.sysman.emCfg.logger.CfmLogger: log: oracle.sysman.top.oms:PerformSecureCommand:runCmd:Command Output stdout: 
' 
Oracle Enterprise Manager 10g Release 10.2.0.1.0 
Copyright (c) 1996, 2005 Oracle Corporation. All rights reserved. 
Please enter repository password: 
Invalid Password 
' 
FINE: oracle.sysman.emCfg.logger.CfmLogger: log: oracle.sysman.top.oms:PerformSecureCommand:runCmd:Command Output stderr: 
' 
stty: : Invalid argument 
stty: : Invalid argument 
' 
FINE: oracle.sysman.emCfg.logger.CfmLogger: log: oracle.sysman.top.oms:getEmKey:OMS Service command=config emkey -repos -force:returned: 

INFO: oracle.sysman.top.oms:Error processing emkey command config emkey -repos -force from repository = 
ExitCode=1 
Processing stopped! 
INFO: oracle.sysman.top.oms:Internal PlugIn for {Micro Step state:step:2:configuration in CfmAggregateInstance:

oracle.sysman.top.oms:10.2.0.1.0:common:family=CFM:oh=/home/oracle/gridcontrol/oms10g:label=1} failed with an unhandled exception: 
java.lang.Exception: Error processing emkey command config emkey -repos -force from repository = 
at oracle.sysman.emcp.util.EmcpPlug.startProcessing(EmcpPlug.java:397) 
at oracle.sysman.emcp.util.EmcpPlug.invoke(EmcpPlug.java:368) 
at oracle.sysman.emCfg.core.PerformMicroStep.runJavaClass(PerformMicroStep.java:509) 
at oracle.sysman.emCfg.core.PerformMicroStep.executeMicroStep(PerformMicroStep.java:121) 
at oracle.sysman.emCfg.core.ActionPerformer.performMicroStep(ActionPerformer.java:917) 
at oracle.sysman.emCfg.core.ActionPerformer$Performer.run(ActionPerformer.java:1038) 

INFO: oracle.sysman.top.oms:The plug-in OMS Configuration has failed its perform. method

Solution:

The connect string that OMS generates in emoms.properties needs to be corrected to point to the virtual hostname as follows:

A. Backup the OMS_HOME/sysman/config/emoms.properties

B. Edit the file and change the phyisical host in the repository connect string to the virtual hostname on which the listener is actually listening

C. Secure the oms manually. See Note <>

D. Run the agent configuration assistant in stand alone mode AGENT_HOME/bin/agentca -f

E. Check that the EM application is running and that the OMS is secured

OMS_HOME/opmn/bin/opmnctl status

OMS_HOME/bin/emctl secure status oms

F. Check that you can access the Grid Control console and that the Agent can upload (AGENT_HOME/bin/emctl upload)

G. Another reason why this error would occur is the environment variable EM_REPOS_PWD may be incorrectly/unnecessarily set in the environment

during the installation (Refer Note Note 401551.1). If this is the case then implement the following steps: 

G.1. Unset the environment variable EM_REPOS_PWD before installation 

OR 

G.2. Unset the environment variable EM_REPOS_PWD and retry the failed emctl config emkey command: 

/bin/emctl config emkey -repos

5. Ensure that the UNIX user carring out the Grid Control installation has adequate permissions to write to /tmp/em directory. This directory is used to place the em.war file for deployment by the OMS configuration Assistant.
This directory may exist if it was created by a previous application installation carried out by another UNIX user (Refer Note <>). Should this be the case then apply the following workaround:

Solution: 

A. Open another terminal session as root 

B. Change /tmp/em directory permission to 777 
eg - chmod 777 /tmp/em 

C. Go back to Grid Control installation OUI, click on try again, the installation should go ahead 
and eventually complete successfully.

6. When installing Grid Control on Windows do not use remote desktop for the installation. Perform. the installation while directly connected to the server, i.e. on the server console or using VNC (Refer Note <>). As there is a known issues where the runtime memory available to OC4J is not enough for OC4J to function.

7. Installing 10.2 Grid Control in an existing database fails at OMS configuration step with the following error message in the installation log (Refer Note <>):

"Command = oracle.sysman.emcp.oms.OmsPlugIn has failed 
Exception : java.lang.Exception: RepManager Create Repository Error = 12. Please check the log 
file at OMS_HOME\sysman\log\emca_repos_create.log 

INFO: Configuration assistant "OMS Configuration" failed" 

emca_repos_create.log available at OMS_HOME/sysman/log contains the following error message: 

"ERROR at line 1: 
ORA-01119: error in creating database file '/t01/oradata/emrep/mgmt.dbf' 
ORA-27040: skgfrcre: create error, unable to create file SVR4 Error: 2: No such file or directory 
ORA-06512: at line 44"

Solution:

Ensure the user account installing Grid Control has permissions to be able to create datafiles in the location specified. As the software owner, typically "oracle" do the following:

A. Change to directory where the datafile is getting created: 

% cd  

B. Create a file with "touch" to be sure you have write permissions in the directory: 

% touch afile 

If the file creates then remove the file and try the database operation again: 

% rm afile 

If the creation of the file failed you will need to either create the datafile in a file system where you have write permissions or change the permissions on the file system to allow file creation.

8. If you receive the following error:

INFO: Command = oracle.sysman.emcp.oms.OmsPlugIn -configureOmsOperation Stopping OPMN Processes is in progress.Operation

EM Deploying is in progress.Operation Unlocking Passwords and Out of Box setup is in progress.Command = oracle.sysman.emcp.oms.OmsPlugIn has failed 

Then perform. the following workaround (Refer Note <>):
Grid Control must be able to resolve IP address, Fully qualified domainname (fqdn) and simple hostname.

To resolve this issue, please implement this solution:

A. At a command prompt on the Windows server, run the command to get the actual registered hostname, ipaddress and domainame of the server: 


ipconfig /all 

B. Take the output and edit the c:\windows\system32\drivers\etc\hosts as follows: 

   


NOTE: Ensure field delineation is with spaces and not tabs. 


C. Deinstall Grid Control. 

D. Reinstall Grid Control.

9. Ensure there are no IPV6 entries in the hosts file as this is a known issue with Grid Control.

Solution:

Comment out all IPv6 entries in the /etc/hosts file 

============================ 
This step applies to SLES9 only: 

Edit /etc/hosts to remove "localhost" from the IPV6 address list. 

Before: 
# special IPv6 addresses 
::1 localhost ipv6-localhost ipv6-loopback 

After: 
# special IPv6 addresses 
::1 ipv6-localhost ipv6-loopback

10. Ensure the hostname of the machine that Grid Control is being installed on does not include a special character such as "_" etc. This will also cause the OMS Plugin to fail at the OMS Configuration Assistant stage.

Solution:

If so then change the hostname of the machine Grid Control is being 
installed to follow the RFC9523 Naming Convention then reinstall 
Grid Control (Refer <>). 

-> Add the amended hostname to the "hosts" file & ensure it is resolvable 
via DNS if DNS resolution is being used.

NOTE: To run the OMS Configuration Assistant independant of the Grid Control Installer GUI then please run the following command from the command line:

OMS_Home/oui/bin/runConfig.sh ORACLE_HOME= ACTION=Configure MODE=Perform.

References

NOTE:405038.1 - Problem: OMS CA: Grid Control 10.2.0.1.0 UNIX Install Fails With: Failed to mkdir /tmp/em: Do not specify an existing file
NOTE:283091.1 - How to Secure the OMS and Agent Components in a 10g Grid Control Setup?
NOTE:353131.1 - Problem: OMS CA: 10.2.0.1.0 Grid Control Install Fails with "Error processing emkey command"
NOTE:362482.1 - 10.2 OMS Configuration Assistant Fails - Database on Virtual Host
NOTE:364901.1 - Problem: OMSCA: Installation of Grid Control 10.2 fails during OMS configuration assistant
NOTE:368766.1 - Problem: OMSCA: Configuration Assistant ''Oms Configuration'' Failed
NOTE:392452.1 - Installation of EM 10g Grid Control Fails on Windows With 'ejb-prov.jar:Syntax error in source'
NOTE:400403.1 - Problem: OMS CA: Installing EM 10g Grid Control on Windows Platforms Fails While Attempting to Unlock Passwords
NOTE:401551.1 - Problem: OMS CA: 10.2.0.1.0 Grid Control Install Fails with java.lang.Exception: Error processing emkey command