Checklist for EM 10g GC 10.2.x to 10.2.0.5 OMS and Repository Upgrades_464674.1

rongshiyuan發表於2014-08-06

Checklist for EM 10g Grid Control 10.2.x to 10.2.0.5 OMS and Repository Upgrades (Doc ID 464674.1)

In this Document

Purpose

Scope

Details

Preparing for the 10.2.0.5 Upgrade:

Post Upgrade 10.2.0.5:

Post-upgrade checks for the Agent:

Known Issues during 10.2.0.5 Upgrade:

References

Applies to:

Enterprise Manager Base Platform - Version 10.2.0.2 to 10.2.0.5 [Release 10.2]
Information in this document applies to any platform.

Purpose

This article is a quick checklist to follow when upgrading Enterprise Manager Grid Control 10.2.0.x to Enterprise Manager Grid Control 10.2.0.5 and to report any Known Issues during Oracle Grid Control 10.2.0.5 Upgrade.

Scope

This article is intended for anyone needing to upgrade Enterprise Manager Grid Control from 10.2.0.x to 10.2.0.5 using the 10.2.0.5 Grid Control patchset.

Details

Important

Oracle recommends that you back up the ORACLE_HOME that must be upgraded using this Patch Set. In the case of a Management Repository, Oracle recommends that you back up the repository database because the Patch Set makes changes to the repository that cannot be rolled back. Also, back up the Oracle Central Inventory directory.

Preparing for the 10.2.0.5 Upgrade:

1. Make sure that you meet the product pre-requisites (operating system patches, packages etc.) for the 10.2.0.5 patch set. For information about platform-specific package and kernel requirements,
refer to the Enterprise Manager Grid Control Installation Guide available here http://download.oracle.com/docs/cd/B16240_01/doc/nav/portal_booklist.htm

2. Before upgrading your Oracle Management Service, ensure that the Shared Pool size is set in your Grid Control repository database as per your infrastructure. Minimum size for shared pool is 512 MB. For more details, refer to requirements described in the Enterprise Manager Grid Control Installation Guide available at : http://download.oracle.com/docs/cd/B16240_01/doc/nav/portal_booklist.htm

Note that SGA_TARGET will need to be at least 50 MB larger than SHARED_POOL_SIZE, so adjust accordingly if this difference is too small after adjusting SHARED_POOL_SIZe.

3. If the database housing the Grid control repository is linked with a Data Guard database, before the upgrade starts, the database has to be forced in logging mode, to force all index maintenance commands to be propagated to the standby database.
The upgrade by default will perform the index maintenance commands in a 'NOLOGGING' mode. This means that any Data Guard database linked to the repository database will not have these index operations replicated to it.
To make sure these commands get replicated to the Data Guard instance, force the database into logging mode:

SQL> CONNECT / AS SYSDBA;
SQL> ALTER DATABASE FORCE LOGGING;

4. If your Enterprise Manager is shutdown for a long period of time, you must follow the below steps before conducting the upgrade:

a.) Log into the Repository Database as SYSMAN

SQL> exec emd_maintenance.analyze_emd_schema('SYSMAN');

b.) Start the Oracle Management Service.

5. Make sure that the Partitioning option is enabled in the Database housing the Grid control repository (SYSMAN schema)

Connect to the Database as sysdba and run the following query to see if the Partitioning Option is installed:

SQL> select value from v$option where parameter = 'Partitioning';

Should give VALUE=TRUE.

Note:
1.No additional partitioning license is required for the Database housing the Grid control repository, as it is a basic Grid Control requirement.
2. Installing Grid Control with Repository in a Database that does not have Partitioning enabled is unsupported. In the earlier versions, running the repository database without partitioning could go un-noticed but starting with 10.2.0.5, the EMGC code expects the partitioning option to be there.
3. If the Partitioning Option is not enabled in the repository Database, trying to upgrade the OMS from a lower version to 10.2.0.5 will fail at the Repository Upgrade Step with these errors in the emrepmgr.log.10.2.0.5 logfile:
*************
SQL> DECLARE
2 MGMT_AUDIT_LOGS VARCHAR2(40) :=MGMT_AUDIT_LOGS';
3 l_sql VARCHAR2(10000) :=CREATE TABLE ' || MGMT_AUDIT_LOGS ||
4 ' (user_session_id_guid RAW(64) DEFAULT NULL,
..
..
ERROR at line 1:
ORA-439: feature not enabled: Partitioning
ORA-6512: at line 63
*************

To avoid the ORA-439 error, ensure that the Partitioning option is enabled in the repository database before the upgrade. Reference:
Note 743857.1: Grid Control Repository: How to Partition the MGMT_METRICS_* Tables if Repository is Created in a Database without Partitioning Option?

6. If the repository database is configured with the UTF8 character set, the value of the NLS_LENGTH_SEMANTICS needs to be set to BYTE.
To check this in the database:

SQL> SELECT value FROM nls_database_parameters WHERE parameter='NLS_LENGTH_SEMANTICS';

If the value is not set to 'BYTE', change it by running the below command:

SQL> ALTER SYSTEM SET NLS_LENGTH_SEMANTICS=BYTE SCOPE=both;

7. Login in the Database as Sysman and remove the following Job:

SQL> SELECT job FROM user_jobs WHERE what='MGMT_CNTR_TT.clean_up_old_ticket_recs;';
SQL> exec dbms_job.remove();
SQL> COMMIT;

The Job "MGMT_CNTR_TT.CLEAN_UP_OLD_TICKET_RECS" is running as SYS user and so if you remove all the jobs from SYSMAN, and put the job_queue parameter to zero, you wont remove this job. You have to manually remove it otherwise it will throw error.

8. Make sure there are no Invalid SYSMAN objects in Grid Control repository database. Perform the following steps before upgrading.

a) Login into the Database as SYS user and

b) Check if there are any invalid SYSMAN objects:

SQL> select object_name, object_type from all_objects where owner='SYSMAN' and status <> 'VALID';

The above query should return 0 rows. If there are rows, then run the below sql statement:

SQL> @admin_recompile_invalid.sql SYSMAN

The admin_recompile_invalid.sql script is available under /sysman/admin/emdrep/sql/core/latest/admin/

Where is the Oracle home directory of the Oracle Management Service (OMS).

c.) Go to step b again to be sure all SYSMAN objects are valid. If you still have invalid SYSMAN objects, please contact Oracle support.

d.) Check there is no invalid SYS object:

SQL> select object_name, object_type from all_objects where status<>'VALID' and object_name like 'DBMS%';

The above query should return 0 rows. If there are rows, then try to "recompile" them. For example:
if the object_type = "PACKAGE" and object_name = "foo", alter package foo compile;
if the object_type = "PACKAGE BODY" and object_name = "foo", alter package foo compile body;

e) If any of the packages fail to become valid even after recompilation, then Contact Oracle Support to assist you. Once this verification is done, continue with the rest of the tasks.

9. If SYSMAN and DBSNMP user doesn't have execute permission to the DBMS_RANDOM package then upgrade will fail during the Repository Upgrade configuration assistant.

Both SYSMAN and the DBSNMP user require access to the DBMS_RANDOM package in the Repository Database. Make sure that execute privileges are granted to this package for both the users

a) Log into the Repository Database as sysdba

b) Run the following SQL:

SQL> grant execute on dbms_random to dbsnmp;
SQL> grant execute on dbms_random to sysman;

c) Verify that below query does not return any invalid objects

SQL> select object_name, object_type, owner from dba_objects where status='INVALID';

10. a) Login as sysman in the Grid control Repository database

b) Run the below sql:

SET SERVEROUTPUT ON SIZE 4000
SET LINE 255
SPOOL run_after_upgrade.sql

begin
for rec in ( select TARGET_TYPE ,PROPERTY_VALUE FROM MGMT_TYPE_PROPERTIES WHERE TARGET_TYPE IN ( 'RHreposURL','SusereposURL','SolarisreposURL','WindowsreposURL','PatchCrossRef' ))

loop
begin

dbms_output.put_line('INSERT INTO MGMT_PATCHING_PROPERTIES(PROPERTY_TYPE,PROPERTY_NAME,PROPERTY_VALUE)values(''os_patching_repos_url'','''||rec.TARGET_TYPE||''','''||rec.PROPERTY_VALUE||''');');

end;
end loop;
dbms_output.put_line('COMMIT;');
end;
/

SPOOL OFF

DELETE MGMT_TYPE_PROPERTIES WHERE TARGET_TYPE IN ('RHreposURL','SusereposURL','SolarisreposURL','WindowsreposURL','PatchCrossRef' );
COMMIT;
/

c) The above step will generate a file 'run_after_upgrade.sql'. For example:

INSERT INTO MGMT_PATCHING_PROPERTIES(PROPERTY_TYPE,PROPERTY_NAME,PROPERTY_VALUE)values('os_patching_repos_url','RHreposURL','ftp://stack34.us.oracle.com/pub/rpm');
INSERT INTO MGMT_PATCHING_PROPERTIES(PROPERTY_TYPE,PROPERTY_NAME,PROPERTY_VALUE)values('os_patching_repos_url','RHreposURL','http://stang05.us.oracle.com/pub/');
COMMIT;

PL/SQL procedure successfully completed.

d) Open the file 'run_after_upgrade.sql' and check for any INSERT statement.

(i) If there are no INSERT statement then you are not required to run this file after the Grid control upgrade is done successfully

(ii) If there are INSERT statements then remove or comments out the line "PL/SQL procedure successfully completed" and then run the sql 'run_after_upgrade.sql'on the Grid control Repository database as sysman user after the Grid control upgrade is done successfully

11. If your 10.1.0.5.0 database containing the Grid Control repository does not have the CPU Jan 2008 patch 6826532 applied, then make sure that you apply this patch. If you have applied the latest CPU Jan 2009 patch 7592360 on your repository database, then you need not apply the CPU Jan 2008 patch 6826532.

Note: CPU Jan 2008 patch 6826532 and CPU Jan 2009 patch 7592360 are not applicable on Windows platform

12. If you have applied the latest CPU Jan 2009 patch 7608318 to your Oracle Management Service (OMS) before upgrading, then you may receive a warning about patch conflict during the product specific prerequisite check. You can ignore this warning and continue with the 10.2.0.5.0 upgrade. The installer will automatically rollback the patch.

13. If you have applied the latest CPU Jan 2009 patch 7608318 to your Oracle Management Service (OMS) before upgrading, then make sure that permission of the following files under the Oracle home directory of OMS /rdbms/admin is set to 664.

(dbmsaqds.plb,dbmsaw.plb,prvtlmd.plb,prvtpexp.plb,prvtlsby.plb,prvtlms.p
lb,prvtaqip.plb,prvtstas.plb,prvtsqlt.plb,prvttxfm.plb,prvtaqdi.plb,prvtaqds.
plb,prvtcdpu.plb,prvtcdpi.plb,prvtmeti.plb,prvtxdbz.plb,exfeapvs.plb,prvtd
upg.plb,prvtupgi.plb,prvtkupc.plb,prvtbpw.plb,prvtaqiu.plb,prvtaqim.plb,pr
vtbsqu.plb,prvtbut4.plb,prvtmetu.plb,prvtmeta.plb,prvtcdcp.plb,prvtcdcu.pl)

If the permission is not set to 664 for the above files, you may get a permission denied error during the upgrade.
For example: Error in writing to file '/u01/app/oracle/product/10.2.0/oms10g/rdbms/admin/dbmsaw.plb' Permission denied

14. Make sure DBMS JOBS and DBMS Scheduler (for version 10.1+ RDBMS) in the Enterprise Manager Grid Control Repository are turned off during the Enterprise Manager schema upgrade process. Log into the repository as SYSMAN and For each instance, perform the tasks below:

a. Write down the values of job_queue_processes for each of the instances after you run the following command:

SQL> select a.instance_name as sid, b.value as jobqueue from gv$instance a, gv$parameter b where
a.inst_id = b.inst_id and b.name='job_queue_processes';

b. Turn off DBMS JOBS by running the following command:

SQL> execute emd_maintenance.remove_em_dbms_jobs;
SQL> commit;
SQL> ALTER SYSTEM SET job_queue_processes=0 SID='*';

Note: This will not kill currently running jobs but allows them to execute till their completion. There onwards, no new jobs can be started.

Make sure there are no active Jobs by running the following :

SQL> SELECT l.id2 job, l.sid, to_char(last_date, 'DD-MON-YYYY:HH24.MI.SS') last_date,
       to_char(this_date, 'DD-MON-YYYY:HH24.MI.SS') this_date, l.inst_id instance
  FROM sys.job$ j, gv$lock l
  WHERE l.type = 'JQ'
    AND j.job (+) = l.id2
  ORDER BY 5, 4;

c. Turn off the DBMS Scheduler by running (for version 10.1+ RDBMS) the following command:

SQL> exec dbms_scheduler.set_scheduler_attribute('SCHEDULER_DISABLED','TRUE');

d. Be sure there were no active schedules running by waiting until the following query returns 0 (for version 10.1+ RDBMS):

SQL> select count(*) from dba_scheduler_running_jobs where SLAVE_PROCESS_ID IS NOT NULL;

15. The 7214697 patchset implements improved provisioning functionality with the 10.2.0.4.0 Grid Control product. These improvements are integrated in the 10.2.0.5.0 version.

a.) For 10.2.0.4.0 Grid Control OMS installed on Unix platforms only, apply patch 7214697 to your Oracle home directory of the Oracle Management Service.

Note: If you have a multiple OMS setup, then apply Patch 7214697 to all the OMS ORACLE_HOMEs. This patch updates the files in the OMS and in repository and hence needs to be applied to all OMSs before proceeding with the Upgrade Step.

b.) You should see preinstall.pl file in the top level directory after extracting the patchset. Run this script before upgrading the OMS using the following command:

$PROMPT> perl preinstall.pl

Here OMS_HOME is the Oracle home directory of the Oracle Management Service. This is a generic script and can be used on all the platforms.

Note: The above step is not required when the OMS is on a Windows platform.

16. During the Upgrade it will ask you for the ias_admin user password. The ias_admin user is the administrative user for Oracle Application Server instances. The password that you specify here is required to access the application server and to deploy the new features. This password is same as the SYSMAN password that you specified when you first installed the Grid Control in your environment.

Note: After installing the Grid Control in your environment, if you change the SYSMAN password, ias_admin password will not be changed.

To change the ias_admin password, go to the Oracle home directory of Oracle Management Service and run the following command:

$ORACLE_HOME/bin/emctl set password

17. If the secure key was exported out of the repository, it has to be put back into the repository during the upgrade. To check if the key is outside of the repository, use this command:

$ emctl status emkey

If the output reports "The Em Key is configured properly", the key is exported out of the repository, and needs to get imported again:

$ emctl config emkey -copy_to_repos

Note: If you don't remember the old ias_admin password, follow the instructions from Document 280116.1 How To Reset or Change the ias_admin User Password for Application Server Console (AS Control)

18. Check for duplicate target associations.

To see if there are duplicate associations, use EMDIAG:

+ Download the EMDiagkit from the Note 421053.1 EMDiagkit Download and Master Index
+ Install it as per instructions in Note 421499.1 How to Install - Deinstall
+ Set the environment as per Note 421586.1 Environment Variables

$ repvfy verify targets -test 6

SQL> CREATE TABLE foo AS SELECT * FROM mgmt_flat_target_assoc;

b.) Clean up the MGMT_FLAT_TARGET_ASSOC table

SQL> TRUNCATE TABLE mgmt_flat_target_assoc;

c.) Re-create all the associations

SQL> exec EM_ASSOC.recreate_full_flat_assoc;
SQL> COMMIT;

d.) If everything has completed OK, drop the temporary table

SQL> DROP TABLE foo;

Post Upgrade 10.2.0.5:

1. After the upgrade is completed, turn on DBMS JOBS and DBMS Scheduler (for version 10.1+ RDBMS) for each instance by performing the following steps:

a. Log into the repository as SYSMAN

b. Turn on DBMS JOBS by setting the job_queue_processes to the values written down earlier for each instance

SQL> ALTER SYSTEM SET job_queue_processes=10 SID='*';

c. Turn on DBMS Scheduler (10.1+ RDBMS) using the following command:

SQL> exec dbms_scheduler.set_scheduler_attribute ('SCHEDULER_DISABLED','FALSE');

2. Submit the EM DBMS jobs:

SQL> EXEC emd_maintenance.submit_em_dbms_jobs;

3. If the database housing the Grid control repository is linked with a Data Guard database, then after the OMS upgrade, revert the Data Guard force logging changes:

SQL> CONNECT / AS SYSDBA;
SQL> ALTER DATA NO FORCE LOGGING;

4. After the upgrade has completed, export the emkey again out of the repository:

$ emctl config emkey -remove_from_repos

Make sure to have a backup of this file, as this file is CRUCIAL for the securing of the repository and new Agents.

5. Optional: After the upgrade has completed, cleanup any duplicate charts defined on a group. To see if there are duplicate charts, use EMDIAG:

The minimal EMDIAG version to use for 10.2.0.5 is 2009.0129. If EMDIAG was already installed and configutred in the environment, use the 'version' command to verify version in use:

$ repvfy version

To see if there are duplicates to get cleaned up:

$ repvfy verify targets -test 130 -details

Once the output has been verified, cleanup the duplicates:

$ repvfy verify targets -test 130 -fix

Post-upgrade checks for the Agent:

Optional step: After upgrading the Agent to 10.2.0.5.0, a backup of the JDK directory will be present in the Oracle home of the Agent. This contains the previous version of the JDK installed in the Agent Oracle home before upgrade

Example:
$ ls -ld jdk*
drwxr-xr-x 6 oraha dba 4096 Jan 28 10:15 jdk
drwxr-x--- 7 oraha dba 4096 Jan 26 12:31 jdk2009-01-28_10-14-39-AM

When the Agent has been verified and validated, this JDK backup directory can be removed from the file system.

Known Issues during 10.2.0.5 Upgrade:

Note 400056.1 OC4J CA: "OC4J Instance Configuration Assistant" failed during Grid installation With Error "Attempt to obtain port of type: ajp failed for an OC4J process"

Note 788178.1 Repository Upgrade Assistant Fails during 10.2.0.5 Upgrade if Execute on DBMS_RANDOM Was Revoked from Public

Note 788749.1 OC4J CA: 10.2.0.5 Upgrade on Windows 2003 Server Fails due to corrupted ssl.conf

After successful Upgrade to 10.2.0.5, Grid Console home page gives error 404 Page Not Found
Workaround: Set the parameter LargeRepository=false in OMS_ORACLE_HOME/sysman/config/emoms.properties file and restart the OMS

As part of the OMS Upgrade to 10.2.0.5 the AS Middle Tier component of OMS also gets upgraded to 10.1.2.3.

References

NOTE:10.2 -
NOTE:400056.1 - Problem: OC4J CA: "OC4J Instance Configuration Assistant" failed during Grid installation With Error "Attempt to obtain port of type: ajp failed for an OC4J process"
NOTE:421053.1 - EMDIAG Troubleshooting Kits Master Index
NOTE:421499.1 - EMDIAG REPVFY Kit for Grid Control 10g and 11g and DB Control 10g and 11g - Download, Install/De-Install and Upgrade
NOTE:421586.1 - EMDIAG REPVFY Kit - Environment Variables
NOTE:558673.1 - Patch 5478872 Breaks Grid Control 10.2.0.3 to 10.2.0.4 Upgrade
NOTE:559658.1 - Grid Control Upgrade from 10.2.0.x to 10.2.0.4 Might Fail with "ORA-20206: Target does not exist :host"
NOTE:559722.1 - Patch 6606233 Breaks Grid Control 10.2.0.3 to 10.2.0.4 upgrade
NOTE:743857.1 - Grid Control Repository: How to Partition the MGMT_METRICS_* Tables if Repository is Created in a Database without Partitioning Option?
NOTE:763351.1 - Document References for Grid Control 10.2.0.5.0 Installation and Upgrade
NOTE:788178.1 - Problem: Repository Upgrade Assistant Fails during 10.2.0.5 Upgrade if Execute on DBMS_RANDOM Was Revoked from Public
NOTE:788749.1 - Problem: 10.2.0.5.0 Upgrade on Windows 2003 Server Fails for ssl.conf
NOTE:809624.1 - Applying Patch 7214697 Fails With Error Message "Software Library has not been setup"
NOTE:815137.1 - Em Grid Control Upgrade To 10.2.0.5.0 On Solaris 5.10 Fails In Webcache Make
NOTE:853691.1 - Upgrade Grid Control 10.2.0.x to 10.2.0.5 - Important Steps Required
NOTE:280116.1 - How To Reset or Change the ias_admin User Password for Application Server Console (AS Control)