Oracle 11G PSU 31326410 PSU apply failed 處理-拾億

• Oracle 版本：11.2.0.4

• Oracle 部署架構：雙節點的RAC

• 主機型別：HP unix

• 補丁資訊：31326410（11G HPUNIX  2020年10月的補丁）

• 處理思路：補丁打了一半報錯，為了節約時間，我們想繼續下去，同時也在問題發生後找到了導致問題的原因
  

第一 打補丁方式

Oracle 補丁包中推薦了好幾種打的方法，此次我們選擇如下方式：

GI: /grid/app/11.2.0.4/grid/OPatch/opatch auto /software/31326410/31305209    -oh /grid/app/11.2.0.4/grid 

ORACLE: /oracle/app/oracle/11.2.0.4/db_1/OPatch/opatch auto /software/31326410/31305209 -oh /oracle/app/oracle/11.2.0.4/db_1

OJVM:/software/31326410/31219953]opatch apply

第二 報錯資訊

在打補丁過程 GI 報錯

jkzcdb1#[/]/grid/app/11.2.0.4/grid/OPatch/opatch auto /oracle/software/11G/patch/31326410/31305209 -oh /grid/app/11.2.0.4/grid
Executing /grid/app/11.2.0.4/grid/perl/bin/perl /grid/app/11.2.0.4/grid/OPatch/crs/patch11203.pl -patchdir /oracle/software/11G/patch/31326410 -patchn 31305209 -oh /grid/app/11.2.0.4/grid -paramfile /grid/app/11.2.0.4/grid/crs/install/crsconfig_params

This is the main log file: /grid/app/11.2.0.4/grid/cfgtoollogs/opatchauto2021-06-04_20-47-47.log

This file will show your detected configuration and all the steps that opatchauto attempted to do on your system:
/grid/app/11.2.0.4/grid/cfgtoollogs/opatchauto2021-06-04_20-47-47.report.log

2021-06-04 20:47:47: Starting Clusterware Patch Setup
Using configuration parameter file: /grid/app/11.2.0.4/grid/crs/install/crsconfig_params

Stopping CRS...
Stopped CRS successfully
patch /oracle/software/11G/patch/31326410/31305209/29938455  apply successful for home  /grid/app/11.2.0.4/grid
patch /oracle/software/11G/patch/31326410/31305209/31103343  apply failed  for home  /grid/app/11.2.0.4/grid

Starting CRS...
Installing Trace File Analyzer
CRS-4123: Oracle High Availability Services has been started.
Oracle Grid Infrastructure stack start initiated but failed to complete at /oracle/software/11G/patch/31326410/31305209/29938455/files/crs/install/crsconfig_lib.pm line 11821.

在打補丁過程 DATABASE 報錯資訊：

root@jkzcdb1:[/]/oracle/app/oracle/11.2.0.4/db_1/OPatch/opatch auto /oracle/software/31326410/31305209 -oh /oracle/app/oracle/11.2.0.4/db_1
Executing /grid/app/11.2.0.4/grid/perl/bin/perl /oracle/app/oracle/11.2.0.4/db_1/OPatch/crs/patch11203.pl -patchdir /oracle/software/31326410 -patchn 31305209 -oh /oracle/app/oracle/11.2.0.4/db_1 -paramfile /grid/app/11.2.0.4/grid/crs/install/crsconfig_params

This is the main log file: /oracle/app/oracle/11.2.0.4/db_1/cfgtoollogs/opatchauto2021-06-04_22-03-39.log

This file will show your detected configuration and all the steps that opatchauto attempted to do on your system:
/oracle/app/oracle/11.2.0.4/db_1/cfgtoollogs/opatchauto2021-06-04_22-03-39.report.log

2021-06-04 22:03:39: Starting Clusterware Patch Setup
Using configuration parameter file: /grid/app/11.2.0.4/grid/crs/install/crsconfig_params

Stopping RAC /oracle/app/oracle/11.2.0.4/db_1 ...
Stopped RAC /oracle/app/oracle/11.2.0.4/db_1 successfully

patch /oracle/software/31326410/31305209/29938455/custom/server/29938455  apply successful for home  /oracle/app/oracle/11.2.0.4/db_1
patch /oracle/software/31326410/31305209/31103343  apply failed  for home  /oracle/app/oracle/11.2.0.4/db_1

Starting RAC /oracle/app/oracle/11.2.0.4/db_1 ...
Started RAC /oracle/app/oracle/11.2.0.4/db_1 successfully

opatch auto succeeded.

第三 處理方法，查詢補丁日誌，手動繼續

查詢補丁的log,如上方的GI 日誌 /grid/app/11.2.0.4/grid/cfgtoollogs/opatchauto2021-06-04_20-47-47.log，

DB 的日誌/oracle/app/oracle/11.2.0.4/db_1/cfgtoollogs/opatchauto2021-06-04_22-03-39.log

例如DB：如下錯誤資訊：

2021-06-04 22:06:08: patch /oracle/software/31326410/31305209/29938455/custom/server/29938455  apply successful for home  /oracle/app/oracle/11.2.0.4/db_1

2021-06-04 22:06:08: Executing command /oracle/app/oracle/11.2.0.4/db_1/OPatch/opatch napply /oracle/software/31326410/31305209/31103343 -local -silent -oh /oracle/app/oracle/11.2.0.4/db_1
-invPtrLoc /oracle/app/oracle/11.2.0.4/db_1/oraInst.loc as oracle
2021-06-04 22:06:08: Running as user oracle: /oracle/app/oracle/11.2.0.4/db_1/OPatch/opatch napply /oracle/software/31326410/31305209/31103343 -local -silent -oh /oracle/app/oracle/11.2.0.4
/db_1 -invPtrLoc /oracle/app/oracle/11.2.0.4/db_1/oraInst.loc
2021-06-04 22:06:08: s_run_as_user2: Running /bin/su oracle -c ' /oracle/app/oracle/11.2.0.4/db_1/OPatch/opatch napply /oracle/software/31326410/31305209/31103343 -local -silent -oh /oracle
/app/oracle/11.2.0.4/db_1 -invPtrLoc /oracle/app/oracle/11.2.0.4/db_1/oraInst.loc '
2021-06-04 22:10:18: Removing file /var/tmp/vaa027086
2021-06-04 22:10:18: Successfully removed file: /var/tmp/vaa027086
2021-06-04 22:10:18: /bin/su exited with rc=73

第四 解決過程

同時我們要注意在PSU打補丁步驟中，需要叢集unlock和patch,我們需要注意

打補丁前：root 使用者 /grid/app/11.2.0.4/grid/perl/bin/perl /grid/app/11.2.0.4/grid/crs/install/rootcrs.pl -unlock

grid或者Oracle 使用者打相關的補丁

打補丁後：root /grid/app/11.2.0.4/grid/perl/bin/perl /grid/app/11.2.0.4/grid/crs/install/rootcrs.pl -patch

GI 的詳細如下：

root@jkzcdb1:[/]/grid/app/11.2.0.4/grid/perl/bin/perl /grid/app/11.2.0.4/grid/crs/install/rootcrs.pl -unlock
Using configuration parameter file: /grid/app/11.2.0.4/grid/crs/install/crsconfig_params
CRS-2791: Starting shutdown of Oracle High Availability Services-managed resources on 'jkzcdb1'
CRS-2673: Attempting to stop 'ora.mdnsd' on 'jkzcdb1'
CRS-2675: Stop of 'ora.mdnsd' on 'jkzcdb1' failed
CRS-2679: Attempting to clean 'ora.mdnsd' on 'jkzcdb1'
CRS-2681: Clean of 'ora.mdnsd' on 'jkzcdb1' succeeded
CRS-2793: Shutdown of Oracle High Availability Services-managed resources on 'jkzcdb1' has completed
CRS-4133: Oracle High Availability Services has been stopped.

jkzcdb1#[/oracle/software]su - grid
(c)Copyright 1983-2006 Hewlett-Packard Development Company, L.P.
(c)Copyright 1979, 1980, 1983, 1985-1993 The Regents of the Univ. of California
(c)Copyright 1980, 1984, 1986 Novell, Inc.
(c)Copyright 1986-2000 Sun Microsystems, Inc.
(c)Copyright 1985, 1986, 1988 Massachusetts Institute of Technology
(c)Copyright 1989-1993  The Open Software Foundation, Inc.
(c)Copyright 1990 Motorola, Inc.
(c)Copyright 1990, 1991, 1992 Cornell University
(c)Copyright 1989-1991 The University of Maryland
(c)Copyright 1988 Carnegie Mellon University
(c)Copyright 1991-2006 Mentat Inc.
(c)Copyright 1996 Morning Star Technologies, Inc.
(c)Copyright 1996 Progressive Systems, Inc.

Confidential computer software. Valid license from HP required for
possession, use or copying.  Consistent with FAR 12.211 and 12.212,
Commercial Computer Software, Computer Software Documentation, and
Technical Data for Commercial Items are licensed to the U.S. Government
under vendor's standard commercial license.
grid@jkzcdb1:[/home/grid]
grid@jkzcdb1:[/home/grid]410/31305209/31103343 -local -silent -oh /grid/app/11.2.0.4/grid -invPtrLoc /grid/app/11.2.0.4/grid/oraInst.loc                      <
Oracle Interim Patch Installer version 11.2.0.3.28
Copyright (c) 2021, Oracle Corporation.  All rights reserved.


Oracle Home       : /grid/app/11.2.0.4/grid
Central Inventory : /grid/app/oraInventory
   from           : /grid/app/11.2.0.4/grid/oraInst.loc
OPatch version    : 11.2.0.3.28
OUI version       : 11.2.0.4.0
Log file location : /grid/app/11.2.0.4/grid/cfgtoollogs/opatch/opatch2021-06-04_21-41-11PM_1.log

Verifying environment and performing prerequisite checks...
OPatch continues with these patches:   17478514  18031668  18522509  19121551  19769489  20299013  20760982  21352635  21948347  22502456  23054359  24006111  24732075  25869727  26609445  26392168  26925576  27338049  27734982  28204707  28729262  29141056  29497421  29913194  30298532  30670774  31103343  

Do you want to proceed? [y|n]
Y (auto-answered by -silent)
User Responded with: Y
All checks passed.

Please shutdown Oracle instances running out of this ORACLE_HOME on the local system.
(Oracle Home = '/grid/app/11.2.0.4/grid')


Is the local system ready for patching? [y|n]
Y (auto-answered by -silent)
User Responded with: Y
Backing up files...
Applying sub-patch '17478514' to OH '/grid/app/11.2.0.4/grid'
ApplySession: Optional component(s) [ oracle.sdo, 11.2.0.4.0 ] , [ oracle.sysman.agent, 10.2.0.4.5 ] , [ oracle.xdk, 11.2.0.4.0 ]  not present in the Oracle Home or a higher version is found.

Patching component oracle.rdbms, 11.2.0.4.0...

Patching component oracle

...........................................省略
Patching component oracle.rdbms, 11.2.0.4.0...

Patching component oracle.rdbms.dbscripts, 11.2.0.4.0...

Patching component oracle.rdbms.rsf, 11.2.0.4.0...
chmod: invalid mode
Composite patch 31103343 successfully applied.
Log file location: /grid/app/11.2.0.4/grid/cfgtoollogs/opatch/opatch2021-06-04_21-41-11PM_1.log

OPatch succeeded.
grid@jkzcdb1:[/home/grid]

root@jkzcdb1:[/]/grid/app/11.2.0.4/grid/crs/install/rootcrs.pl -patch
Using configuration parameter file: /grid/app/11.2.0.4/grid/crs/install/crsconfig_params
Installing Trace File Analyzer
CRS-4123: Oracle High Availability Services has been started.

第五 最終結果

grid@jkzcdb1:[/home/grid]opatch lspatches
31103343;Database Patch Set Update : 11.2.0.4.200714 (31103343)
29938455;OCW Patch Set Update : 11.2.0.4.191015 (29938455)
OPatch succeeded.

oracle@jkzcdb1:[/oracle/software/31326410/31219953]opatch lspatches
31219953;OJVM PATCH SET UPDATE 11.2.0.4.200714
31103343;Database Patch Set Update : 11.2.0.4.200714 (31103343)
29938455;OCW Patch Set Update : 11.2.0.4.191015 (29938455)

OPatch succeeded.

第六 補丁第一次failed 原因

root 使用者下的LD_LIBARAY_PATH 環境變數問題導致！