ORACLE opatch 打補丁fuser command output for /u01/.../crsctl.bin is Failure

    最近，本地測試資料庫oracle 11.2.0.4 for rhel7打補丁的時候失敗了，檢查opatch應用補丁的日誌顯示錯誤提示如下：

CRS-4133: Oracle High Availability Services has been stopped.

Successfully unlock /u01/app/11.2.0/grid

 2020-06-26 17:05:46: Invoking removeproc to clean oracle client procs

 2020-06-26 17:05:46: Executing cmd: /sbin/fuser -k /u01/app/11.2.0/grid/bin/crsctl.bin

 2020-06-26 17:05:46: Command output:

>  Failure in execution (rc=-1, 0, No such file or directory) for command /sbin/fuser -k /u01/app/11.2.0/grid/bin/crsctl.bin

>End Command output

 2020-06-26 17:05:46: f user command output for /u01/app/11.2.0/grid/bin/crsctl.bin is Failure in execution (rc=-1, 0, No such file or directory) for command /sbin/fuser -k

/u01/app/11.2.0/grid/bin/crsctl.bin

 2020-06-26 17:05:46: Oracle user for /u01/app/11.2.0/grid is grid

 2020-06-26 17:05:46: Opening file /etc/oracle/ocr.loc

 2020-06-26 17:05:46: Value (FALSE) is set for key=local_only

 2020-06-26 17:05:46: Running as user grid: /u01/app/11.2.0/grid/OPatch/opatch prereq CheckApplicable -ph /tmp/psu/27107360/26925576 -oh /u01/app/11.2.0/grid

 2020-06-26 17:05:46: s_run_as_user2: Running /bin/su grid -c ' /u01/app/11.2.0/grid/OPatch/opatch prereq CheckApplicable -ph /tmp/psu/27107360/26925576 -oh /u01/app/11.2.0

/grid '

 2020-06-26 17:05:47: Removing file /tmp/filewsfQVx

 2020-06-26 17:05:47: Successfully removed file: /tmp/filewsfQVx

 2020-06-26 17:05:47: /bin/su exited with rc=2

 2020-06-26 17:05:47: Error : The opatch Applicable check failed.  The patch /tmp/psu/27107360/26925576 is not applicable to /u01/app/11.2.0/grid

 2020-06-26 17:05:47: Running as user grid: /u01/app/11.2.0/grid/OPatch/opatch prereq CheckApplicable -ph /tmp/psu/27107360/26609929 -oh /u01/app/11.2.0/grid

 2020-06-26 17:05:47: s_run_as_user2: Running /bin/su grid -c ' /u01/app/11.2.0/grid/OPatch/opatch prereq CheckApplicable -ph /tmp/psu/27107360/26609929 -oh /u01/app/11.2.0

/grid '

 2020-06-26 17:05:48: Removing file /tmp/filePKM5qO

 2020-06-26 17:05:48: Successfully removed file: /tmp/filePKM5qO

 2020-06-26 17:05:48: /bin/su exited with rc=2

 2020-06-26 17:05:48: Error : The opatch Applicable check failed.  The patch /tmp/psu/27107360/26609929 is not applicable to /u01/app/11.2.0/grid

 2020-06-26 17:05:48: Running as user grid: /u01/app/11.2.0/grid/OPatch/opatch prereq CheckApplicable -ph /tmp/psu/27107360/22502505 -oh /u01/app/11.2.0/grid

 2020-06-26 17:05:48: s_run_as_user2: Running /bin/su grid -c ' /u01/app/11.2.0/grid/OPatch/opatch prereq CheckApplicable -ph /tmp/psu/27107360/22502505 -oh /u01/app/11.2.0

/grid '

 2020-06-26 17:05:49: Removing file /tmp/filecn51l7

 2020-06-26 17:05:49: Successfully removed file: /tmp/filecn51l7

 2020-06-26 17:05:49: /bin/su exited with rc=2

 2020-06-26 17:05:49: Error : The opatch Applicable check failed.  The patch /tmp/psu/27107360/22502505 is not applicable to /u01/app/11.2.0/grid

 2020-06-26 17:05:49: Status of Applicable  check  for /u01/app/11.2.0/grid is 0

 2020-06-26 17:05:49: Error:Patch Applicable check failed for /u01/app/11.2.0/grid

 2020-06-26 17:05:49: Executing cmd: /bin/rpm -q sles-release

 2020-06-26 17:05:49: Command output:

>  package sles-release is not installed

    觀察日誌，提示比較明顯的錯誤資訊是：The opatch Applicable check failed.  The patch /tmp/psu/27107360/...... is not applicable to /u01/app/11.2.0/grid。

按照以往的補丁應用經驗，按照如下過程進行異常排查：

    1、重新上傳補丁包，排除補丁包不完整的情況

    2、賦予補丁集777許可權，grid解壓，排除補丁包宿主許可權問題

    3、將補丁集放在單獨的空目錄，grid執行解壓

    經過上訴三個步驟，在應用補丁集時還是提示相同的錯誤，仔細觀察補丁集應用日誌，發現如下觀察錯誤提示：

 2020-06-26 17:05:46: Command output:

>  Failure in execution (rc=-1, 0, No such file or directory) for command /sbin/fuser -k /u01/app/11.2.0/grid/bin/crsctl.bin

>End Command output

 2020-06-26 17:05:46: fuser command output for /u01/app/11.2.0/grid/bin/crsctl.bin is Failure in execution (rc=-1, 0, No such file or directory) for command /sbin/fuser -k

/u01/app/11.2.0/grid/bin/crsctl.bin

    難道是/sbin/fuser命令不存在，經過排查，發現/sbin/fuser命令真的不在，死馬當活馬醫，執行yum install psmisc，在叢集

兩個節點安裝了psmisc.x86_64 0:22.20-15.el7，再次執行補丁集應用，補丁集應用成功。

# /u01/app/11.2.0/grid/OPatch/opatch auto /u01/soft/27107360 -oh /u01/app/11.2.0/grid

Executing /u01/app/11.2.0/grid/perl/bin/perl /u01/app/11.2.0/grid/OPatch/crs/patch11203.pl -patchdir /u01/soft -patchn 27107360 -oh /u01/app/11.2.0/grid -paramfile /u01/app/11.2.0/grid/crs/install/crsconfig_params

This is the main log file: /u01/app/11.2.0/grid/cfgtoollogs/opatchauto 2020-06-26_18-20-05.log

This file will show your detected configuration and all the steps that opatchauto attempted to do on your system:

/u01/app/11.2.0/grid/cfgtoollogs/opatchauto 2020-06-26_18-20-05.report.log

 2020-06-26 18:20:05: Starting Clusterware Patch Setup

Using configuration parameter file: /u01/app/11.2.0/grid/crs/install/crsconfig_params

Stopping CRS...

Stopped CRS successfully

patch /u01/soft/27107360/26925576  apply successful for home  /u01/app/11.2.0/grid

patch /u01/soft/27107360/26609929  apply successful for home  /u01/app/11.2.0/grid

patch /u01/soft/27107360/22502505  apply successful for home  /u01/app/11.2.0/grid

Starting CRS...

Installing Trace File Analyzer

ACFS driver install actions failed

CRS-4123: Oracle High Availability Services has been started.

opatch auto succeeded.