In this Document

Applies to:

Purpose

This note lists the top opatch or patch related questions or problems in Oracle Clusterware (11gR2 Grid Infrastructure or pre-11.2 CRS) and RAC environments.

It does not intend to replace the readme that comes with a patch, rather it's recommended to go through the patch readme thoroughly and follow it. Occasionally a patch readme may contain incorrect information, in that case, please engage Oracle Support to seek clarification.

For general opatch questions or problems, refer to Document 293369.1

Questions and Answers

Oracle Home Type

What type of home do I have?

In a pre-11.2 Clusterware environment you will have:

•  A single Clusterware (CRS) home
•  One or more Database (RDBMS) homes at the same or at a lower release level than the Clusterware home (to the 4th dot in the release)
•  Optionally a separate dedicated RDBMS home to run ASM (Automatic Storage Management) at the same or at a lower release level than the Clusterware home (to the 4th dot in the release)
• In pre-11.2 installations, ASM was implemented using the RDBMS binary installation so this should be treated as a typical RDBMS home

In a 11.2 Grid Infrastructure environment you will have:

•  A single Grid Infrastructure (GI) Home that implements both ASM and the Clusterware
  
  Note: During upgrade from pre-11gR2 to 11gR2 GI, the ASM upgrade can optionally be performed independently of the Clusterware (not recommended).  In this case the pre-11gR2 ASM Home would still be active after the GI Upgrade.  This is NOT the recommended method of upgrading to Grid Infrastructure and will therefore not be discussed in this note.
  
  Note: Starting with 11gR2 all upgrades of and to Grid Infrastructure MUST be performed "out-of-place" (into a new software home) for this reason there may be more than one clusterware home, however only one will be the active.  This "inactive" Clusterware or GI home should be removed once satisfied that the upgrade was successful.
•  One or more database (RDBMS) homes at the same or at a lower release level than the GI home (to the 4th dot in the release)

Types of Patches

What Types of Patches are available for Oracle Clusterware, Grid Infrastructure and/or the Database?

Generally patches for Clusterware, Grid Infrastructure and/or the Database are catagorized into the following:

• Patchsets
• Patchset Updates (PSUs)
• Critical Patch Updates (CPUs)
• Bundle Patches
• Interim (one-off) Patches

Patchsets Q&A

Compared to all other patch types, a Patchset is released the least frequently. It contains fixes for most known issues for the release and potentially also introduces new features. A patchset is cumulative and when applied it changes the fourth digit of the product release banner - for example, 10.2.0.5 is 4th patch set for 10.2, and 11.2.0.2 is the 1st patch set for 11.2.

A patchset must be installed via the Oracle Universal Installer (OUI) and is generally considered an "upgrade".

Prior to 11gR2, a base release had to be installed before a patchset could be applied. For example, to install 10.2.0.5 on Linux, the 10.2.0.1 base release has to be installed first and then upgraded to 10.2.0.5.

Prior to 11gR2, the same patchset download is used to patch the Clusterware, ASM and Database homes.  For example, Patch 8202632 is the 10.2.0.5 patchset, this same patch (Patch 8202632) will be used to patch the 10.2.0.x Clusterware, 10.2.0.x ASM and 10.2.0.x Database to 10.2.0.5.

Starting with 11gR2, patchset releases are now full releases and no longer require a "base" release, e.g. 11.2.0.2 can be installed directly without having to install 11.2.0.1 first.

Prior to 11gR2 - even though the CRS and RDBMS base releases were provided on separate media (downloadable zip file or separate DVD/CD) - the patchsets for both products were delivered as one i.e. the same patchset could be applied to the CRS as well as the RDBMS home.

Starting with 11gR2 the patchsets for Grid Infrastructure and RDBMS are delivered separately (as they are full releases).

Clusterware patchsets can be applied in a rolling fashion, while database patchsets cannot. For example, you can rolling upgrade clusterware to 11.2.0.2, but you have to shutdown database on all nodes to upgrade to database 11.2.0.2.

Patch Set Updates (PSUs) Q&A

As the name implies PSUs are patches that are applied on top of a given patchset release. They are released on a quarterly basis and contain fixes for known critical issues for the patchset. PSUs are subject to thorough testing and do not include changes that would alter the functionality of the software.  With this in mind, PSUs are designed to provide "low risk" and "high value" allowing for customers to more easily adopt proactive patching strategies. Consider the following PSU facts:

• All PSUs are installed via "opatch" and are not considered an "upgrade".
• Database PSUs always contain the CPU for the respective quarter that the PSU is released in.  PSUs and CPUs are NOT compatible meaning if you apply the 11.2.0.2.2 Database PSU and then want to apply the 11.2.0.2 July CPU this would result in the rollback of the 11.2.0.2.2 Database PSU.  That said, once a PSU patching strategy is adopted it must be maintained.

• Independent PSUs are released for both the Database and Clusterware or Grid Infrastructure installations.
• Clusterware PSUs (pre-11.2) are refered to as CRS PSUs
• Grid Infrastructure PSUs are refered to as GI PSUs
• GI PSUs do contain the Database PSU for the corresponding release, e.g.  11.2.0.2.3 GI PSU contains the 11.2.0.2.3 Database PSU
• Database PSUs hold true to their name

• Both Clusterware/Grid Infrastructure and Database PSU patches are cumulative. Clusterware PSU refers to CRS PSU for pre-11gR2 and GI PSU for 11gR2.

• GI PSUs are always cumulative meaning that you can apply higher version GI PSU directly without having to apply a lower version one first. For example, the 11.2.0.2.2 GI PSU can be applied to a 11.2.0.2 home without having to apply GI PSU 11.2.0.2.1 first.
• Database PSUs can be subject to overlay PSU packaging.  In these cases, the PSUs are still cumulative, but a higher PSU may require a lower PSU to be applied first; for example, to apply database PSU 10.2.0.4.7, you must apply database PSU 10.2.0.4.4 first.  If a previous PSU is a prerequisite to a later PSU the requirement will be clearly documented in the PSU readme.
• For more information on PSUs please review Document 854428.1.

Generally speaking PSU are released on quarterly basis for both Clusterware/Grid Infrastructure and Database.  There's cases where a Clusterware PSUs is not released for corresponding Database PSU. For example, there's database PSU 10.2.0.5.4 but no CRS PSU 10.2.0.5.4.

A PSU will not physically update the 5th-digit of the release information, the updates to the 5th digit in the release are for Documentation purposes only. So the third GI PSU that was released for 11.2.0.2 will have a documentation version of 11.2.0.2.3. You will NOT see this change reflected in the actual software version if you query it from the inventory, clusterware or database.

Unlike other Grid Infrastructure patches (discussed later), 11gR2 GI PSUs contains both GI PSU and Database PSU (YES, both GI and DB PSU) for a particular quarter. For example, 11.2.0.2.2 GI PSU contains both the 11.2.0.2.2 GI PSU and the 11.2.0.2.2 Database PSU. 

You are able to take note of this when you extract a GI PSU, you will see 2 directories (named with the Patch number) one for the GI PSU and one for the RDBMS PSU.

How do I find out whether a bug is fixed in a Clusterware or Grid Infrastructure PSU ? To find out, check the patch readme and the following notes:

Once the GI PSU is applied, "opatch lsinventory" will show that both GI PSU and DB PSU are applied, i.e.:

And "opatch lsinventory -bugs_fixed" will list each individual bugs that have been fixed by all installed patches, i.e.:

No, only CRS PSUs, GI PSUs or other Clusterware/GI patches can be applied to a Clusterware/GI home.

Critical Patch Updates (CPUs) Q&A

CPU Patches are collection of high-priority fixes for security related issues and are only applicable to the Database Home (and pre-11.2 ASM Home(s)) .  CPUs are released quarterly on the same cycle as PSUs and are cumulative with respect to prior security fixes but may contain other fixes in order to address patch conflicts with non-security patches.  PSUs always contain the CPU for that respective quarter.  PSUs and CPUs are NOT compatible meaning if you apply the 11.2.0.2.2 Database PSU and then want to apply the 11.2.0.2 July CPU this would result in the rollback of the 11.2.0.2.2 Database PSU. That said, once a PSU patching strategy is adopted it must be maintained.  PSU patching is preferred over CPU Patching.

Bundle Patches Q&A

A Clusterware or Grid Infrastructure (GI) patch can be in the form of bundle or Patchset Update (PSU).  The biggest difference between a GI/Clusterware bundle and a PSU is that PSUs are bound to a quarterly release schedule while a bundle may be released at any time throughout the course of a given quarter.  If a GI/Clusterware bundle is released in a given quarter, the fixes in that bundle will be included in the PSU that will be released for that quarter.  This concept allows for development to provide critical bug fixes in a more granular timeline if necessary.

Interim (one-off) Patch Q&A

An interim patch contains fixes for one or in some cases several bugs (merge patch).

Clusterware interim patches are rare, they usually are build on top of the latest PSU (at the time) and include the entire PSU they were built on.

The same does not hold true for database interim patches, they usually do not include a PSU.

An interim clusterware patch on top of a PSU includes the PSU, for example, 11.2.0.2.2 patch 12647618 includes 11.2.0.2 GI PSU2 (11.2.0.2.2). But the same is not true for database interim patch, for example, 11.2.0.2.2 database patch 11890804 can only be applied on top of a 11.2.0.2.2 database home.

General Patch Q&A

Generally speaking Clusterware/Grid Infrastructure patches modify files that belong to the Clusterware or Grid Infrastructure product, while Database patches change files that belong to the database product.  As they apply to different sets of files, they do not conflict with each other.

Please note:

•  "files" in this context can refer to binaries/executables, scripts, libraries etc
•  Clusterware files can reside in all types of Oracle software homes like clusterware home, database home and ASM home
•  Prior to 11gR2, RDBMS files reside in DB/ASM homes only, while with 11gR2 RDBMS files will also reside in the GI home (as ASM is now part of GI)
•  A GI PSU is a special type of clusterware patch as it also includes a database PSU and modifies database binaries.

Clusterware and Grid Infrastructure patches have at least two portions, one for the clusterware home, and the other for the Database home(s). Those two portions have same version number. The clusterware home portion is in the top level directory of the unzipped location, while the other portion is in custom/server/.  For example, if 11.2.0.2.2 GI PSU is unzipped to /op/db/11.2.0.2/GIPSU2, Grid Infrastructure home portion will be in /op/db/11.2.0.2/GIPSU2/12311357 and database home specific portion will be in /op/db/11.2.0.2/GIPSU2/12311357/custom/server/12311357.  This is just an example, full details will be in the README for the patch that is being applied.  ALWAYS consult the patch README prior to applying any patch.

Which Patch Applies to Which Home

What's Oracle's patching recommendation?

Oracle recommends to apply the latest patchset and latest PSU as a general best practice.

Which oracle home does a patch apply to?

A home must meet the patch version specification to be applicable.

In CRS environments, a Clusterware patch (interim, MLR, bundle or PSU) applies to all homes (CRS, ASM and database), while a Database patch applies to the ASM and Database home only

In Grid Infrastructure environments, a GI patch (interim, bundle or PSU) applies to all homes (GI and Database), while a Database patch could be applicable to GI/ASM home if the fix applies to ASM (in this case patch README will tell clearly that it applies to GI/ASM home).

How do I tell from the patch readme which home the patch applies to?

The patch README usually tells which home it applies to.

Common identifier in patch README:

Do Exadata patches apply to GI or RAC homes?

General speaking, Exadata patches should only be applied to Exadata environments.

Can I upgrade the Clusterware or Grid Infrastructure to a higher version while leaving database at a lower version?

Yes, as long as Clusterware/Grid Infrastructure is at higher version than the RAC Database homes on the cluster this is perfectly acceptable, refer to Document 337737.1 for details

Do I need downtime to apply a Clusterware or Grid Infrastructure patch?

If the Clusterware/Grid Infrastructure home is not shared (common), Clusterware/Grid Infrastructure patches can be applied in rolling fashion so no downtime (of the entire cluster) is needed.

Which PSU patch applies to what home in mixed environments (clusterware and database at different version)?

Example 1:

Example 2:

OPatch Q&A

How to find out the opatch version?

To find out the opatch version perform the following as the Oracle software owner:

How do I install the latest OPatch release?

Prior to applying a patch to a system it is HIGHLY recommended (often required) to download and install the latest version of the OPatch utility into every ORACLE_HOME on every cluster node that is to be patched.  The latest version of OPatch can be found on MOS under Patch 6880880.  Be sure to download the release of OPatch that is applicable for your platform and major release (e.g. Linux-x86-64 11.2.0.0.0).  Once OPatch has been downloaded and staged on your target system(s) it can be installed by executing the following as the Oracle Software installation owner:

Why is "opatch auto" not patching my RAC database home?

Why is "opatch auto" not patching my RAC database home, refer to note 1479651.1 

What's the difference between manual opatch apply and opatch auto?

In a single instance environment to apply database patch it is very simple.  Shutdown all processes running from a home to be patched and invoke "opatch apply" or "opatch napply".  Very simple.

Now, to apply a Clusterware patch to a Clusterware Home or a GI patch to Grid Infrastructure Home, the clusterware stack needs to be down, the clusterware home needs to be unlocked, the configuration needs to be saved, then the patch can be applied; once the patch is applied, the configuration needs to be restored, the clusterware home needs to be re-locked again, then the clusterware stack can start.  Once the Clusterware patch or GI patch was applied to the Clusterware/GI home, you then have to go and apply the database components of that patch to the Database Home with a whole other list of steps.  This is a complex process that had proven to be error prone if instructions were not carefully followed.

This is where OPatch Auto comes in.  The whole goal of OPatch auto is for an administrator to only have to execute a single command to apply a patch to a RAC system, thus simplifying the process.

How do I apply a Grid Infrastructure patch before the root script (root.sh or rootupgrade.sh) is executed?

Refer to note 1410202.1 for details.

How to apply a patch after the root script (root.sh or rootupgrade.sh) has failed?

In some cases root script fails because of missing patch (e.g. Document 1212703.1 Oracle Grid Infrastructure 11.2.0.2 Installation or Upgrade may fail due to Multicasting Requirement).

When this happens we must first determine whether the root script had made it to the point in which the ownership/permissions of the GI software has changed.  This will allow us to make the determination if unlocking the GI software is necessary prior to installing the patch.  To determine if unlocking is required, go to the GI home and execute:

If the above command returns no files, unlocking the software is not necessary.  Should the command return root owned files you must unlock the GI Home by executing the following as the root user:

Once the GI home is unlocked (or it was determined that unlocking was not necessary, refer to answer for the question "How do I apply a Grid Infrastructure patch before the root script (root.sh or rootupgrade.sh) is executed?" to install the required patch.

How to apply a Clusterware or Grid Infrastructure patch manually?

It's recommended to use OPatch auto when applicable; but in the case that OPatch auto does not apply or fails to apply the patch, refer to patch README and the following notes for manual patch instructions.  The following notes may also be of use when in this situation:

Examples

OPatch Auto Example to Apply a GI PSU (includes Database PSU)

• Platform is Linux 64-bit
• All Homes (GI and Database) are not shared
• All Homes are 11.2.0.2
• The Grid Infrastructure owner is grid
• The Database owner is oracle
• The Grid Infrastructure Home is /ocw/grid
• Database Home1 is /db/11.2/db1
• Database Home2 is /db/11.2/db2
• The 11.2.0.2.3 GI PSU will be applied to in this example
• ACFS is NOT in use on this cluster

 

1. Create an EMPTY directory to stage the GI PSU as the GI software owner (our example uses a directory named gipsu3):

2. Extract the GI PSU into the empty stage directory as the GI software owner:

3. Verify that opatch in ALL 11.2.0.2 homes that will be patched meet minimum version requirement documented in the PSU README (see "How to find out the opatch version?").  If the version of OPatch in any one (or all) of the homes does not meet the minimum version required for the patch, OPatch must be upgraded in this/these homes prior to continuing (see "How do I install the latest OPatch release?").

4. As grid user repeat the following to validate inventory for ALL applicable homes on ALL nodes:

5. As the user root, execute OPatch auto to apply GI PSU 11.2.0.2.3 to all 11.2.0.2 Grid Infrastructure and Database homes:

6. As the grid user repeat validate inventory for all patched homes:

7. Repeat steps 1-6 on each of the remaining cluster nodes, 1 node at a time.

8. If you applied the Databse PSU to the Database Homes (as shown in this example), you must now load the Modified SQL Files into the Database(s) as follows:

EXAMPLE:  Apply a CRS patch manually

The example assumes the following:

• Platform is Solaris SPARC 64-bit
• All homes (CRS, ASM and DB) are non-shared
• All homes are version 11.1.0.7
• The Clusterware Home is /ocw/crs
• The Clusterware, ASM and Database owner is oracle
• The ASM Home is /db/11.2/asm
• Database Home 1 is /db/11.1/db1
• Database Home 2 is /db/11.1/db2

 1. Create an EMPTY directory to stage the 11.1.0.7.7 CRS PSU as the CRS software owner (our example uses a directory named crspsu7):

2. Extract the CRS PSU into the empty stage directory as the CRS software owner:

3. Verify that opatch in ALL 11.1.0.7 homes (for the Database homes there is a Database compnent to CRS patches) meet minimum version requirement documented in the PSU README (see "How to find out the opatch version?"). If the version of OPatch in any one (or all) of the homes does not meet the minimum version required for the patch, OPatch must be upgraded in this/these homes prior to continuing (see "How do I install the latest OPatch release?").

4. As oracle repeat the following to validate inventory for ALL applicable homes:

5. On the local node, stop all instances, listeners, ASM and CRS:

6 As root execute the preroot patch script:

7. As the oracle user execute the prepatch script for the CRS Home:

8. As the oracle user execute the prepatch script for the Database/ASM Homes:

9. As oracle  apply the CRS PSU to the CRS Home using opatch napply:

10. As the oracle user apply the database compnent of CRS PSU to the Database/ASM Homes using opatch napply:

11. As the oracle user execute the postpatch script for the CRS Home:

12. As the oracle user execute the postpatch script for the Database/ASM Homes:

13. As root execute the postrootpatch script (this script will start the CRS stack):

14. As the oracle user repeat the following to validate inventory for ALL applicable homes:

15. Start all instances, listeners and ASM on local node (CRS was started with the postrootpatch script):

16. Repeat steps 1-15 on each node in the cluster, one node at a time.

EXAMPLE: Applying a GI PSU patch manually

The example assumes the following:

• Platform is Linux 64-bit
• All Homes (GI and Database) are not shared
• The Grid Infrastructure owner is grid
• The Database owner is oracle
• The Grid Infrastructure Home is /ocw/grid
• Database Home1 is /db/11.2/db1
• Database Home2 is /db/11.2/db2
• All Homes are 11.2.0.2
• ACFS is NOT in use on this cluster

1. Create an EMPTY directory to stage the GI PSU as the GI software owner (our example uses a directory named gipsu3):

2. Extract the GI PSU into the empty stage directory as the GI software owner:

3. Verify that opatch in ALL 11.2.0.2 homes that will be patched meet minimum version requirement documented in the PSU README (see "How to find out the opatch version?"). If the version of OPatch in any one (or all) of the homes does not meet the minimum version required for the patch, OPatch must be upgraded in this/these homes prior to continuing (see "How do I install the latest OPatch release?").

4. As grid user repeat the following to validate inventory for ALL applicable homes on ALL nodes:

5. On the local node, use the "srvctl stop home" command to stop the database resources:

6. As root unlock the Grid Infrastructure Home as follows:

7. Execute the prepatch script for the Database Homes as the oracle user:

8. As the grid user apply the patch to the local GI Home using opatch napply:

9. As the oracle user apply the patch to the local Database Homes using opatch napply:

10. Execute the postpatch script for the Database Homes as the oracle user:

11. At the root user execute the rootadd_rdbms script from the GI Home:

12. As root relock the Grid Infrastructure Home as follows (this will also start the GI stack):

 
13. Restart the database resources on the local node using the "srvctl start home" command:

14. As grid user repeat the following to validate inventory for ALL patched homes:

15. Repeat steps 1-14 on each node in the cluster, one node at a time.

16. If you applied the Databse PSU to the Database Homes (as shown in this example), you must now load the Modified SQL Files into the Database(s) as follows:

Common OPatch Failures and Troubleshooting

What files to review if opatch auto fails?

The key to OPatch Auto troubleshooting is understanding where and how the OPatch Auto logs are generated. When OPatch auto is invoked, it will generate a opatchauto.log log in the $ORACLE_HOME/cfgtoollogs directory of the ORACLE_HOME that opatch auto was invoked. This log will contain information around the execution of OPatch auto itself and will be the starting point for troubleshooting issues. Now, OPatch auto will also invoke additional OPatch sessions using the OPatch executable from the home that is being patch to perform the pre-req checks and actually apply the patch. Each of these subsequent invokations OPatch will generate a new opatch.log log under $ORACLE_HOME/cfgtoollogs/opatch within the ORACLE_HOME that is being patched. These logs are often times where the root cause of an OPatch auto failure will be found.

Should OPatch auto fail to restart the stack (noted in the error message) one should consult the rootcrs or roothas log (11.2 only) in $GRID_HOME/cfgtoollogs/crsconfig, as OPatch auto actually makes calls "rootcrs.pl" or "roothas.pl".  Should an SR need to be opened up in the event of of a OPatch auto failure it is recommended to collect the OPatch logs (discussed above) as well as the output of the diagcollection script as described Document 330358.1> and provide this diagnostic data at the time of SR creation.

"opatch apply" or "opatch napply" failure if clusterware home is not unlocked

If "opatch napply" or "opatch apply" is executed without unlocking clusterware home first, the following will be reported:

The solution is to carefully follow the instructions in the README packaged with the patch which will provide instructions use "opatch auto" if applicable or the manual opatch process.

OPatch reports: "Prerequisite check CheckSystemSpace failed"

On AIX, due to unpublished bug 9780505 opatch needs about 22GB of free disk space in $GRID_HOME to apply a GI PSU/Bundle compared to about 4.4GB on other platforms. "opatch util cleanup -oh " can be used to reclaim space in .patch_storage, refer to Document 550522.1 and Document 1088455.1 for a full explanation on this issue as well as and other tips to work around this issue.

Common causes of "The opatch version check failed"

ZOP-49: Not able to execute the prereq. OPatch cannot inform if the patch satisfies minimum version requirement.

• Incorrect patch location specified
• OPatch version does not meet the requirements for the patch (specified in the README), you must upgrade OPatch.
• The grid or database user can not read unzipped patch directory/files, check directory permissions

OPatch reports: "Patch nnn requires component(s) that are not installed in OracleHome"

These not-installed components are oracle.crs:, oracle.usm:

The patch is not applicable to the target home, i.e. if apply 11.2.0.3 GI PSU1 to 11.2.0.2 GI home, the error will be reported, refer to Document 763680.1 for troubleshooting instructions.

Common causes of "The opatch Component check failed"

• Incorrect patch location specified
• The patch was unzipped in non-empty directory
• The proper version of the OPatch executable was is not installed into the ORACLE_HOME being patched and OPatch was not launched from that ORACLE_HOME
• The current working directory is $ORACLE_HOME/OPatch of the ORACLE_HOME that is being patched
• The $ORACLE_HOME/.patch_storage cannot be created, the solution is to create the directory manually with corresponding grid or database user with permission of 750
• The listener is running from database home, the solution is to shutdown listeners from database home
• The grid and database users are different, the workaround is to apply manually
• The patch is intended for other version, i.e. when applying 11.2.0.2 GI PSU5 to 11.2.0.3 GI home

Refer to Document 1169036.1 for troubleshooting instructions.

Common causes of "The opatch Conflict check failed"

• A true patch conflict, the solution is to open an SR and request merge patch
• A Clusterware patch may report a conflict even it is super set of the currently installed patch.  Normally OPatch should rollback the installed installed patch in favor of the super set, but in some rare case, OPatch errors out; the solution is to manually rollback and apply the new patch.

Common causes of "The opatch Applicable check failed"

ZOP-46: The patch(es) are not applicable on the Oracle Home because some patch actions are not applicable. All required components, however, are installed.

• Some of the files in target home are still being locked - some processes still running, or "rootcrs.pl -unlock" / "roothas.pl -unlock" has not been executed
• Unzipped patch files in stage area are unreadable by grid or database user: "Copy Action: Source File "/patches/B202GIPSU4/12827731/files/bin/appvipcfg.pl" does not exists or is not readable". The solution is to unzip the patch by any OS user that has primary group oinstall, for example, grid user.
• Database control (Enterprise Manager)/EM agent is not stopped: "Copy Action: Desctination File "/ocw/grid/11.2/bin/emcrsp.bin" is not writeable.". To stop, as the ORACLE_HOME owner execute:  $ /bin/emctl stop dbconsole
• A directory is provides when prompted "OPatch  is bundled with OCM, Enter the absolute OCM response file path", the solution is to enter absolute OCM response file directory and filename

For other potential causes, review opatch logfile.  The OPatch log file can be found in $ORACLE_HOME/cfgtoollogs/opatch.

Common causes of "patch   apply  failed  for home "

• dbconsole is not down - it needs to be stopped prior to patching, as database home owner, execute "/bin/emctl stop dbconsole" to stop it
• opatch bug 12732495, refer to note 1472242.1 for more details.
• opatch bug 14308442 which is closed as duplicate of bug 14358407, affects AIX only, refer to note 1518022.1 for more details.
• other opatch bug - use latest opatch to avoid known issues.

For other potential causes, review opatch logfile.  The OPatch log file can be found in $ORACLE_HOME/cfgtoollogs/opatch.

When applying online patch in RAC: Syntax Error... Unrecognized Option for Apply .. OPatch failed with error code 14

Basically the earlier patch readme template is wrong for online patch for RAC, refer to note 1356093.1 for details.

opatch Fails to Rollback Online(Hot) Patch in RAC With oracle/ops/mgmt/cluster/NoSuchNodeException and error code 30

Refer to note 1416823.1 for details.

opatch Fails to Rollback Online(Hot) Patch With Prerequisite check "CheckRollbackSid" and error code 30

Refer to note 1422190.1 for details.

Common causes of "defined(@array) is deprecated at crs/crsconfig_lib.pm"

"opatch auto" reports the following warning message:

The issue happens when applying 11.2.0.3.1 (patch 13348650) with latest opatch, it's being tracked via internal bug 13556626 bug 13555176, the message can be safely ignored

opatch auto Reports: The path "/bin/acfsdriverstate" does not exist

"opatch auto" reports the following message:

The message can be ignored, refer to note 1469969.1 for details 

Applying [PSU] patch takes very long time (hours) after "verifying the update"

opatch or "opatch auto" takes hours to finish after the following message is printed:

This is due to bug 13963741 which at the time of this writing is still being worked by Development, the workaround is to unset environment variable "JAVA_COMPILER=NONE", refer to note 1541186.1 for more details.

New OPatch Features

To find out new features in opatch 11.2.0.3.2, refer to note 1497639.1 

For searchability:
database home; RAC home; RDBMS home; GI home; DB home; ASM home
RDBMS_HOME; ASM_HOME; DB_HOME; ASM_HOME
GI user; ASM user; DB user; database user

References