Master Note for RAC Oracle Clusterware and Oracle Grid Infrastructure 1096952.

In this Document

Applies to:

Details

This Master Note is intended to provide an index and references to the most frequently used My Oracle Support Notes with respect to Oracle Real Application Clusters, Oracle Clusterware and Oracle Grid Infrastructure implementations. This Master Note is subdivided into categories to allow for easy access and reference to notes that are applicable to your area of interest, within the RAC, Clusterware and Grid Infrastructure spaces. This includes the following categories:

• RAC Best Practice and Starter Kit Information
• Additional Notes on RAC Installation and Frequently asked Questions:
• RAC and Clusterware Monitoring, Troubleshooting & Data Collection:
  • Tools
  • Troubleshooting and Data Collection
  • My Oracle Support references
• RAC and Clusterware Testing and Performance Tuning
• RAC and Clusterware Maintenance and Patching References
  • General Maintenance References
  • Patching Best Practices and References
• Upgrade and Platform Migration for RAC
• RAC and Oracle Applications
• RAC ASM Documentation
• Maximum Availability Architecture References

Scope and Application

Oracle Real Application Clusters (RAC) is a cluster database with a shared cache architecture that overcomes the limitations of traditional shared-nothing and shared-disk approaches to provide a highly scalable and available database solution for all your business applications. Oracle RAC provides the foundation for enterprise grid computing.

Oracle Clusterware is a portable cluster software that allows clustering of single servers so that they cooperate as a single system. Oracle Clusterware provides the required infrastructure for Oracle Real Application Clusters while also enabling high availability protection of any Oracle or third party application.

Oracle Automatic Storage Management (ASM) provides a virtualization layer between the database and storage. It treats multiple disks as a single disk group, and lets you dynamically add or remove disks while keeping databases online. ASM greatly simplifies the storage complexities often associated with an Oracle database while increasing availability and performance of the database.

With Oracle grid infrastructure 11g release 2 (11.2), Oracle Automatic Storage Management (Oracle ASM) and Oracle Clusterware are installed into a single home directory, which is referred to as the Grid Infrastructure home. The installation of the combined products is called Oracle Grid Infrastructure. However, Oracle Clusterware and Oracle Automatic Storage Management remain separate products.

This note applies to the following versions of these products:

Oracle Clusterware - Version 10.1.0.2 to 11.1.0.7
Oracle Grid Infrastructure Version 11.2.0.x
Oracle Real Application Clusters Version 10.1.0.2 to 11.2.0.x

Actions

Master Note for Real Application Clusters (RAC) Oracle Clusterware and Oracle Grid Infrastructure

RAC Best Practice and Starter Kit Information

The goal of the Oracle Real Application Clusters (RAC) series of Best Practice and Starter Kit notes is to provide the latest information on generic and platform specific best practices for implementing an Oracle cluster using Oracle Clusterware or Oracle Grid Infrastructure (11gR2) and Oracle Real Application Clusters. These documents are compiled and maintained based on Oracle Support's ongoing experience with its global RAC customer base. The process used to install, configure, and create an Oracle Cluster and an Oracle Real Application Clusters (RAC) database shares much in common across all supported operating system platforms. Despite these commonalities, many aspects of the deployment process are O/S specific. As such, there is a Generic Starter Kit, as well as a separate Starter Kit for each supported platform. These Starter Kits provide the latest information in terms of installation help, as well as best practices for ongoing/existing implementations.

• Document 810394.1 RAC Assurance Support Team: RAC and Oracle Clusterware Starter Kit and Best Practices (Generic)
• Document 811306.1 RAC Assurance Support Team RAC Starter Kit (Linux) 
• Document 811280.1 RAC Assurance Support Team RAC Starter Kit (Solaris) 
• Document 811271.1 RAC Assurance Support Team RAC Starter Kit (Windows) 
• Document 811293.1 RAC Assurance Support Team RAC Starter Kit (AIX) 
• Document 811303.1 RAC Assurance Support Team RAC Starter Kit (HP-UX)

The Generic Starter Kit has attached documents with sample test plan outlines for system testing and basic load-testing to validate the infrastructure is setup correctly, as well as a sample project plan outline for your RAC implementation. In addition, best practices for Storage and networking and other areas are included. The Platform-specific Starter Kit notes complement this with Step-by-Step Installation Guides, for each particular platform, as well as best practices for the specific platform in question.

RACcheck - RAC Configuration Audit Tool

RACcheck is a RAC Configuration Audit tool designed to audit various important configuration settings within Real Application Clusters (RAC), Oracle Clusterware (CRS), Automatic Storage Management (ASM) and Grid Infrastructure environments. This utility is to be used to validate the Best Pracices and Success Factors defined in the series of Oracle Real Application Clusters (RAC) Best Practice and Starter Kit notes which are maintained by the RAC Assurance development and support teams. At present RACcheck supports Linux (x86 and x86_64), Solais (SPARC and x86_64), HP-UX (Itanium and PA-RISC with the bash shell) and AIX (with the bash shell) platforms. Those customers running RAC on the RACcheck supported platorms are strongly encouraged to utilize this tool identify potential configuration issues that could impact the stability of the cluster.

• Document 1268927.1 RACcheck - RAC Configuration Audit Tool

Note:  Oracle is constantly generating and maintaining Best Practices and Success Factors from the global customer base.  As a result the RACcheck utility is frequently updated with this information.  That said, it is recommended that you ensure you are using the version of RACcheck prior to execution.

Additional Notes on RAC and Clusterware Installation and Frequently asked Questions

While the Starter Kits referenced above will contain much of what you need to get started, additional notes and resources that are important to take heed of, particularly for a new implementation, can be found here in the following FAQ???s and other references:

• Document 220970.1 RAC: Frequently Asked Questions
• Document 316817.1 CLUSTER VERIFICATION UTILITY FAQ 
• Document 332257.1 Using Oracle Clusterware with Vendor Clusterware FAQ 
• Document 397460.1 Oracle's Policy for Supporting RAC 10g with Symantec SFRAC 
• Document 391771.1 OCFS2 - Frequently Asked Questions 
• Document 787420.1 Cluster Interconnect in Oracle 10g and 11g
• Document 1328466.1 Cluster Health Monitor (CHM) FAQ

Key notes related to OS Installation requirements for a RAC or Oracle Clusterware environment

• Document 359515.1 Mount Options for Oracle files when used with NAS devices
• Document 401132.1 How to install Oracle Clusterware with shared storage on block devices 
• Document 357472.1 Configuring device-mapper for CRS/ASM 
• Document 169706.1 Oracle Database on AIX,HP-UX,Linux,MacOSX,Solaris,Tru64 Operating Systems Installation and Configuration Requirements Quick Reference (8.0.5 to 11.1) 

Key notes introducing differences in Oracle 11gR2

• Document 1053147.1 11gR2 Clusterware and Grid Home - What You Need to Know
• Document 948456.1 Pre 11.2 Database Issues in 11gR2 Grid Infrastructure Environment 
• Document 1050908.1 How to Troubleshoot Grid Infrastructure Startup Issues
• Document 887522.1 11gR2 Grid Infrastructure Single Client Access Name (SCAN)Explained

RAC and Clusterware Monitoring, Troubleshooting and Data Collection Notes

To effectively manage your RAC cluster, it is important to know how to monitor, troubleshoot and collect data, both for your own diagnosis, and also to provide supporting information when logging a service request. The following Notes will give you information on how to do both:


Tools for Monitoring RAC and Clusterware Environments

• Document 301137.1 OS Watcher User Guide
• Document 459694.1 Procwatcher: Script to Monitor and Examine Oracle and CRS 
• Document 736752.1 Introducing Cluster Health Monitor 
• Document 811151.1 How to install Oracle Cluster Health Monitor (former IPD/OS) on Windows 
• Document 314422.1 Remote Diagnostics Agent (RDA) 
• Document 422893.1 11g Understanding Automatic Diagnostic Repository 
• Document 559365.1 Using Diagwait as a Diagnostic.
• Document 1268927.1 RACcheck - RAC Configuration Audit Tool
• Document 135714.1  Script to Collect RAC Diagnostic Information (racdiag.sql)

References needed for Troubleshooting and Data Collection

• Document 289690.1 Data Gathering for Troubleshooting CRS issues 
• Document 330358.1 CRS 10gR2/ 11gR1/ 11gR2 Diagnostic Collection Guide 
• Document 557204.1 Crsd Ocssd Evmd Log Rotation Policy 
• Document 563566.1 Troubleshooting gc block lost and Poor Network Performance in a RAC Environment
• Document 219361.1 Troubleshoot ORA-29740 errors in a RAC Environment. 
• Document 745960.1 11g How to Unpack a Package in to ADR 
• Document 443529.1 11g Quick Steps to Package and Send Critical Error Diagnostic Information to Support 
• Document 726446.1 What Is The Difference Between RDA and Configuration Manager 
• Document 77483.1 External Support FTP site: Information Sheet 
• Document 783456.1 CRS Diagnostic Data Gathering: A Summary of Common tools and their Usage
• Document 1050693.1 Troubleshooting 11.2 Clusterware Node Evictions (Reboots)
• Document 265769.1 Troubleshooting 10g and 11.1 Clusterware Reboots
• Document 1367153.1 Top 5 Issues That Cause Node Reboots or Unexpected Recycle of CRS
• Document 1485042.1  11gR2 GI/ASM/RAC Universal Collection Guide

RAC and Clusterware Testing and Performance Tuning

Much of RAC performance tuning is no different from single instance database?? tuning. However, there are some additional areas to check in a RAC environment when tuning or troubleshooting performance related issues. The following notes will make you aware of some of the areas where performance tuning and diagnosis has a specific RAC flavor:

• Document 390374.1 Oracle Performance Diagnostic Guide (OPDG)
• Document 181489.1 Tuning Inter-Instance Performance in RAC and OPS 
• Document 395314.1 RAC Hangs due to small cache size on SYS.AUDSES$ 
• Document 563566.1 gc lost blocks diagnostics 
• Document 361323.1 HugePages on Linux: What It Is... and What It Is Not... 
• Document 280939.1 Checklist for Performance Problems with Parallel Execution 
• Document 759565.1 Turning NUMA on can cause database hangs

RAC and Clusterware Certification, Maintenance, and Patching References

Maintenance of a RAC environment is similar in many respects to maintaining a single-instance environment. However, with the Clusterware and Grid Infrastructure stacks, there are some additional tasks to be aware of regarding ongoing maintenance, as well as maintenance that may be brought on by changes to your environment. Below are some references speaking to the areas of general maintenance that may be needed or affect RAC environments in a particular way, as well as references to patching, which is a key part of an ongoing maintenance strategy.

Certification References for RAC and Clusterware Environments

• Document 184875.1 How To Check The Certification Matrix for Real Application Clusters
• Document 1065024.1 Database 11g Release 2 Certification Highlights
• Document 337737.1 Oracle Clusterware - ASM - Database Version Compatibility

General Maintenance References for RAC and Clusterware Environments

• Document 428681.1 How to ADD/REMOVE/REPLACE/MOVE Oracle Cluster Registry (OCR) and Voting Disk
• Document 276434.1 Modifying the VIP or VIP Hostname of a 10g or 11g Oracle Clusterware Node. 
• Document 283684.1 How to Change Interconnect/Public Interface IP or Subnet in Oracle Clusterware 
• Document 754594.1 Potential disk corruption when adding a node to a cluster
• Document 403743.1 VIP Failover Take Long Time After Network Cable Pulled 
• Document 563905.1 Implementing LIBUMEM for CRS on Solaris 64 with 3rd Party Clusterware 
• Document 92602.1 How to Password Protect the Listener 
• Document 1373500.1  Top 5 Database and/or Instance Performance Issues in RAC Environment

Patching Best Practices and References for RAC and Oracle Clusterware
As part of an overall maintenance strategy, it is critical that customers have a formal strategy to stay in front of known issues and bugs. To make it easier for customers to obtain and deploy fixes for known critical issues please refer to the following list of references.

• Document 209768.1 Database, FMW, Em Grid Control and OCS Software Error Correction Support Policy.
• Document 316900.1 Oracle 10g Release 2 (10.2) Support Status and Alerts 
• Document 454507.1 Oracle 11g Release 1 (11.1) Support Status and Alerts 
• Document 742060.1 Release Schedule of Current Database Releases. 
• Document 756671.1 Oracle Recommended Patches -- Oracle Database. 
• Document 854428.1 Introduction to Patch Set Updates (PSUs) 
• Document 850471.1 Oracle Announces First Patch Set Update For Oracle Database Release 10.2 
• Document 405820.1 10.2.0.X CRS Bundle Patch Information. 
• Document 810663.1 11.1.0.X CRS Bundle Patch Information. 
• Document 1082394.1 11.2.0.X Grid Infrastructure Bundle Patch Information. 
• Document 363254.1 Applying one-off Oracle Clusterware patches in a mixed version home environment 
• Document 761111.1 Online Patches (Hot Patching). 
• Document 438314.1 Critical Patch Update - Introduction to Database n-Apply CPUs. 
• Document 1082394.1 11.2.0.X Grid Infrastructure PSU Known Issues

Upgrade and Platform Migration for RAC and Oracle Clusterware

When migrating a RAC cluster to a new platform or upgrading a RAC cluster from an older version of Oracle to the current version of Oracle, the following references will be helpful:

• Document 466181.1 Oracle 10gR2 Upgrade Companion.
• Document 601807.1 Oracle 11gR1 Upgrade Companion. 
• Document 785351.1 Oracle 11gR2 Upgrade Companion. 
• Document 429825.1 Complete Checklist for Manual Upgrades to 11gR1. 
• Document 747457.1 How to Convert 10g Single-Instance database to 10g RAC using Manual Conversion procedure. 
• Document 338706.1 Cluster Ready Services (CRS) rolling upgrade

References for running RAC with Applications

• Document 790189.1 Oracle Clusterware and Application Failover Management

RAC and Oracle E-Business

• Document 362135.1 Configuring Oracle Applications Release 11i with Oracle10g Release 2 Real Application Clusters and Automatic Storage Management.
• Document 380489.1 Using Load-Balancers with Oracle E-Business Suite Release 12 
• Document 727171.1 Implementing Load Balancing On Oracle E-Business Suite - Documentation For Specific Load Balancer Hardware 
• Document 217368.1 Advanced Configurations and Topologies for Enterprise Deployments of E-Business Suite 11i 
• Document 403347.1 MAA Roadmap for the E-Business Suite 
• Document 388577.1 Using Oracle 10gR2 RAC and Automatic Storage Management with Oracle E-Business Suite Release 12

RAC ASM Documentation

Automatic Storage Management (ASM) is an evolution in file system and volume management functionality for Oracle database files. ASM further enhances automation and simplicity in storage management that is critical to the success of the Oracle grid architecture. ASM also improves file system scalability and performance, manageability and database availability for RAC environments. Use of ASM with RAC is an Oracle Best Practice.

• Document 1187723.1 Master Note for Automatic Storage Management (ASM)
• Document 1187674.1 Master Note for Oracle Database Machine and Exadata Storage Server

MAA/Standby Documentation

Data Guard provides the management, monitoring, and automation software infrastructure to create and maintain one or more standby databases to protect Oracle data from failures, disasters, errors, and data corruptions. As users commit transactions at a primary database, Oracle generates redo records and writes them to a local online log file.

• Document 1101938.1 Master Note for Data Guard

Using My Oracle Support Effectively

• Document 166650.1 Working Effectively with Global Customer Support
• Document 747242.5 My Oracle Support FAQ
• Document 868955.1 My Oracle Support Health Checks Catalog
• Document 199389.1 Escalating Service Requests with Oracle Support Services

References