Configuring non-raw Multipath Devices for Oracle Clusterware 11g

Configuring non-raw Multipath Devices for Oracle Clusterware 11g (11.1.0, 11.2.0) on RHEL6/OL6

In this Document
 Goal
 Solution
  Udev and Device Name Persistency
  Multipath and Udev
  Configuring non-raw multipath devices for Oracle Clusterware 11g (11.1.0/11,2,0) on RHEL6/OL6
  Assumptions
  1. Use scsi_id to  Return Unique Device Identifiers
  1b. List all SCSI (Clusterware) devices
  1c. Obtain Clusterware device unique SCSI identifiers
  2. Configure Multipath for Persistent Naming of Clusterware Devices
  2a. Configure Multipathing
  2b. Verify Multipath Devices
  3. Script Multipath Device Ownership and Permissions
  4. Test Multipath Device Ownership and Permission Script
  5. Install Oracle 11gR1/2 Clusterware
 References

--------------------------------------------------------------------------------

Applies to:
Linux OS - Version Oracle Linux 6.0 to Oracle Linux 6.4 [Release OL6 to OL6U4]
Linux Itanium
Linux x86-64
Linux x86

Goal
This article is intended for Oracle on Linux Database and System Administrators, specifically those intending to install (or migrate to) Oracle Real Application Clusters 11g Release 1/2 (11.1.0/11.2.0) on Red Hat/Oracle Enterprise Linux 6 (EL6) using multipathed devices. The article is not intended as a complete multipath reference.

Examples were taken from a working system with the following configuration, but should work for later versions:

Oracle Linux Server release 6.3 -Kernel 2.6.39-200.24.1.el6uek.x86_64
Oracle Clusterware 11g Release 1 (11.1.0)
Shared storage for Clusterware files served via iSCSI
From version 11gR1 (11.1.0), Oracle Clusterware no longer requires the use of raw devices as a means to access block devices for OCR and Voting disks - block devices may now be accessed directly.

Solution
Udev and Device Name Persistency
Unlike devlabel in the 2.4 kernel, udev (the 2.6 kernel device file naming scheme) dynamically creates device file names at boot time. However, this can give rise to the possibility that device file names may change - a device that may have once been named /dev/sdd say, may be renamed as /dev/sdf after reboot. Without specific configuration, if udev is left to dynamically name devices, the potential exists for devices referred to, or inadvertently accessed by, their arbitrary kernel-assigned name (e.g. Oracle Clusterware files; Cluster Registry, Voting disks, etc.) to fail to start or become corrupt.

Multipath and Udev
The creation of multipath devices is performed through the use of device-mapper-multipath, which although drawing upon device-mapper, is not completely integrated into the udev naming scheme. Significant modification of default udev rules is required to manipulate multipath device naming, therefore introduces potential supportability issues. Therefore, other means are required to configure device naming persistency and user-defined naming of multipath devices.

Configuring non-raw multipath devices for Oracle Clusterware 11g (11.1.0/11,2,0) on RHEL6/OL6
The following procedure outlines the steps necessary to configure persistent multipath devices with user-defined names in preparation for installation of Oracle Clusterware 11gR1/2(11.1.0/11.2.0).

Assumptions
The following procedure is assumed to have been completed:

Clusterware devices have been created on supported shared storage
Clusterware devices have been appropriately sized according to Oracle11g Release 1/2 (11.1.0/11.2.0) RAC documentation
Clusterware devices have been partitioned
All cluster nodes have multipath access to shared devices
Cluster nodes are configured to satisfy Oracle Universal Installer (OUI) requirements
Unless otherwise stated, all steps should be performed on each cluster node and as a privileged user.

1. Use scsi_id to  Return Unique Device Identifiers
For example:

#/lib/udev/scsi_id --whitelisted --device=/dev/sdd
3600144f0adf5cc4c0000514a72c10003
 

1b. List all SCSI (Clusterware) devices
Clusterware devices must be visible and accessible to all cluster nodes. Typically, cluster node operating systems need to be updated in order to see newly provisioned (or modified) devices on shared storage i.e. use '/sbin/partprobe ' or '/sbin/sfdisk -r ', etc., or simply reboot. Resolve any issues preventing cluster nodes from correctly seeing or accessing Clusterware devices before proceeding.

Run fdisk(8) and/or 'cat /proc/partitions' commands to ensure Clusterware devices are visible, for example:

#cat /proc/partitions
major minor  #blocks  name
   8        0  488386584 sda
   8        1     512000 sda1
   8        2  487873536 sda2
 252        0   52428800 dm-0
 252        1    8077312 dm-1
 252        2  427364352 dm-2
   8       16   20971520 sdb
   8       32   20971520 sdc
   8       48   20971520 sdd
   8       64   20971520 sde
 252        3   20971520 dm-3
 252        4   20971520 dm-4

In the output above, although perhaps not entirely evident just yet, the kernel has assigned two device file names per multipathed device i.e. devices /dev/sdc and /dev/sde both refer to the same device/LUN on shared storage, as do /dev/sdb and /dev/sdd and so on.

Note, at this point, each cluster node may refer to the would-be Clusterware devices by different device file names - this is expected.

1c. Obtain Clusterware device unique SCSI identifiers
Run  scsi_id(8) command against Clusterware devices from one cluster node to obtain their unique device identifiers. Record the unique SCSI identifiers of Clusterware devices - these are required later (Step 2a.), for example:

#for i in `cat /proc/partitions | awk '{print $4}' |grep sd | grep [a-z]$`; do echo "### $i: `/lib/udev/scsi_id --whitelisted --device=/dev/$i`"; done
### sda: 350014ee104864a44
### sdb: 3600144f0adf5cc4c0000514a72c10003
### sdc: 3600144f0adf5cc4c0000514a72ce0004
### sdd: 3600144f0adf5cc4c0000514a72c10003
### sde: 3600144f0adf5cc4c0000514a72ce0004
From the output above, note that multiple devices share common scsi identifiers. It should now be evident that devices such as /dev/sdc and /dev/sde refer to the same shared storage device (LUN).

Note: Irrespective of which cluster node the scsi_id command is run from, the uuid value returned for a given device (LUN) should always be the same.

2. Configure Multipath for Persistent Naming of Clusterware Devices
The purpose of this step is to provide persistent and meaningful user-defined Clusterware multipath device names - it is provided to ensure correct use of the intended Clusterware multipath devices that could otherwise be confused if solely relying on default multipath-assigned naming (e.g. mpathN/mpathNpN), that may be confused when many devices are involved.

2a. Configure Multipathing
Configure multipathing by modifying multipath configuration file /etc/multipath.conf. Comment and uncomment various stanzas accordingly to include (whitelist) or exclude (blacklist) specific devices/types as candidates for multipathing. Specific devices, such as intended Clusterware devices, should be explicitly whitelisted as multipathing candidates. This can be accomplished by defining dedicated multipath stanzas for each device. Ideally, at a minimum, each device stanza should include the device wwid and alias, for example:

# cat /etc/multipath.conf
...
multipath {
        wwid 3600144f0adf5cc4c0000514a72c10003
        alias voting1
        path_grouping_policy failover
        }
...

Following is a sample multipath.conf file. Modify and configure according to your own environment and preferences, but ensure to include Clusterware device-specific multipath stanzas - substitute wwid values for your own i.e. those returned when running Step 1c. above.

# grep -v ^# /etc/multipath.conf
defaults {
       udev_dir                /dev
       polling_interval        5
       path_grouping_policy    failover
       getuid_callout          "/lib/udev/scsi_id --whitelisted --device=/dev/%n"
       prio                    const
       path_checker            directio
       rr_min_io               1000
       rr_weight               uniform
       failback                manual
       no_path_retry           fail
       user_friendly_names     yes
}
devnode_blacklist {
        devnode "^(ram|raw|loop|fd|md|dm-|sr|scd|st)[0-9]*"
        devnode "^hd[a-z]"
        devnode "^cciss!c[0-9]d[0-9]*"
}
multipaths {
   multipath {
      wwid 3600144f0adf5cc4c0000514a72c10003
      alias voting1
      path_grouping_policy failover
   }
   multipath {
      wwid  3600144f0adf5cc4c0000514a72ce0004
      alias ocr1
      path_grouping_policy failover
   }
}
In the example above, devices with a specific wwid value (per scsi_id(8)) are assigned persistent, user-defined names (aliases) i.e. voting1,  ocr1.

Note: Do not configure user-defined aliases with the name 'mpath'.

2b. Verify Multipath Devices
Once multipathing has been configured and multipathd service (re)started, you should now have multipathed Clusterware devices referable by user-defined names, for example:

# multipath -ll
ocr1 (3600144f0adf5cc4c0000514a72ce0004) dm-4 SUN,ZFS Storage 7120
size=20G features='0' hwhandler='0' wp=rw
|-+- policy='round-robin 0' prio=1 status=active
| `- 7:0:0:18  sdc 8:32 active ready running
`-+- policy='round-robin 0' prio=1 status=enabled
  `- 18:0:0:18 sde 8:64 active ready running
voting1 (3600144f0adf5cc4c0000514a72c10003) dm-3 SUN,ZFS Storage 7120
size=20G features='0' hwhandler='0' wp=rw
|-+- policy='round-robin 0' prio=1 status=active
| `- 7:0:0:17  sdb 8:16 active ready running
`-+- policy='round-robin 0' prio=1 status=enabled
  `- 18:0:0:17 sdd 8:48 active ready running
In fact, various device names are created and used to refer to multipathed devices i.e.:

# dmsetup ls | sort
ocr1    (252:4)
voting1    (252:3)

# ls -l /dev/disk/by-id/
lrwxrwxrwx. 1 root root 10 Mar 21 15:13 dm-name-ocr1 -> ../../dm-4
lrwxrwxrwx. 1 root root 10 Mar 21 15:13 dm-name-voting1 -> ../../dm-3

# ls -l /dev/dm-*
brw-rw----. 1 root disk 252, 3 Mar 21 15:13 /dev/dm-3
brw-rw----. 1 root disk 252, 4 Mar 21 15:13 /dev/dm-4

# ls -l /dev/mapper/
total 0
crw-rw----. 1 root root 10, 236 Mar 18 09:14 control
lrwxrwxrwx. 1 root root       7 Mar 21 15:13 ocr1 -> ../dm-4
lrwxrwxrwx. 1 root root       7 Mar 21 15:13 voting1 -> ../dm-3

The /dev/dm-N devices are used internally by device-mapper-multipath and are non-persistent across reboot, so should not be used. /dev/mapper/ devices are persistent, created sufficiently early during the boot process and makes use of their defined aliases - use only these devices to access and interact with multipathed devices.

3. Script Multipath Device Ownership and Permissions
After verifying proper multipath configuration and access to Clusterware devices, use a custom or existing script such as /etc/rc.local to set multipath device ownership and permissions. For example:

# cat /etc/rc.local

#!/bin/sh
#
# This script will be executed *after* all the other init scripts.
# You can put your own initialization stuff in here if you don't
# want to do the full Sys V style init stuff.

touch /var/lock/subsys/local

# OCR disks 11gR1
chown root:oinstall /dev/mapper/ocr*
chmod 0640 /dev/mapper/ocr*
# Voting disks 11gR1
chown oracle:oinstall /dev/mapper/voting*
chmod 0640 /dev/mapper/voting*
 

4. Test Multipath Device Ownership and Permission Script
Execute the /etc/rc.local script to test the proper setting of ownership and permissions of multipath Clusterware devices. Additionally, reboot the server(s) to further verify proper boot-time assignment of devices, for example:

# /etc/rc.local
# ls -l /dev/mapper/
brw-r----- 1 root   oinstall 253,  4 Jul  5 03:03 ocr1
brw-r----- 1 root   oinstall 253,  6 Jul  5 03:03 ocr1p1
brw-r----- 1 oracle oinstall 253,  0 Jul  5 03:03 voting1
brw-r----- 1 oracle oinstall 253,  3 Jul  5 03:03 voting1p1
5. Install Oracle 11gR1/2 Clusterware
Proceed to install Oracle 11g/R1/2 Clusterware, ensuring to specify the appropriate multipath devices (/dev/mapper/) for OCR and Voting disks when prompted. Upon completion of the installation, the Clusterware should be up and running, making use of multipathed block devices directly i.e.:

# ocrcheck
Status of Oracle Cluster Registry is as follows :
        Version                  :          2
        Total space (kbytes)     :    xxxxx
        Used space (kbytes)      :      xxx
        Available space (kbytes) :    xxxxxx
        ID                       : 1752979968
        Device/File Name         : /dev/mapper/ocr1p1
                                    Device/File integrity check succeeded
        Cluster registry integrity check succeeded

# crsctl query css votedisk
 0.     0    /dev/mapper/voting1p1
Located 1 voting disk(s).

# crsctl check crs
Cluster Synchronization Services appears healthy
Cluster Ready Services appears healthy
Event Manager appears healthy

來自 “ ITPUB部落格 ” ，連結：http://blog.itpub.net/24996904/viewspace-1063072/，如需轉載，請註明出處，否則將追究法律責任。