Configuring non-raw multipath devices for Oracle Clusterware 11g (11.1.0) on RHEL5/OEL5

Subject: Configuring non-raw multipath devices for Oracle Clusterware 11g (11.1.0) on RHEL5/OEL5
Doc ID: 605828.1 Type: HOWTO
Modified Date : 04-MAR-2009 Status: PUBLISHED

In this Document
Goal
Solution
Udev and Device Name Persistency
Multipath and Udev
Configuring non-raw multipath devices for Oracle Clusterware 11g (11.1.0) on RHEL5/OEL5
Assumptions
1. Configure SCSI_ID to Return Unique Device Identifiers
1a. Whitelist SCSI devices
1b. List all SCSI (Clusterware) devices
1c. Obtain Clusterware device unique SCSI identifiers
2. Configure Multipath for Persistent Naming of Clusterware Devices
2a. Configure Multipathing
2b. Verify Multipath Devices
3. Script Multipath Device Ownership and Permissions
4. Test Multipath Device Ownership and Permission Script
5. Install Oracle 11g Clusterware
References



--------------------------------------------------------------------------------



Applies to:
Linux Kernel - Version: 2.6.18
Linux Itanium
Linux x86-64

Goal
This article is intended for Oracle on Linux Database and System Administrators, specifically those intending to install (or migrate to) Oracle Real Application Clusters 11g Release 1 (11.1.0) on Red Hat/Oracle Enterprise Linux 5 (RHEL5/OEL5) using multipathed devices. The article is not intended to be a complete multipath reference.

Examples were taken from a working system with the following configuration:

Enterprise Linux 5 (GA) - 2.6.18-8.el5
Oracle Clusterware 11g Release 1 (11.1.0)
Shared storage for Clusterware files served via iSCSI
Oracle Clusterware 11g R1 (11.1.0) no longer requires the use of raw devices as a means to access block devices that are to be used for OCR and Voting disks - block devices may now be accessed directly. Raw devices may still optionally be used with Oracle11g Clusterware if preferred, in which case refer to Note 465001.1 and Note 564580.1 that describe the configuration of raw devices to singlepath and multipath Clusterware devices.


Solution
Udev and Device Name Persistency

Unlike devlabel in the 2.4 kernel, udev (the 2.6 kernel device file naming scheme) dynamically creates device file names at boot time. However, this can give rise to the possibility that device file names may change - a device that may have once been named /dev/sdd say, may be renamed as /dev/sdf after reboot. Without specific configuration, if udev is left to dynamically name devices, the potential exists for devices referred to, or inadvertently accessed by, their arbitrary kernel-assigned name (e.g. Oracle Clusterware files; Cluster Registry, Voting disks, etc.) to fail to start or become corrupt.


Multipath and Udev
The creation of multipath devices is performed through the use of device-mapper-multipath, which although drawing upon device-mapper, is not completely integrated into the udev naming scheme. Significant modification of default udev rules is required to manipulate multipath device naming, therefore introduces potential supportability issues. Therefore, other means are required to configure device naming persistency and user-defined naming of multipath devices.

Configuring non-raw multipath devices for Oracle Clusterware 11g (11.1.0) on RHEL5/OEL5
The following procedure outlines the steps necessary to configure persistent multipath devices with user-defined names in preparation for installation of Oracle Clusterware 11g (11.1.0). The procedure may also be used as a basis for configuring multipath devices on RHEL4/OEL4 (Update 2 or higher) with little modification.

Assumptions
The following procedure assumes the following to have occured:

Clusterware devices have been created on supported shared storage
Clusterware devices have been appropriately sized according to Oracle11g Release 1 (11.1.0) RAC documentation
Clusterware devices have been partitioned
All cluster nodes have multipath access to shared devices
Cluster nodes are configured to satisfy Oracle Universal Installer (OUI) requirements
Unless otherwise stated, all steps should be performed on each cluster node and as a privileged user.

1. Configure SCSI_ID to Return Unique Device Identifiers
1a. Whitelist SCSI devices
Before being able to configure udev to explicitly name devices, scsi_id(8) should first be configured to return their device identifiers. Modify the /etc/scsi_id.config file - add. or replace the existing 'option=-b' parameter/value pair (if exists), with 'option=-g', for example:

# grep -v ^# /etc/scsi_id.config
vendor="ATA",options=-p 0x80
options=-g

1b. List all SCSI (Clusterware) devices
Clusterware devices must be visible and accessible to all cluster nodes. Typically, cluster node operating systems need to be updated in order to see newly provisioned (or modified) devices on shared storage i.e. use '/sbin/partprobe ' or '/sbin/sfdisk -r ', etc., or simply reboot. Resolve any issues preventing cluster nodes from correctly seeing or accessing Clusterware devices before proceeding.

Run the fdisk(8) and/or 'cat /proc/partitions' commands to ensure Clusterware devices are visible, for example:

# cat /proc/partitions

major minor #blocks name

8 0 6291456 sda
8 1 5735173 sda1
8 2 554242 sda2
8 16 2097152 sdb
8 17 2096451 sdb1
8 32 2097152 sdc
8 33 2096451 sdc1
8 48 987966 sdd
8 49 987681 sdd1
8 64 987966 sde
8 65 987681 sde1
8 80 987966 sdf
8 81 987681 sdf1
8 96 987966 sdg
8 97 987681 sdg1
8 112 987966 sdh
8 113 987681 sdh1
8 128 987966 sdi
8 129 987681 sdi1
8 144 1004031 sdj
8 145 1003873 sdj1
8 160 1004031 sdk
8 161 1003873 sdk1
8 176 1004031 sdl
8 177 1003873 sdl1
8 192 1004031 sdm
8 193 1003873 sdm1
8 208 1004031 sdn
8 209 1003873 sdn1
8 224 1004031 sdo
8 225 1003873 sdo1

In the output above, although perhaps not entirely evident just yet, the kernel has assigned two device file names per multipathed device i.e. devices /dev/sdd and /dev/sde both refer to the same device/LUN on shared storage, as do /dev/sdf and /dev/sdg and so on.

Note, at this point, each cluster node may refer to the would-be Clusterware devices by different device file names - this is expected.

1c. Obtain Clusterware device unique SCSI identifiers
Run the scsi_id(8) command against Clusterware devices from one cluster node to obtain their unique device identifiers. When running the scsi_id command with the '-s' argument, the device path and name passed should be that relative to sysfs directory /sys/ i.e. /block/ when referring to /sys/block/. Record the unique SCSI identifiers of Clusterware devices - these are required later (Step 2a.), for example:

# for i in `cat /proc/partitions | awk '{print $4}' |grep sd`; do echo "### $i: `scsi_id -g -u -s /block/$i`"; done
...
### sdd: 149455400000000000000000001000000d90200000d000000
### sdd1:
### sde: 149455400000000000000000001000000d90200000d000000
### sde1:
### sdf: 149455400000000000000000001000000de0200000d000000
### sdf1:
### sdg: 149455400000000000000000001000000de0200000d000000
### sdg1:
### sdh: 149455400000000000000000001000000e30200000d000000
### sdh1:
### sdi: 149455400000000000000000001000000e30200000d000000
### sdi1:
### sdj: 149455400000000000000000001000000e80200000d000000
### sdj1:
### sdk: 149455400000000000000000001000000e80200000d000000
### sdk1:
### sdl: 149455400000000000000000001000000ed0200000d000000
### sdl1:
### sdm: 149455400000000000000000001000000ed0200000d000000
### sdm1:
### sdn: 149455400000000000000000001000000ca0200000d000000
### sdn1:
### sdo: 149455400000000000000000001000000ca0200000d000000
### sdo1:


From the output above, note that multiple devices share common scsi identifiers. It should now be evident that devices such as /dev/sdd and /dev/sde refer to the same shared storage device (LUN).

Note: Irrespective of which cluster node the scsi_id command is run from, the uuid value returned for a given device (LUN) should always be the same.

2. Configure Multipath for Persistent Naming of Clusterware Devices
The purpose of this step is to provide persistent and meaningful user-defined Clusterware multipath device names - it is provided to ensure correct use of the intended Clusterware multipath devices that could otherwise be confused if solely relying on default multipath-assigned naming (e.g. mpathN/mpathNpN), especially when many devices are involved.

2a. Configure Multipathing
Configure multipathing by modifying multipath configuration file /etc/multipath.conf. Comment and uncomment various stanzas accordingly to include (whitelist) or exclude (blacklist) specific devices/types as candidates for multipathing. Specific devices, such as our intended Clusterware devices, should be explicitly whitelisted as multipathing candidates. This can be accomplished by defining dedicated multipath stanzas for each device. At a minimum, each device stanza should include the device wwid and an alias, for example:

# cat /etc/multipath.conf
...
multipath {
wwid 149455400000000000000000001000000d90200000d000000
alias voting1
path_grouping_policy failover
}
...

Following is a sample multipath.conf file. Modify your configuration according to your own environment and preferences, but ensuring to include Clusterware device-specific multipath stanzas - substitute wwid values for your own i.e. those returned when running Step 1c. above.
# grep -v ^# /etc/multipath.conf
defaults {
user_friendly_names yes
}
defaults {
udev_dir /dev
polling_interval 10
selector "round-robin 0"
path_grouping_policy failover
getuid_callout "/sbin/scsi_id -g -u -s /block/%n"
prio_callout /bin/true
path_checker readsector0
rr_min_io 100
rr_weight priorities
failback immediate
#no_path_retry fail
user_friendly_name yes
}
devnode_blacklist {
devnode "^(ram|raw|loop|fd|md|dm-|sr|scd|st)[0-9]*"
devnode "^hd[a-z]"
devnode "^cciss!c[0-9]d[0-9]*"
}
multipaths {
multipath {
wwid 149455400000000000000000001000000d90200000d000000
alias voting1
path_grouping_policy failover
}
multipath {
wwid 149455400000000000000000001000000de0200000d000000
alias voting2
path_grouping_policy failover
}
multipath {
wwid 149455400000000000000000001000000e30200000d000000
alias voting3
path_grouping_policy failover
}
multipath {
wwid 149455400000000000000000001000000e80200000d000000
alias ocr1
path_grouping_policy failover
}
multipath {
wwid 149455400000000000000000001000000ed0200000d000000
alias ocr2
path_grouping_policy failover
}
multipath {
wwid 149455400000000000000000001000000ca0200000d000000
alias ocr3
path_grouping_policy failover
}
}

In the example above, devices with specific wwid's (per scsi_id) are assigned persistent, user-defined names (aliases) i.e. voting1, voting2, voting3, ocr1, ocr2 and ocr3.

2b. Verify Multipath Devices
Once multipathing has been configured and multipathd service started, you should now have multipathed Clusterware devices referable by user-defined names, for example:

# multipath -ll

ocr3 (149455400000000000000000001000000ca0200000d000000) dm-9 IET,VIRTUAL-DISK
[size=980M][features=0][hwhandler=0]
_ round-robin 0 [prio=0][active]
_ 1:0:0:11 sdo 8:224 [active][ready]
_ round-robin 0 [prio=0][enabled]
_ 1:0:0:10 sdn 8:208 [active][ready]
ocr2 (149455400000000000000000001000000ed0200000d000000) dm-8 IET,VIRTUAL-DISK
[size=980M][features=0][hwhandler=0]
_ round-robin 0 [prio=0][active]
_ 1:0:0:9 sdm 8:192 [active][ready]
_ round-robin 0 [prio=0][enabled]
_ 1:0:0:8 sdl 8:176 [active][ready]
ocr1 (149455400000000000000000001000000e80200000d000000) dm-4 IET,VIRTUAL-DISK
[size=980M][features=0][hwhandler=0]
_ round-robin 0 [prio=0][active]
_ 1:0:0:6 sdj 8:144 [active][ready]
_ round-robin 0 [prio=0][enabled]
_ 1:0:0:7 sdk 8:160 [active][ready]
voting3 (149455400000000000000000001000000e30200000d000000) dm-1 IET,VIRTUAL-DISK
[size=965M][features=0][hwhandler=0]
_ round-robin 0 [prio=0][active]
_ 1:0:0:5 sdi 8:128 [active][ready]
_ round-robin 0 [prio=0][enabled]
_ 1:0:0:4 sdh 8:112 [active][ready]
voting2 (149455400000000000000000001000000de0200000d000000) dm-2 IET,VIRTUAL-DISK
[size=965M][features=0][hwhandler=0]
_ round-robin 0 [prio=0][active]
_ 1:0:0:3 sdg 8:96 [active][ready]
_ round-robin 0 [prio=0][enabled]
_ 1:0:0:2 sdf 8:80 [active][ready]
voting1 (149455400000000000000000001000000d90200000d000000) dm-0 IET,VIRTUAL-DISK
[size=965M][features=0][hwhandler=0]
_ round-robin 0 [prio=0][active]
_ 1:0:0:1 sde 8:64 [active][ready]
_ round-robin 0 [prio=0][enabled]
_ 1:0:0:0 sdd 8:48 [active][ready]


In fact, various device names are created and used to refer to multipathed devices i.e.:
# # dmsetup ls | sort
ocr1 (253, 4)
ocr1p1 (253, 6)
ocr2 (253, 8)
ocr2p1 (253, 10)
ocr3 (253, 9)
ocr3p1 (253, 11)
voting1 (253, 0)
voting1p1 (253, 3)
voting2 (253, 2)
voting2p1 (253, 7)
voting3 (253, 1)
voting3p1 (253, 5)

# ls -l /dev/disk/by-id/
total 0
lrwxrwxrwx 1 root root 9 Jul 5 03:02 scsi-149455400000000000000000001000000ca0200000d000000 -> ../../sdo
lrwxrwxrwx 1 root root 10 Jul 5 03:02 scsi-149455400000000000000000001000000ca0200000d000000-part1 -> ../../sdn1
lrwxrwxrwx 1 root root 9 Jul 5 03:02 scsi-149455400000000000000000001000000d90200000d000000 -> ../../sdd
lrwxrwxrwx 1 root root 10 Jul 5 03:02 scsi-149455400000000000000000001000000d90200000d000000-part1 -> ../../sdd1
lrwxrwxrwx 1 root root 9 Jul 5 03:02 scsi-149455400000000000000000001000000de0200000d000000 -> ../../sdg
lrwxrwxrwx 1 root root 10 Jul 5 03:02 scsi-149455400000000000000000001000000de0200000d000000-part1 -> ../../sdg1
lrwxrwxrwx 1 root root 9 Jul 5 03:02 scsi-149455400000000000000000001000000e30200000d000000 -> ../../sdi
lrwxrwxrwx 1 root root 10 Jul 5 03:02 scsi-149455400000000000000000001000000e30200000d000000-part1 -> ../../sdi1
lrwxrwxrwx 1 root root 9 Jul 5 03:02 scsi-149455400000000000000000001000000e80200000d000000 -> ../../sdk
lrwxrwxrwx 1 root root 10 Jul 5 03:02 scsi-149455400000000000000000001000000e80200000d000000-part1 -> ../../sdk1
lrwxrwxrwx 1 root root 9 Jul 5 03:02 scsi-149455400000000000000000001000000ed0200000d000000 -> ../../sdm
lrwxrwxrwx 1 root root 10 Jul 5 03:02 scsi-149455400000000000000000001000000ed0200000d000000-part1 -> ../../sdm1

# ls -l /dev/dm-*
brw-rw---- 1 root root 253, 0 Jul 5 03:02 /dev/dm-0
brw-rw---- 1 root root 253, 1 Jul 5 03:02 /dev/dm-1
brw-rw---- 1 root root 253, 10 Jul 5 03:02 /dev/dm-10
brw-rw---- 1 root root 253, 11 Jul 5 03:02 /dev/dm-11
brw-rw---- 1 root root 253, 2 Jul 5 03:02 /dev/dm-2
brw-rw---- 1 root root 253, 3 Jul 5 03:02 /dev/dm-3
brw-rw---- 1 root root 253, 4 Jul 5 03:02 /dev/dm-4
brw-rw---- 1 root root 253, 5 Jul 5 03:02 /dev/dm-5
brw-rw---- 1 root root 253, 6 Jul 5 03:02 /dev/dm-6
brw-rw---- 1 root root 253, 7 Jul 5 03:02 /dev/dm-7
brw-rw---- 1 root root 253, 8 Jul 5 03:02 /dev/dm-8
brw-rw---- 1 root root 253, 9 Jul 5 03:02 /dev/dm-9

# ls -l /dev/mpath/*
lrwxrwxrwx 1 root root 7 Jul 5 03:02 ocr1 -> ../dm-4
lrwxrwxrwx 1 root root 7 Jul 5 03:02 ocr1p1 -> ../dm-6
lrwxrwxrwx 1 root root 7 Jul 5 03:02 ocr2 -> ../dm-8
lrwxrwxrwx 1 root root 7 Jul 5 03:02 ocr2p1 -> ../dm-10
lrwxrwxrwx 1 root root 7 Jul 5 03:02 ocr3 -> ../dm-9
lrwxrwxrwx 1 root root 7 Jul 5 03:02 ocr3p1 -> ../dm-11
lrwxrwxrwx 1 root root 7 Jul 5 03:02 voting1 -> ../dm-0
lrwxrwxrwx 1 root root 7 Jul 5 03:02 voting1p1 -> ../dm-3
lrwxrwxrwx 1 root root 7 Jul 5 03:02 voting2 -> ../dm-2
lrwxrwxrwx 1 root root 7 Jul 5 03:02 voting2p1 -> ../dm-7
lrwxrwxrwx 1 root root 7 Jul 5 03:02 voting3 -> ../dm-1
lrwxrwxrwx 1 root root 7 Jul 5 03:02 voting3p1 -> ../dm-5

# ls -l /dev/mapper/
brw-r----- 1 root root 253, 4 Jul 5 03:00 ocr1
brw-r----- 1 root root 253, 6 Jul 5 03:00 ocr1p1
brw-r----- 1 root root 253, 8 Jul 5 03:00 ocr2
brw-r----- 1 root root 253, 10 Jul 5 03:00 ocr2p1
brw-r----- 1 root root 253, 9 Jul 5 03:00 ocr3
brw-r----- 1 root root 253, 11 Jul 5 03:00 ocr3p1
brw-r----- 1 root root 253, 0 Jul 5 03:00 voting1
brw-r----- 1 root root 253, 3 Jul 5 03:00 voting1p1
brw-r----- 1 root root 253, 2 Jul 5 03:00 voting2
brw-r----- 1 root root 253, 7 Jul 5 03:00 voting2p1
brw-r----- 1 root root 253, 1 Jul 5 03:00 voting3
brw-r----- 1 root root 253, 5 Jul 5 03:00 voting3p1


The /dev/dm-N devices are used internally by device-mapper-multipath and are non-persistent across reboot, so should not be used. The /dev/mpath/ devices are created for multipath devices to be visible together, however, may not be available during early stages of boot, so, again, should not be used. However, /dev/mapper/ devices are persistent, created sufficiently early during the boot process and makes use of their defined aliases - use these devices only to access and interact with multipathed devices.

3. Script Multipath Device Ownership and Permissions
After verifying proper multipath configuration and access to Clusterware devices, define a script that sets their required ownership and permissions. Given the obsolescence of RHEL4/OEL4's /etc/sysconfig/rawdevices configuration file and rawdevices service, the /etc/rc.local file is used for this purpose. For example:

# cat /etc/rc.local

#!/bin/sh
#
# This script will be executed *after* all the other init scripts.
# You can put your own initialization stuff in here if you don't
# want to do the full Sys V style init stuff.

touch /var/lock/subsys/local

# OCR disks 11gR1
chown root:oinstall /dev/mapper/ocr*
chmod 0640 /dev/mapper/ocr*
# Voting disks 11gR1
chown oracle:oinstall /dev/mapper/voting*
chmod 0640 /dev/mapper/voting*



4. Test Multipath Device Ownership and Permission Script
Execute the /etc/rc.local script to test the proper setting of ownership and permissions of multipath Clusterware devices. Additionally, reboot the server(s) to further verify proper boot-time assignment of devices, for example:

# /etc/rc.local
#

# ls -l /dev/mapper/
brw-r----- 1 root oinstall 253, 4 Jul 5 03:03 ocr1
brw-r----- 1 root oinstall 253, 6 Jul 5 03:03 ocr1p1
brw-r----- 1 root oinstall 253, 8 Jul 5 03:03 ocr2
brw-r----- 1 root oinstall 253, 10 Jul 5 03:03 ocr2p1
brw-r----- 1 root oinstall 253, 9 Jul 5 03:03 ocr3
brw-r----- 1 root oinstall 253, 11 Jul 5 03:03 ocr3p1
brw-r----- 1 oracle oinstall 253, 0 Jul 5 03:03 voting1
brw-r----- 1 oracle oinstall 253, 3 Jul 5 03:03 voting1p1
brw-r----- 1 oracle oinstall 253, 2 Jul 5 03:03 voting2
brw-r----- 1 oracle oinstall 253, 7 Jul 5 03:03 voting2p1
brw-r----- 1 oracle oinstall 253, 1 Jul 5 03:03 voting3
brw-r----- 1 oracle oinstall 253, 5 Jul 5 03:03 voting3p1



5. Install Oracle 11g Clusterware
Proceed to install Oracle 11g Clusterware, ensuring to specify the appropriate multipath devices (/dev/mapper/) for OCR and Voting disks when prompted. Upon completion of the installation, the Clusterware should be up and running, making use of multipathed block devices directly i.e.:

# ocrcheck
Status of Oracle Cluster Registry is as follows :
Version : 2
Total space (kbytes) : 1003604
Used space (kbytes) : 1180
Available space (kbytes) : 1002424
ID : 1752979968
Device/File Name : /dev/mapper/ocr1p1
Device/File integrity check succeeded
Device/File Name : /dev/mapper/ocr2p1
Device/File integrity check succeeded

Cluster registry integrity check succeeded

# crsctl query css votedisk
0. 0 /dev/mapper/voting1p1
1. 0 /dev/mapper/voting2p1
2. 0 /dev/mapper/voting3p1
Located 3 voting disk(s).

# crsctl check crs
Cluster Synchronization Services appears healthy
Cluster Ready Services appears healthy
Event Manager appears healthy



References
Note 465001.1 - Configuring raw devices (singlepath) for Oracle Clusterware 10g Release 2 (10.2.0) on RHEL5/OEL5
Note 564580.1 - Configuring raw devices (multipath) for Oracle Clusterware 10g Release 2 (10.2.0) on RHEL5/OEL5

Keywords
OCR ; RAC ; OUI ; REAL~APPLICATION~CLUSTERS ; CLUSTERWARE ; STORAGE ; RAW ; RAW~DEVICE ;

[@more@]