Applies to:

Linux OS - Version Oracle Linux 6.0 to Oracle Linux 6.4 [Release OL6 to OL6U4]
Linux x86-64
Linux x86

Goal

This article is intended for Oracle on Linux System Administrators who are using multipathed devices. The article is not intended as a complete multipath reference.

Examples were taken from a working system with the following configuration, but should work for later versions:

Oracle Linux Server release 6.3 -Kernel 2.6.39-200.24.1.el6uek.x86_64

Solution

Udev and Device Name Persistency

Unlike devlabel in the 2.4 kernel, udev (the 2.6 kernel device file naming scheme) dynamically creates device file names at boot time. However, this can give rise to the possibility that device file names may change - a device that may have once been named /dev/sdd say, may be renamed as /dev/sdf after reboot. Without specific configuration, if udev is left to dynamically named devices, the potential exists for devices referred to, or inadvertently accessed by, their arbitrary kernel-assigned name (e.g. Oracle Clusterware files; Cluster Registry, Voting disks, etc.) to fail to start or become corrupt.

Multipath and Udev

The creation of multipath devices is performed through the use of device-mapper-multipath, which although drawing upon device-mapper, is not completely integrated into the udev naming scheme. Significant modification of default udev rules is required to manipulate multipath device naming, therefore introduces potential supportability issues. Therefore, other means are required to configure device naming persistency and user-defined naming of multipath devices.

Configuring non-raw multipath devices on RHEL6/OL6

The following procedure outlines the steps necessary to configure persistent multipath devices with user-defined names on rhel6/ol6.

1. Use scsi_id to Return Unique Device Identifiers

For example:

#/lib/udev/scsi_id --whitelisted --device=/dev/sdd
3600144f0adf5cc4c0000514a72c10003

1b. List all SCSI (Clusterware) devices

Clusterware devices must be visible and accessible to all cluster nodes. Typically, cluster node operating systems need to be updated in order to see newly provisioned (or modified) devices on shared storage i.e. use '/sbin/partprobe ' or '/sbin/sfdisk -r ', etc., or simply reboot. Resolve any issues preventing cluster nodes from correctly seeing or accessing Clusterware devices before proceeding.

Run fdisk(8) and/or 'cat /proc/partitions' commands to ensure Clusterware devices are visible, for example:

#cat /proc/partitions
major minor #blocks name
   8        0 488386584 sda
   8        1     512000 sda1
   8        2 487873536 sda2
252        0   52428800 dm-0
252        1    8077312 dm-1
252        2 427364352 dm-2
   8       16   20971520 sdb
   8       32   20971520 sdc
   8       48   20971520 sdd
   8       64   20971520 sde
252        3   20971520 dm-3
252        4   20971520 dm-4

In the output above, although perhaps not entirely evident just yet, the kernel has assigned two device file names per multipathed device i.e. devices /dev/sdc and /dev/sde both refer to the same device/LUN on shared storage, as do /dev/sdb and /dev/sdd and so on.

Note, at this point, each cluster node may refer to the would-be Clusterware devices by different device file names - this is expected.

1c. Obtain Clusterware device unique SCSI identifiers

Run scsi_id(8) command against Clusterware devices from one cluster node to obtain their unique device identifiers. Record the unique SCSI identifiers of Clusterware devices - these are required later (Step 2a.), for example:

#for i in `cat /proc/partitions | awk '{print $4}' |grep sd | grep [a-z]$`; do echo "### $i: `/lib/udev/scsi_id --whitelisted --device=/dev/$i`"; done
### sda: 350014ee104864a44
### sdb: 3600144f0adf5cc4c0000514a72c10003
### sdc: 3600144f0adf5cc4c0000514a72ce0004
### sdd: 3600144f0adf5cc4c0000514a72c10003
### sde: 3600144f0adf5cc4c0000514a72ce0004

From the output above, note that multiple devices share common scsi identifiers. It should now be evident that devices such as /dev/sdc and /dev/sde refer to the same shared storage device (LUN).

Note: Irrespective of which cluster node the scsi_id command is run from, the uuid value returned for a given device (LUN) should always be the same.

2. Configure Multipath for Persistent Naming of Clusterware Devices

The purpose of this step is to provide persistent and meaningful user-defined Clusterware multipath device names - it is provided to ensure correct use of the intended Clusterware multipath devices that could otherwise be confused if solely relying on default multipath-assigned naming (e.g. mpathN/mpathNpN), that may be confused when many devices are involved.

2a. Configure Multipathing

Configure multipathing by modifying multipath configuration file /etc/multipath.conf. Comment and uncomment various stanzas accordingly to include (whitelist) or exclude (blacklist) specific devices/types as candidates for multipathing. Specific devices, such as intended Clusterware devices, should be explicitly whitelisted as multipathing candidates. This can be accomplished by defining dedicated multipath stanzas for each device. Ideally, at a minimum, each device stanza should include the device wwid and alias, for example:

# cat /etc/multipath.conf
...
multipath {
        wwid 3600144f0adf5cc4c0000514a72c10003
        alias voting1
        path_grouping_policy failover
        }
...

Following is a sample multipath.conf file. Modify and configure according to your own environment and preferences, but ensure to include Clusterware device-specific multipath stanzas - substitute wwid values for your own i.e. those returned when running Step 1c. above.

# grep -v ^# /etc/multipath.conf
defaults {
       udev_dir                /dev
       polling_interval        5
       path_grouping_policy    failover
       getuid_callout          "/lib/udev/scsi_id --whitelisted --device=/dev/%n"
       prio                    const
       path_checker            directio
       rr_min_io               1000
       rr_weight               uniform
       failback                manual
       no_path_retry           fail
       user_friendly_names     yes
}
devnode_blacklist {
        devnode "^(ram|raw|loop|fd|md|dm-|sr|scd|st)[0-9]*"
        devnode "^hd[a-z]"
        devnode "^cciss!c[0-9]d[0-9]*"
}
multipaths {
   multipath {
wwid 3600144f0adf5cc4c0000514a72c10003
alias voting1
      path_grouping_policy failover
   }
   multipath {
      wwid 3600144f0adf5cc4c0000514a72ce0004
      alias ocr1
      path_grouping_policy failover
   }
}

In the example above, devices with a specific wwid value (per scsi_id(8)) are assigned persistent, user-defined names (aliases) i.e. voting1, ocr1.

Note: Do not configure user-defined aliases with the name 'mpath'.

2b. Verify Multipath Devices

Once multipathing has been configured and multipathd service is (re)started, there should now be multipathed Clusterware devices referable by user-defined names, for example:

# multipath -ll
ocr1 (3600144f0adf5cc4c0000514a72ce0004) dm-4 SUN,ZFS Storage 7120
size=20G features='0' hwhandler='0' wp=rw
|-+- policy='round-robin 0' prio=1 status=active
| `- 7:0:0:18 sdc 8:32 active ready running
`-+- policy='round-robin 0' prio=1 status=enabled
`- 18:0:0:18 sde 8:64 active ready running
voting1 (3600144f0adf5cc4c0000514a72c10003) dm-3 SUN,ZFS Storage 7120
size=20G features='0' hwhandler='0' wp=rw
|-+- policy='round-robin 0' prio=1 status=active
| `- 7:0:0:17 sdb 8:16 active ready running
`-+- policy='round-robin 0' prio=1 status=enabled
`- 18:0:0:17 sdd 8:48 active ready running

In fact, various device names are created and used to refer to multipathed devices i.e.:

# dmsetup ls | sort
ocr1   (252:4)
voting1   (252:3)

# ls -l /dev/disk/by-id/
lrwxrwxrwx. 1 root root 10 Mar 21 15:13 dm-name-ocr1 -> ../../dm-4
lrwxrwxrwx. 1 root root 10 Mar 21 15:13 dm-name-voting1 -> ../../dm-3

# ls -l /dev/dm-*
brw-rw----. 1 root disk 252, 3 Mar 21 15:13 /dev/dm-3
brw-rw----. 1 root disk 252, 4 Mar 21 15:13 /dev/dm-4

# ls -l /dev/mapper/
total 0
crw-rw----. 1 root root 10, 236 Mar 18 09:14 control
lrwxrwxrwx. 1 root root       7 Mar 21 15:13 ocr1 -> ../dm-4
lrwxrwxrwx. 1 root root       7 Mar 21 15:13 voting1 -> ../dm-3

The /dev/dm-N devices are used internally by device-mapper-multipath and are non-persistent across reboot, so should not be used. /dev/mapper/ devices are persistent, created sufficiently early during the boot process and makes use of their defined aliases - use only these devices to access and interact with multipathed devices.