Configuring Multipath Devices on RHEL6/OL6 (文件 ID 1538626.1)

In this Document

APPLIES TO:

GOAL

This article is intended for Oracle on Linux System Administrators who are using multipathed devices. The article is not intended as a complete multipath reference.

Examples were taken from a working system with the following configuration, but should work for later versions:

• Oracle Linux Server release 6.3 -Kernel 2.6.39-200.24.1.el6uek.x86_64

SOLUTION

Udev and Device Name Persistency

Unlike devlabel in the 2.4 kernel, udev (the 2.6 kernel device file naming scheme) dynamically creates device file names at boot time. However, this can give rise to the possibility that device file names may change - a device that may have once been named /dev/sdd say, may be renamed as/dev/sdf after reboot. Without specific configuration, if udev is left to dynamically named devices, the potential exists for devices referred to, or inadvertently accessed by, their arbitrary kernel-assigned name (e.g. Oracle Clusterware files; Cluster Registry, Voting disks, etc.) to fail to start or become corrupt.

Multipath and Udev

The creation of multipath devices is performed through the use of device-mapper-multipath, which although drawing upon device-mapper, is not completely integrated into the udev naming scheme. Significant modification of default udev rules is required to manipulate multipath device naming, therefore introduces potential supportability issues. Therefore, other means are required to configure device naming persistency and user-defined naming of multipath devices.

Configuring non-raw multipath devices on RHEL6/OL6

The following procedure outlines the steps necessary to configure persistent multipath devices with user-defined names on rhel6/ol6.

1. Use scsi_id to  Return Unique Device Identifiers

For example:

1b. List all SCSI (Clusterware) devices

Clusterware devices must be visible and accessible to all cluster nodes. Typically, cluster node operating systems need to be updated in order to see newly provisioned (or modified) devices on shared storage i.e. use '/sbin/partprobe ' or '/sbin/sfdisk -r ', etc., or simply reboot. Resolve any issues preventing cluster nodes from correctly seeing or accessing Clusterware devices before proceeding.

Run fdisk(8) and/or 'cat /proc/partitions' commands to ensure Clusterware devices are visible, for example:

In the output above, although perhaps not entirely evident just yet, the kernel has assigned two device file names per multipathed device i.e. devices /dev/sdc and /dev/sde both refer to the same device/LUN on shared storage, as do /dev/sdb and /dev/sdd and so on.

Note, at this point, each cluster node may refer to the would-be Clusterware devices by different device file names - this is expected.

1c. Obtain Clusterware device unique SCSI identifiers

Run  scsi_id(8) command against Clusterware devices from one cluster node to obtain their unique device identifiers. Record the unique SCSI identifiers of Clusterware devices - these are required later (Step 2a.), for example:

From the output above, note that multiple devices share common scsi identifiers. It should now be evident that devices such as /dev/sdc and/dev/sde refer to the same shared storage device (LUN).

Note: Irrespective of which cluster node the scsi_id command is run from, the uuid value returned for a given device (LUN) should always be the same.

2. Configure Multipath for Persistent Naming of Clusterware Devices

The purpose of this step is to provide persistent and meaningful user-defined Clusterware multipath device names - it is provided to ensure correct use of the intended Clusterware multipath devices that could otherwise be confused if solely relying on default multipath-assigned naming (e.g.mpathN/mpathNpN), that may be confused when many devices are involved.

2a. Configure Multipathing

Configure multipathing by modifying multipath configuration file /etc/multipath.conf. Comment and uncomment various stanzas accordingly to include (whitelist) or exclude (blacklist) specific devices/types as candidates for multipathing. Specific devices, such as intended Clusterware devices, should be explicitly whitelisted as multipathing candidates. This can be accomplished by defining dedicated multipath stanzas for each device. Ideally, at a minimum, each device stanza should include the device wwid and alias, for example:

Following is a sample multipath.conf file. Modify and configure according to your own environment and preferences, but ensure to include Clusterware device-specific multipath stanzas - substitute wwid values for your own i.e. those returned when running Step 1c. above.

In the example above, devices with a specific wwid value (per scsi_id(8)) are assigned persistent, user-defined names (aliases) i.e. voting1,  ocr1.

Note: Do not configure user-defined aliases with the name 'mpath'.

2b. Verify Multipath Devices

Once multipathing has been configured and multipathd service is (re)started, there should now be multipathed Clusterware devices referable by user-defined names, for example:

In fact, various device names are created and used to refer to multipathed devices i.e.:

The /dev/dm-N devices are used internally by device-mapper-multipath and are non-persistent across reboot, so should not be used. /dev/mapper/devices are persistent, created sufficiently early during the boot process and makes use of their defined aliases - use only these devices to access and interact with multipathed devices.

REFERENCES