This guide explains how to set up software RAID1 on an already running RHEL 4 system. The GRUB bootloader will be configured in such a way that the system will still be able to boot if one of the hard drives fails (no matter which one).

[@more@]

Introduce my environment:

In my test environment I'm using a RHEL 4 U 5 system with two hard drives, /dev/sda and /dev/sdb which are identical in size. /dev/sdb is currently unused, and /dev/sda has the following partitions:

/dev/sda1: /boot partition, ext3;

/dev/sda2: swap;

/dev/sda3: / partition, ext3

[root@server1 ~]# df –h

Filesystem Size Used Avail Use% Mounted on

/dev/sda3 4.3G 2.4G 1.8G 58% /

/dev/sda1 99M 12M 83M 13% /boot

tmpfs 185M 0 185M 0% /dev/shm

[root@server1 ~]# fdisk -l

Disk /dev/sda: 5368 MB, 5368709120 bytes

255 heads, 63 sectors/track, 652 cylinders

Units = cylinders of 16065 * 512 = 8225280 bytes

Disk identifier: 0x0007b217

Device Boot Start End Blocks Id System

/dev/sda1 * 1 13 104391 83 Linux

/dev/sda2 14 77 514080 82 Linux swap / Solaris

/dev/sda3 78 652 4618687+ 83 Linux

Disk /dev/sdb: 5368 MB, 5368709120 bytes

255 heads, 63 sectors/track, 652 cylinders

Units = cylinders of 16065 * 512 = 8225280 bytes

Disk identifier: 0x00000000

Disk /dev/sdb doesn't contain a valid partition table

2 Preparing /dev/sdb

2.1 Copy the partition table from /dev/sda to /dev/sdb so that both disks have exactly the same layout:

[root@server1 ~]# sfdisk –d /dev/sda | sfdisk /dev/sdb

2.2 Next we must change the partition type of our three partitions on /dev/sdb to Linux raid autodetect:

[root@server1 ~]# fdisk /dev/sdb
Command (m for help): Partition number (1-4): Hex code (type L to list codes): Changed system type of partition 1 to fd (Linux raid autodetect)Command (m for help): Partition number (1-4): Hex code (type L to list codes): Changed system type of partition 2 to fd (Linux raid autodetect)
Command (m for help): Partition number (1-4): Hex code (type L to list codes): Changed system type of partition 3 to fd (Linux raid autodetect)

Command (m for help):
The partition table has been altered!

Calling ioctl() to re-read partition table.
Syncing disks.
[root@server1 ~]#

2.3 To make sure that there are no remains from previous RAID installations on /dev/sdb

[root@server1 ~]# mdadm --zero-superblock /dev/sdb1
[root@server1 ~]# mdadm --zero-superblock /dev/sdb2
[root@server1 ~]# mdadm --zero-superblock /dev/sdb3

3 Creating Our RAID Arrays

[root@server1 ~]# mdadm --create /dev/md0 --level=1 --raid-disks=2 missing /dev/sdb1
[root@server1 ~]# mdadm --create /dev/md1 --level=1 --raid-disks=2 missing /dev/sdb2
[root@server1 ~]# mdadm --create /dev/md2 --level=1 --raid-disks=2 missing /dev/sdb3

3.1 Because the system is currently running on /dev/sda, therefore we use the placeholder missing in the above three commands.

[root@server1 ~]# cat /proc/mdstat
Personalities : [linear] [multipath] [raid0] [raid1] [raid6] [raid5] [raid4] [raid10]
md2 : active raid1 sdb3[1]
4618560 blocks [2/1] [_U]

md1 : active raid1 sdb2[1]
513984 blocks [2/1] [_U]

md0 : active raid1 sdb1[1]
104320 blocks [2/1] [_U]

unused devices:
[root@server1 ~]#

3.2 Next we create filesystems on our RAID arrays (ext3 on /dev/md0 and /dev/md2 and swap on /dev/md1):

[root@server1 ~]# mkfs.ext3 /dev/md0
[root@server1 ~]# mkswap /dev/md1
[root@server1 ~]# mkfs.ext3 /dev/md2

3.3 Next we create /etc/mdadm.conf as follows:

[root@server1 ~]# mdadm --detail --scan > /etc/mdadm.conf

[root@server1 ~]# cat /etc/mdadm.conf

ARRAY /dev/md0 level=raid1 num-devices=2 UUID=2848a3f5:cd1c26b6:e762ed83:696752f9

ARRAY /dev/md1 level=raid1 num-devices=2 UUID=8a004bac:92261691:227767de:4adf6592

ARRAY /dev/md2 level=raid1 num-devices=2 UUID=939f1c71:be9c10fd:d9e5f8c6:a46bcd49

4 Adjusting The System To RAID1

4.1 Now let's mount /dev/md0 and /dev/md2 (we don't need to mount the swap array /dev/md1):

[root@server1 ~]# mkdir /mnt/md0[root@server1 ~]# mkdir /mnt/md2[root@server1 ~]# mount /dev/md0 /mnt/md0[root@server1 ~]# mount /dev/md2 /mnt/md2

4.2 Now we copy the contents of /dev/sda1 and /dev/sda3 to /dev/md0 and /dev/md2 (which are mounted on /mnt/md0 and /mnt/md2):

[root@server1 ~]# cp -dpRx / /mnt/md2[root@server1 ~]# cd /boot[root@server1 ~]# cp -dpRx . /mnt/md0
複製檔案建議放在修改完grub.conf之後進行，否則會不同步，後面會有解釋

4.3 Next we modify /etc/fstab. Replace LABEL=/boot with /dev/md0, LABEL=SWAP-sda2 with /dev/md1, and LABEL=/ with /dev/md2 so that the file looks as follows:

[root@server1 ~]# cat /etc/fstab

/dev/md2 / ext3 defaults 1 1
/dev/md0 /boot ext3 defaults 1 2
tmpfs /dev/shm tmpfs defaults 0 0
devpts /dev/pts devpts gid=5,mode=620 0 0
sysfs /sys sysfs defaults 0 0
proc /proc proc defaults 0 0
/dev/md1 swap swap defaults 0 0

4.4 Next replace LABEL=/boot with /dev/md0 and LABEL=/ with /dev/md2 in /etc/mtab

[root@server1 ~]# cat /etc/mtab
/dev/md2 / ext3 rw 0 0
proc /proc proc rw 0 0
sysfs /sys sysfs rw 0 0
devpts /dev/pts devpts rw,gid=5,mode=620 0 0
/dev/md0 /boot ext3 rw 0 0
tmpfs /dev/shm tmpfs rw 0 0
none /proc/sys/fs/binfmt_misc binfmt_misc rw 0 0
sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw 0 0

4.5 Now up to the GRUB boot loader. Open /boot/grub/grub.conf and add fallback=1 right after default=0, This makes that if the first kernel (counting starts with 0, so the first kernel is 0) fails to boot, kernel #2 will be booted.

[root@server1 ~]# vi /boot/grub/grub.conf

[...]

default=0

fallback=1

[...]

4.6 In the same file, go to the bottom where you should find some kernel stanzas. Copy the first of them and paste the stanza before the first existing stanza; replace root=LABEL=/ with root=/dev/md2 and root (hd0,0) with root (hd1,0).The whole file should look something like this:

[root@server1 ~]# cat /boot/grub/grub.conf

# grub.conf generated by anaconda

#

# Note that you do not have to rerun grub after making changes to this file

# NOTICE: You have a /boot partition. This means that

# all kernel and initrd paths are relative to /boot/, eg.

# root (hd0,0)

# kernel /vmlinuz-version ro root=/dev/md2

# initrd /initrd-version.img

#boot=/dev/sda1

default=0

fallback=1

timeout=5

splashimage=(hd0,0)/grub/splash.xpm.gz

hiddenmenu

title Red Hat Enterprise Linux AS (2.6.9-42.ELsmp)

root (hd1,0)

kernel /vmlinuz-2.6.9-42.ELsmp ro root=/dev/md2 rhgb quiet

initrd /initrd-2.6.9-42.ELsmp.img

title Red Hat Enterprise Linux AS (2.6.9-42.ELsmp)

root (hd0,0)

kernel /vmlinuz-2.6.9-42.ELsmp ro root=LABEL=/ rhgb quiet

initrd /initrd-2.6.9-42.ELsmp.img

4.7 Next we adjust the ramdisk to the new situation:

[root@server1 ~]# mv /boot/initrd-`uname -r`.img /boot/initrd-`uname -r`.img_orig

[root@server1 ~]# mkinitrd /boot/initrd-`uname -r`.img `uname -r`

--這一步驟很重要，如果沒有做的話用第二塊磁碟啟動時會提示“could not find filesystem '/dev/root'的錯誤訊息

5 Preparing GRUB

5.1 Install the GRUB bootloader on the second hard drive /dev/sdb:

[root@server1 ~]# grub

grub> root (hd0,0)
Filesystem type is ext2fs, partition type 0x83

grub> setup (hd0)
Checking if "/boot/grub/stage1" exists... no
Checking if "/grub/stage1" exists... yes
Checking if "/grub/stage2" exists... yes
Checking if "/grub/e2fs_stage1_5" exists... yes
Running "embed /grub/e2fs_stage1_5 (hd0)"... 16 sectors are embedded.
succeeded
Running "install /grub/stage1 (hd0) (hd0)1+16 p (hd0,0)/grub/stage2 /grub/grub.conf"... succeeded
Done.

grub> root (hd1,0)
Filesystem type is ext2fs, partition type 0xfd

grub> setup (hd1)
Checking if "/boot/grub/stage1" exists... no
Checking if "/grub/stage1" exists... yes
Checking if "/grub/stage2" exists... yes
Checking if "/grub/e2fs_stage1_5" exists... yes
Running "embed /grub/e2fs_stage1_5 (hd1)"... 16 sectors are embedded.
succeeded
Running "install /grub/stage1 (hd1) (hd1)1+16 p (hd1,0)/grub/stage2 /grub/grub.conf"... succeeded
Done.

grub> quit

[root@server1 ~]# reboot

Now, back on the normal shell, reboot the system and hope that it boots ok from our RAID arrays:
這裡我在伺服器上測試時出現了問題
grub.conf的設定是用root=/dev/md2來啟動系統，啟動後發現mount的磁碟還是/dev/sda1 /dev/sda2...，並不是/dev/md0 /dev/md1.
原因是我們在修改/etc/fstab /etc/mtab的之前是把之前的/etc/fstab /etc/mtab檔案整個複製到/sdb磁碟的，所以在修改這些檔案的時候並沒有更新到sdb的磁碟。
在用sdb來啟動的時候自然還是用的舊的/etc/fstab /etc/mtab來啟動的，就會掛載/sda1 /sda2之類的磁碟，而不是/dev/md0 /dev/md1了。

6 Preparing /dev/sda

6.1 If all goes well, we should now find /dev/md0 and /dev/md2 in the output of df -h

root@server1 ~]# df -h
Filesystem Size Used Avail Use% Mounted on
/dev/md2 4.4G 2.4G 1.8G 58% /
/dev/md0 99M 15M 80M 16% /boot
tmpfs 185M 0 185M 0% /dev/shm

6.2 Now we must change the partition types of our three partitions on /dev/sda to Linux raid autodetect as well:

[root@server1 ~]# fdisk /dev/sda

Command (m for help): Partition number (1-4): Hex code (type L to list codes): Changed system type of partition 1 to fd (Linux raid autodetect)

Command (m for help): Partition number (1-4): Hex code (type L to list codes): Changed system type of partition 2 to fd (Linux raid autodetect)

Command (m for help): Partition number (1-4): Hex code (type L to list codes): Changed system type of partition 3 to fd (Linux raid autodetect)

Command (m for help): The partition table has been altered!

Calling ioctl() to re-read partition table.
Syncing disks.

6.3 Now we can add /dev/sda1, /dev/sda2, and /dev/sda3 to the respective RAID arrays:

[root@server1 ~]# mdadm --add /dev/md0 /dev/sda1

[root@server1 ~]# mdadm --add /dev/md1 /dev/sda2

[root@server1 ~]# mdadm --add /dev/md2 /dev/sda3

[root@server1 ~]# cat /proc/mdstat
Personalities : [raid1] [raid6] [raid5] [raid4]
md0 : active raid1 sda1[0] sdb1[1]
104320 blocks [2/2] [UU]

md1 : active raid1 sda2[0] sdb2[1]
513984 blocks [2/2] [UU]

md2 : active raid1 sda3[2] sdb3[1]
4618560 blocks [2/1] [_U]
[=====>...............] recovery = 29.9% (1384256/4618560) finish=2.3min speed=22626K/sec

unused devices:

6.4 Then adjust /etc/mdadm.conf to the new situation:

[root@server1 ~]# mdadm --detail --scan > /etc/mdadm.conf

7 Preparing GRUB again

7.1 Now we must modify /boot/grub/grub.conf again. Right now it is configured to boot from /dev/sdb (hd1,0), we still want the system to be able to boot from /dev/sdb in case /dev/sdb fails. Therefore we copy the first kernel stanza (which contains hd1), paste it below and replace hd1 with hd0. so that it looks as follows:

[root@server1 ~]# cat /boot/grub/grub.conf

# grub.conf generated by anaconda

#

# Note that you do not have to rerun grub after making changes to this file

# NOTICE: You have a /boot partition. This means that

# all kernel and initrd paths are relative to /boot/, eg.

# root (hd0,0)

# kernel /vmlinuz-version ro root=/dev/md2

# initrd /initrd-version.img

#boot=/dev/sda1

default=0

fallback=1

timeout=5

splashimage=(hd0,0)/grub/splash.xpm.gz

hiddenmenu

title Red Hat Enterprise Linux AS (2.6.9-42.ELsmp)

root (hd1,0)

kernel /vmlinuz-2.6.9-42.ELsmp ro root=/dev/md2 rhgb quiet

initrd /initrd-2.6.9-42.ELsmp.img

title Red Hat Enterprise Linux AS (2.6.9-42.ELsmp)

root (hd0,0)

kernel /vmlinuz-2.6.9-42.ELsmp ro root=/dev/md2 rhgb quiet

initrd /initrd-2.6.9-42.ELsmp.img

7.2 Afterwards, update the ramdisk:

[root@server1 ~]#mv /boot/initrd-`uname -r`.img /boot/initrd-`uname -r`.img_orig2

[root@server1 ~]#mkinitrd /boot/initrd-`uname -r`.img `uname -r`

7.3 ... and reboot the system:

[root@server1 ~]# reboot

That's it – now I have successfully set up software RAID1 on running RHEL4 system!

8 Testing

8.1 Now let's simulate a hard drive failure. It doesn't matter if you select /dev/sda or /dev/sdb here. In this example I assume that /dev/sdb has failed.

To simulate the hard drive failure, you can either shut down the system and remove /dev/sdb from the system, or you (soft-)remove it like this:

[root@server1 ~]# mdadm --manage /dev/md0 --fail /dev/sdb1

[root@server1 ~]# mdadm --manage /dev/md1 --fail /dev/sdb2

[root@server1 ~]# mdadm --manage /dev/md2 --fail /dev/sdb3

[root@server1 ~]# mdadm --manage /dev/md0 --remove /dev/sdb1

[root@server1 ~]# mdadm --manage /dev/md1 --remove /dev/sdb2

[root@server1 ~]# mdadm --manage /dev/md2 --remove /dev/sdb3

After reboot system and put in a new /dev/sdb drive, file /proc/mdstat may be like below:

[root@server1 ~]# cat /proc/mdstat
Personalities : [raid1] [raid6] [raid5] [raid4]
md0 : active raid1 sda1[0]
104320 blocks [2/1] [U_]

md1 : active raid1 sda2[0]
513984 blocks [2/1] [U_]

md2 : active raid1 sda3[0]
4618560 blocks [2/1] [U_]

unused devices:
[root@server1 ~]#

Now we copy the partition table of /dev/sda to /dev/sdb:

[root@server1 ~]# sfdisk -d /dev/sda | sfdisk /dev/sdb

(If you get an error, you can try the --force option: sfdisk -d /dev/sda | sfdisk --force /dev/sdb)

Afterwards we remove any remains of a previous RAID array from /dev/sdb...

[root@server1 ~]# mdadm --zero-superblock /dev/sdb1

[root@server1 ~]# mdadm --zero-superblock /dev/sdb2

[root@server1 ~]# mdadm --zero-superblock /dev/sdb3

... and add /dev/sdb to the RAID array:

[root@server1 ~]# mdadm -a /dev/md0 /dev/sdb1

[root@server1 ~]# mdadm -a /dev/md1 /dev/sdb2

[root@server1 ~]# mdadm -a /dev/md2 /dev/sdb3

Wait until the synchronization has finished:

[root@server1 ~]# cat /proc/mdstat
Personalities : [raid1] [raid6] [raid5] [raid4]
md0 : active raid1 sdb1[1] sda1[0]
104320 blocks [2/2] [UU]

md1 : active raid1 sdb2[1] sda2[0]
513984 blocks [2/2] [UU]

md2 : active raid1 sdb3[1] sda3[0]
4618560 blocks [2/2] [UU]

unused devices:

Then run

[root@server1 ~]# Grub

and install the bootloader on both hard disks:

grub>root (hd0,0)
grub>setup (hd0)
grub>root (hd1,0)
grub>setup (hd1)
grub>quit

That's it. You've just replaced a failed hard drive in your RAID1 array.