Here is a transcript of the initial creation of the logical volume from a RAID 5 on the shelf with address 7. I'm using the latest LVM2 and device mapper from the "dm" and "lvm2" directories at "ftp://sources.redhat.com/pub". root@nai ~# for f in `find /proc | grep 'raid.*speed'`; do echo 100000 > $f; done root@nai ~# mdadm -C -l 5 -n 9 -x 1 /dev/md0 /dev/etherd/e7.[0-9] mdadm: array /dev/md0 started. root@nai ~# which pvcreate /opt/device-mapper.1.01.01/lib root@nai ~# echo $LD_LIBRARY_PATH /opt/LVM2.2.01.09/sbin/pvcreate root@nai ~# pvcreate /dev/md0 Physical volume "/dev/md0" successfully created root@nai ~# vgcreate pool0 /dev/md0 Volume group "pool0" successfully created root@nai ~# vgdisplay pool0 --- Volume group --- VG Name pool0 System ID Format lvm2 Metadata Areas 1 Metadata Sequence No 1 VG Access read/write VG Status resizable MAX LV 0 Cur LV 0 Open LV 0 Max PV 0 Cur PV 1 Act PV 1 VG Size 306.77 GB PE Size 4.00 MB Total PE 78533 Alloc PE / Size 0 / 0 Free PE / Size 78533 / 306.77 GB VG UUID Q1K0CQ-SlKz-wvHL-vIe4-ynum-L2rl-i0BpPQ root@nai ~# lvcreate -l 78533 -n vol0 pool0 Logical volume "vol0" created root@nai ~# mkfs -t jfs /dev/pool0/vol0 mkfs.jfs version 1.1.7, 22-Jul-2004 Warning! All data on device /dev/pool0/vol0 will be lost! Continue? (Y/N) y \ Format completed successfully. 321671168 kilobytes total disk space. root@nai ~# mkdir /mnt/vol0 root@nai ~# mount /dev/pool0/vol0 /mnt/vol0 root@nai ~# rsync -a /etc/init.d /mnt/vol0 Now the logical volume, "vol0", has a jfs mounted on /mnt/vol0. Next we're going to add another shelf (with shelf address 5). Usually you'd do this some time later, and the first RAID 5 would have fully initialized, but in my example here, I'm doing all this while the first array is still building. On this Fedora Core 3 system, there's no /dev/md1 file, so we use mdadm's "auto" option to create it as needed. root@nai ~# ls /dev/md1 ls: /dev/md1: No such file or directory root@nai ~# mdadm -C -l 5 -n 10 --auto=md /dev/md1 /dev/etherd/e5.[0-9] mdadm: array /dev/md1 started. root@nai ~# cat /proc/mdstat Personalities : [raid5] md1 : active raid5 etherd/e5.9[10] etherd/e5.8[8] etherd/e5.7[7] etherd/e5.6[6] etherd/e5.5[5] etherd/e5.4[4] etherd/e5.3[3] etherd/e5.2[2] etherd/e5.1[1] etherd/e5.0[0] 361881216 blocks level 5, 64k chunk, algorithm 2 [10/9] [UUUUUUUUU_] [>....................] recovery = 0.2% (90408/40209024) finish=155.1min speed=4305K/sec md0 : active raid5 etherd/e7.8[9] etherd/e7.9[10] etherd/e7.7[7] etherd/e7.6[6] etherd/e7.5[5] etherd/e7.4[4] etherd/e7.3[3] etherd/e7.2[2] etherd/e7.1[1] etherd/e7.0[0] 321672192 blocks level 5, 64k chunk, algorithm 2 [9/8] [UUUUUUUU_] [>....................] recovery = 1.9% (789580/40209024) finish=156.1min speed=4206K/sec unused devices: We make an LVM physical volume out of the new RAID 5 and then add it to the volume group. Then the logical volume is extended to take up the new space. root@nai ~# pvcreate /dev/md1 Physical volume "/dev/md1" successfully created root@nai ~# vgextend pool0 /dev/md1 Volume group "pool0" successfully extended root@nai ~# vgdisplay pool0 --- Volume group --- VG Name pool0 System ID Format lvm2 Metadata Areas 2 Metadata Sequence No 3 VG Access read/write VG Status resizable MAX LV 0 Cur LV 1 Open LV 0 Max PV 0 Cur PV 2 Act PV 2 VG Size 651.88 GB PE Size 4.00 MB Total PE 166882 Alloc PE / Size 78533 / 306.77 GB Free PE / Size 88349 / 345.11 GB VG UUID Q1K0CQ-SlKz-wvHL-vIe4-ynum-L2rl-i0BpPQ root@nai ~# lvextend -l +88349 /dev/pool0/vol0 Extending logical volume vol0 to 651.88 GB Logical volume vol0 successfully resized root@nai ~# The filesystem doesn't know that the block device it's on is bigger, so next we grow the jfs (while it's mounted and usable). This took quite some time for me, since the RAIDs were still initializing. root@nai ~# df -h /mnt/vol0 Filesystem Size Used Avail Use% Mounted on /dev/mapper/pool0-vol0 307G 39M 307G 1% /mnt/vol0 root@nai ~# mount -o remount,resize /mnt/vol0 root@nai ~# df -h /mnt/vol0 Filesystem Size Used Avail Use% Mounted on /dev/mapper/pool0-vol0 652G 82M 652G 1% /mnt/vol0 root@nai ~# To shut down the storage cleanly, shut down each layer from the top layer down. root@nai ~# umount /mnt/vol0 root@nai ~# vgchange -an pool0 0 logical volume(s) in volume group "pool0" now active root@nai ~# mdadm -S /dev/md0 root@nai ~# mdadm -S /dev/md1 Later, to bring the storage back online, make each layer active from the bottom up. root@nai ~# mdadm -A /dev/etherd/e7.[0-9] mdadm: /dev/etherd/e7.0 does not appear to be an md device root@nai ~# mdadm -A /dev/md0 /dev/etherd/e7.[0-9] mdadm: /dev/md0 has been started with 8 drives (out of 9) and 2 spares. root@nai ~# mdadm -A /dev/md1 /dev/etherd/e5.[0-9] mdadm: /dev/md1 has been started with 9 drives (out of 10) and 1 spare. root@nai ~# vgscan Reading all physical volumes. This may take a while... Found volume group "VolGroup00" using metadata type lvm2 Found volume group "pool0" using metadata type lvm2 root@nai ~# vgchange -ay pool0 1 logical volume(s) in volume group "pool0" now active root@nai ~# mount /dev/pool0/vol0 /mnt/vol0 root@nai ~# ls /mnt/vol0/ init.d root@nai ~# Now the only thing left to do is to tell mdadm to use the one hot spare for a faulty disk in either RAID. Below is the text of the mdadm configuration file named "raid5s-mdadm.conf". Once this file is created, mdadm should be run in monitor mode like this: mdadm --monitor --scan --config=raid5s-mdadm.conf # raid5s-mdadm.conf # # see mdadm.conf manpage for syntax and info # # There's a "spare group" called vol0, after the logical # volume name, so that mdadm can use the hot spare for # either RAID 5 that needs it. # DEVICE /dev/etherd/e5.[0-9] DEVICE /dev/etherd/e7.[0-9] ARRAY /dev/md0 devices=/dev/etherd/e7.[0-9] spare-group=vol0 ARRAY /dev/md1 devices=/dev/etherd/e5.[0-9] spare-group=vol0 MAILADDR root # This is normally a program that handles events instead # of just /bin/echo. If you run the mdadm monitor in the # forground, though, using echo allows you to see what events # are occurring. # PROGRAM /bin/echo