This section contains material that is no longer relevant to a majority of readers. It has been placed in this appendix with minimal editing.
Let us assume we have five AoE targets that are virtual LUNs numbered 0 through 4, exported from a Coraid VS appliance that has been assigned shelf address 0. Let us further assume we want to use these five LUNs to create a level-5 RAID array. Using a text editor, we create a Software RAID configuration file named "/etc/rt". The transcript below shows its contents.
$ cat /etc/rt
raiddev /dev/md0
raid-level 5
nr-raid-disks 5
chunk-size 32
persistent-superblock 1
device /dev/etherd/e0.0
raid-disk 0
device /dev/etherd/e0.1
raid-disk 1
device /dev/etherd/e0.2
raid-disk 2
device /dev/etherd/e0.3
raid-disk 3
device /dev/etherd/e0.4
raid-disk 4
Here is an example for setting up and using the RAID array described
by the above configuration file, /etc/rt.
$ mkraid -c /etc/rt /dev/md0
DESTROYING the contents of /dev/md0 in 5 seconds, Ctrl-C if unsure!
handling MD device /dev/md0
analyzing super-block
disk 0: /dev/etherd/00:00, 19535040kB, raid superblock at 19534976kB
disk 1: /dev/etherd/00:01, 19535040kB, raid superblock at 19534976kB
disk 2: /dev/etherd/00:02, 19535040kB, raid superblock at 19534976kB
disk 3: /dev/etherd/00:03, 19535040kB, raid superblock at 19534976kB
disk 4: /dev/etherd/00:04, 19535040kB, raid superblock at 19534976kB
$
To make an ext3 filesystem on the RAID array and mount it, the following commands can be issued:
$ mkfs.ext3 /dev/md0
... (mkfs output)
$ mount /dev/md0 /mnt/raid
$
The resulting storage is single-fault tolerant. Add hot spares to
make the array even more robust (see the Software RAID documentation
for more information.) Remember that it takes the md driver some time
to initialize a new RAID 5 array. During that time, you can use the
device, but performance is sub-optimal until md finishes. Check
/proc/mdstat for information on the initialization's
progress.
Today, the Linux kernel supports a raid10 personality, and you can create a RAID 10 with one mdadm command. Things used to be more complicated. The section below shows the steps that used to be necessary to create a RAID 10 by first creating several RAID 1 mirrors that could serve as components for the larger RAID 0.
RAID 10 is striping over mirrors. That is, a RAID 0 is created to stripe data over several RAID 1 devices. Each RAID 1 is a mirrored pair of disks. For a given (even) number of disks, a RAID 10 has less capacity and throughput than a RAID 5. Nevertheless, storage experts often prefer RAID 10 for its superior resiliancy to failure, its low re-initialization time, and its low computational overhead.
The first example shows how to create a RAID 10 and a hot spare from eight AoE targets that share shelf address 1. After checking the mdadm manpage, it should be easy for you to create startup and shutdown scripts.
# make-raid10.sh
# create a RAID 10 from shelf 1 to be used with mdadm-aoe.conf
set -xe # shell flags: be verbose, exits on errors
shelf=1
# create the mirrors
mdadm -C /dev/md1 -l 1 -n 2 /dev/etherd/e$shelf.0 /dev/etherd/e$shelf.1
mdadm -C /dev/md2 -l 1 -n 2 /dev/etherd/e$shelf.2 /dev/etherd/e$shelf.3
mdadm -C /dev/md3 -l 1 -n 2 /dev/etherd/e$shelf.4 /dev/etherd/e$shelf.5
mdadm -C /dev/md4 -l 1 -n 2 -x 2 /dev/etherd/e$shelf.6 /dev/etherd/e$shelf.7 \
/dev/etherd/e$shelf.8
sleep 1
# create the stripe over the mirrors
mdadm -C /dev/md0 -l 0 -n 4 /dev/md1 /dev/md2 /dev/md3 /dev/md4
Notice that the make-raid10.sh script above sets up
md4 with the hot spare drive. What if one of the drives in
md1 fails? The "spare group" mdadm feature allows an mdadm
process running in monitor mode to dynamically allocate hot spares as
needed, so that the single hot spare can replace a faulty disk in any
RAID 1 of the four.
The configuration file below tells the mdadm monitor process that it can use the hot spare to replace any drive in the RAID 10.
# mdadm-aoe.conf
# see mdadm.conf manpage for syntax and info
#
# There's a "spare group" called e1, after the shelf
# with address 1, so that mdadm can use hot spares for
# any RAID 1 in the RAID 10 on shelf 1.
#
DEVICE /dev/etherd/e1.[0-9]
ARRAY /dev/md1
devices=/dev/etherd/e1.0,/dev/etherd/e1.1
spare-group=e1
ARRAY /dev/md2
devices=/dev/etherd/e1.2,/dev/etherd/e1.3
spare-group=e1
ARRAY /dev/md3
devices=/dev/etherd/e1.4,/dev/etherd/e1.5
spare-group=e1
ARRAY /dev/md4
devices=/dev/etherd/e1.6,/dev/etherd/e1.7,/dev/etherd/e1.8
spare-group=e1
ARRAY /dev/md0
devices=/dev/md1,/dev/md2,/dev/md3,/dev/md4
MAILADDR root
# This is normally a program that handles events instead
# of just /bin/echo. If you run the mdadm monitor in the
# forground, though, using echo allows you to see what events
# are occurring.
#
PROGRAM /bin/echo
/etc/rt rather than the conventional
/etc/raidtab. The kernel uses the existence of
/etc/raidtab to trigger starting the RAID device on boot
before any other initializations are performed. This is done to
permit users the ability to use a Software RAID device for their root
filesystem. Unfortunately, because the kernel has not yet initialized
the network it is unable to access the EtherDrive storage at this point
and the kernel hangs. The workaround for this is to place
EtherDrive-based RAID configurations in another file such as /etc/rt
and add calls in an rc.local file similar to the following for startup
on boot:
raidstart -c /etc/rt /dev/md0
mount /dev/md0 /mnt/raid
These questions are no longer frequently asked, probably because they relate to software that is no longer widely used.
A: When the hotplug service was first making its way into Linux distributions, it could slow things down and cause problems when the aoe module loaded. For some systems, it is may be easiest to disable it on your system. Usually the right commands look like this:
chkconfig hotplug off
/etc/init.d/hotplug stop
More recent distributions may need hotplug working in conjunction with udev. See the udev question in this FAQ for more information.