3. Faster, Safer Storage With EtherDrive Blades

Creating higher performance, fault-tolerant storage with EtherDrive blades can be achieved using Linux's "Software RAID." Reading the Linux Software RAID HOWTO before you start to work with RAID will likely save time in the long run.

Starting with an example, let us assume we have five EtherDrive blades in slots 0-4, in a shelf assigned address 0. Let us further assume we want to use these five blades to create a level-5 RAID array. Using a text editor, we create a Software RAID configuration file named "/etc/rt". The transcript below shows its contents.

$ cat /etc/rt
raiddev /dev/md0
        raid-level      5
        nr-raid-disks   5
        chunk-size      32
        persistent-superblock 1
        device          /dev/etherd/e0.0
        raid-disk       0
        device          /dev/etherd/e0.1
        raid-disk       1
        device          /dev/etherd/e0.2
        raid-disk       2
        device          /dev/etherd/e0.3
        raid-disk       3
        device          /dev/etherd/e0.4
        raid-disk       4

Here is an example for setting up and using the RAID array described by the above configuration file, /etc/rt. Each blade in this example has an attached 20GB ATA disk.

$ mkraid -c /etc/rt /dev/md0
DESTROYING the contents of /dev/md0 in 5 seconds, Ctrl-C if unsure!
handling MD device /dev/md0
analyzing super-block
disk 0: /dev/etherd/00:00, 19535040kB, raid superblock at 19534976kB
disk 1: /dev/etherd/00:01, 19535040kB, raid superblock at 19534976kB
disk 2: /dev/etherd/00:02, 19535040kB, raid superblock at 19534976kB
disk 3: /dev/etherd/00:03, 19535040kB, raid superblock at 19534976kB
disk 4: /dev/etherd/00:04, 19535040kB, raid superblock at 19534976kB
$

To make an ext3 filesystem on the RAID array and mount it, the following commands can be issued:

$ mkfs.ext3 /dev/md0
... (mkfs output)
$ mount /dev/md0 /mnt/raid
$

The resulting storage is single-fault tolerant. Add hot spares to make the array even more robust (see the Software RAID documentation for more information.) Remember that it takes the md driver some time to initialize a new RAID 5 array. A 9-blade RAID 5 with 40GB disks takes 18 hours to fully initialize. Initialization is necessary once at RAID 5 creation and not afterwards, unless a disk fails or the raid is not shut down cleanly.

During RAID 5 initialization, you can use the md device, but performance is sub-optimal until md finishes. Creating a filesystem is not recommended until array initialization is complete. The progress of the RAID construction may be observed in the output of cat /proc/mdstat.

RAID 10 is the best RAID method for performance and reliability. It is a combination of RAID 1 and RAID 0. Several RAID 1 devices are created from pairs of EtherDrive blades, and then striping is performed over the mirrors.

See the Software RAID HOWTO for more information on administering md RAIDs.

Important notes:

1. You may note above that the example creates the RAID device configuration file as /etc/rt rather than the conventional /etc/raidtab. The kernel uses the existence of /etc/raidtab to trigger starting the RAID device on boot before any other initializations are performed. This is done to permit users the ability to use a Software RAID device for their root filesystem. Unfortunately, because the kernel has not yet initialized the network it is unable to access the EtherDrive blades at this point and the kernel hangs. The workaround for this is to place EtherDrive-based RAID configurations in another file such as /etc/rt and add calls in an rc.local file similar to the following for startup on boot:

   raidstart -c /etc/rt /dev/md0
   mount /dev/md0 /mnt/raid
   

2. Some Linux distributions come with an mdmonitor service running by default. It is important to turn off this service with chkconfig mdmonitor off or your system's equivalent command. If mdadm is running in its "monitor" mode, it may interfere with failover to hot spares, stopping the RAID, and other actions.
3. Unlike the 2.6 kernel, the 2.4 kernel may use unprintable characters in /proc/mdstat. You can always see clearly by using sed as shown below. (This is sed's "listing" feature, so that's a lowercase letter "L", and not the number one.)

   sed -n l /proc/mdstat