Table of Contents
This example of booting from AoE storage was contributed via email by Jayson Vantuyl and edited by Ed L. Cashin.
I don't have anything booting off of PXE, I usually flash etherboot directly into the flash ROM.
Getting pxeboot (PXE version of etherboot) is fairly easy and I'm sure you guys can do that from the Network Boot and Exotic Root HOWTO. I also include a link to PXELINUX, the PXE port of the SYSLINUX boot loader, below.
After that, it gets a little interesting (defined "complicated"). The way I do it for my clients involves building a Gentoo system in a // chroot chroot and using it to build a stripped down boot environment that can be loaded either as an initrd or using initramfs.
I highly recommend this approach. It's the only way I know of, short of using a Linux From Scratch approach, to make a stripped down boot environment. Following is an EXTREME crash course in doing this. Once this is done, it is very easy to keep up to date. It also has the advantage that it can be configured to make binary packages that can be used to update any test chroot builds you have.
Note that the following instructions will create a decent preboot environment for setting up an AoE device for use as a root filesystem. However, it is fairly easy to add software RAID and LVM support.
You must first download the Gentoo embedded stage3 from the web. Any mirror will work. An example URL:
http://gentoo.osuosl.org/experimental/x86/embedded/stages/stage3-x86-uclibc-2005.0.tar.bz2
Note that this is a 35 megabyte build of a Gentoo embedded system. Make a directory, uncompress it. All further commands are executed in this chroot. It is extremely likely I've left something out, as this is initially quite a process. Note that it is not necessary to do this on a Gentoo Linux system. I originally did this on a Debian Sarge system (and have done it on Redhat as well).
It may be necessary to modify files in the etc directory. The etc/make.conf
file can be tweaked to create packages that can later be utilized to build
alternate chroots without rebuilding the packages from source (which Gentoo
does by default). It is also where you specify USE flags (which allow
various build options to be tweaked). You may also need to create
etc/portage/packages.unmask and etc/portage/packages.keywords in
order to open up the packages you want. I only modify the make.conf file.
In my build system, I have the following /etc/make.conf:
CFLAGS="-Os -pipe" CHOST="i386-pc-linux-uclibc" CXXFLAGS="-Os -pipe" FEATURES="buildpkg"
From here you should update Portage (the packaging system for Gentoo). This can be done by issuing:
emerge sync
Packages can be updated by issueing:
emerge -u world
From here, you can make a chroot for use as an initrd roughly like this:
mkdir -p /images/aoe ROOT=/images/aoe emerge busybox mdadm
This builds a base system with nothing but mdadm and busybox. This is all I've needed to start a Software RAID and mount it as root. Note that you may not need the mdadm, as you have RAID already available for you. You might wish to try to build LVM2, though, since it IS extremely useful for your customers. Similarly, CLVM might be an option too. This system is based on uclibc to make it small, so these LVM options may not respond well to that. Note that klibc is also an option, but not available this way.
At this point I usually make links in /bin as such:
ln -s busybox sh ln -s busybox ls ln -s busybox echo ln -s busybox sleep
And links in /sbin as such:
ln -s ../bin/busybox ifconfig ln -s ../bin/busybox ip ln -s ../bin/busybox modprobe ln -s ../bin/busybox mount
This system needs to have a /linuxrc file that provides instructions
for when it is used as an initrd (not sure how this works with
initramfs). I create one with the following contents:
#!/bin/sh
# Load Config . /config
# Load Modules for MOD in $MODULES ; do /sbin/modprobe $MOD done
# Bring Up Interfaces for IFACE in $AOE_IFACES ; do /sbin/ip link set up dev $IFACE done
# Settle For A Moment (sometimes link takes a sec to come up) /bin/sleep 4
# Discover AOE /bin/echo 1 > /dev/aoe/discover
# Wait for AOE to Figure Itself Out /bin/sleep 2
# Start Any Raids /sbin/mdadm -A -s
# Upon Exit, Should find the new root, mount it, and be happy...
Note that the above file doesn't change per unit, but the config file it references does. Also, I think it could be optimized a little, as busybox will execute its own internal commands without the symlinks. I may refine this further. If I do, I'll send you an update.
I create a config file that is customized as follows:
MODULES=e1000 # Or whatever modules you need AOE_IFACES=eth1 # Or whatever interfaces are appropriate
I also create an /etc/mdadm.conf to locate the RAID. Something like this:
DEVICE /dev/aoe/e[0-9].[0-9] DEVICE /dev/aoe/e[0-9].[0-9]p[1-9] DEVICE /dev/aoe/e[0-9].[0-9]p1[0-5] ARRAY /dev/md0 uuid=01234567:89ABCDEF:01234567:89ABCDEF
You can then run mkcramfs on the directory to make an approximately
2MB initrd. When you boot the kernel, make sure to load the initrd
and to put on the command line root=/dev/md0 or whatever is
appropriate. The great thing about using an initrd this way is that
your linuxrc only has to set up the environment, but it still uses the
"root=" option to find the root. This plays well with most bootloader
set ups (which expect to modify root= instead of tweaking the initrd
like some setups do) and also preserves the ability to use
init=/bin/sh for disaster recovery or major maintenance.
To clarify, the early boot process of the 2.6 kernel (and 2.4 is similar) goes something like this:
root=…
init=…
If you let the kernel do all of the work, you can see that your linuxrc file
need only configure whatever is necessary for your magic. It could even load
the config file from the network. If you specify ip=dhcp on the kernel
command line and build your kernel with IP Autoconfiguration DHCP support,
you can even have the interface already configured via DHCP and retrieve the
configuration into a ramfs mounted /tmp using busybox's wget
command.
This is a ridiculously powerful preboot environment considering it can be obtained with less than 5MB of kernel and initrd. I am strongly convinced that this is evidence of some extreme mojo in the design of the Linux kernel. These guys really have a plan for preboot and it exists specifically to accommodate more complex boot devices just like an AoE system from Coraid.
Also, I don't usually remove unnecessary files from the chroot because I'm, well, lazy. If you remove them, then it gets down to 1.3MB. This should be small enough that it (along with the kernel) will fit on most flash ROMs. Should you be designing a product, you may even be able to use a bootloader like ROLO to flash it all directly into the BIOS ROM. Exciting!
At this point you'll need to boot of the network. You might try the pxelinux bootloader. A pretty comprehensive description is available here:
http://syslinux.zytor.com/pxe.php
It doesn't take much more than the right TFTP server and some DHCP magic. I know that the appropriate versions of software are available in the Gentoo distribution. I THINK Debian Sarge has it all too (although you may need to build a different TFTP server).
Make no mistake, the above is probably the ultimate system for building and embedded Linux system available right now. You get the ability to have a stripped down system but still have the flexibility required to have binary packages that can be used for speedy and flexible system image generation.
This document is licensed under the Creative Commons Attribution License. It's basically Public Domain provided that you credit me with having written it and don't sue me for providing it.
The text of the license is here:
http://creativecommons.org/licenses/by/2.5/legalcode
Creative Commons Attribution License
THE WORK (AS DEFINED BELOW) IS PROVIDED UNDER THE TERMS OF THIS
CREATIVE COMMONS PUBLIC LICENSE ("CCPL" OR "LICENSE"). THE WORK IS
PROTECTED BY COPYRIGHT AND/OR OTHER APPLICABLE LAW. ANY USE OF THE
WORK OTHER THAN AS AUTHORIZED UNDER THIS LICENSE OR COPYRIGHT LAW IS
PROHIBITED.
BY EXERCISING ANY RIGHTS TO THE WORK PROVIDED HERE, YOU ACCEPT AND
AGREE TO BE BOUND BY THE TERMS OF THIS LICENSE. THE LICENSOR GRANTS
YOU THE RIGHTS CONTAINED HERE IN CONSIDERATION OF YOUR ACCEPTANCE OF
SUCH TERMS AND CONDITIONS.
1. Definitions
1. "Collective Work" means a work, such as a periodical issue,
anthology or encyclopedia, in which the Work in its entirety in
unmodified form, along with a number of other contributions,
constituting separate and independent works in themselves, are
assembled into a collective whole. A work that constitutes a
Collective Work will not be considered a Derivative Work (as
defined below) for the purposes of this License.
2. "Derivative Work" means a work based upon the Work or upon the
Work and other pre-existing works, such as a translation,
musical arrangement, dramatization, fictionalization, motion
picture version, sound recording, art reproduction, abridgment,
condensation, or any other form in which the Work may be recast,
transformed, or adapted, except that a work that constitutes a
Collective Work will not be considered a Derivative Work for the
purpose of this License. For the avoidance of doubt, where the
Work is a musical composition or sound recording, the
synchronization of the Work in timed-relation with a moving
image ("synching") will be considered a Derivative Work for the
purpose of this License.
3. "Licensor" means the individual or entity that offers the Work
under the terms of this License.
4. "Original Author" means the individual or entity who created the
Work.
5. "Work" means the copyrightable work of authorship offered under
the terms of this License.
6. "You" means an individual or entity exercising rights under this
License who has not previously violated the terms of this
License with respect to the Work, or who has received express
permission from the Licensor to exercise rights under this
License despite a previous violation.
2. Fair Use Rights. Nothing in this license is intended to reduce,
limit, or restrict any rights arising from fair use, first sale or
other limitations on the exclusive rights of the copyright owner
under copyright law or other applicable laws.
3. License Grant. Subject to the terms and conditions of this License,
Licensor hereby grants You a worldwide, royalty-free,
non-exclusive, perpetual (for the duration of the applicable
copyright) license to exercise the rights in the Work as stated
below:
1. to reproduce the Work, to incorporate the Work into one or more
Collective Works, and to reproduce the Work as incorporated in
the Collective Works;
2. to create and reproduce Derivative Works;
3. to distribute copies or phonorecords of, display publicly,
perform publicly, and perform publicly by means of a digital
audio transmission the Work including as incorporated in
Collective Works;
4. to distribute copies or phonorecords of, display publicly,
perform publicly, and perform publicly by means of a digital
audio transmission Derivative Works.
5.
For the avoidance of doubt, where the work is a musical composition:
1. Performance Royalties Under Blanket Licenses. Licensor
waives the exclusive right to collect, whether
individually or via a performance rights society
(e.g. ASCAP, BMI, SESAC), royalties for the public
performance or public digital performance (e.g. webcast)
of the Work.
2. Mechanical Rights and Statutory Royalties. Licensor waives
the exclusive right to collect, whether individually or
via a music rights agency or designated agent (e.g. Harry
Fox Agency), royalties for any phonorecord You create from
the Work ("cover version") and distribute, subject to the
compulsory license created by 17 USC Section 115 of the US
Copyright Act (or the equivalent in other jurisdictions).
6. Webcasting Rights and Statutory Royalties. For the avoidance of
doubt, where the Work is a sound recording, Licensor waives the
exclusive right to collect, whether individually or via a
performance-rights society (e.g. SoundExchange), royalties for
the public digital performance (e.g. webcast) of the Work,
subject to the compulsory license created by 17 USC Section 114
of the US Copyright Act (or the equivalent in other
jurisdictions).
The above rights may be exercised in all media and formats whether now
known or hereafter devised. The above rights include the right to make
such modifications as are technically necessary to exercise the rights
in other media and formats. All rights not expressly granted by
Licensor are hereby reserved.
4. Restrictions.The license granted in Section 3 above is expressly
made subject to and limited by the following restrictions:
1. You may distribute, publicly display, publicly perform, or
publicly digitally perform the Work only under the terms of this
License, and You must include a copy of, or the Uniform Resource
Identifier for, this License with every copy or phonorecord of
the Work You distribute, publicly display, publicly perform, or
publicly digitally perform. You may not offer or impose any
terms on the Work that alter or restrict the terms of this
License or the recipients' exercise of the rights granted
hereunder. You may not sublicense the Work. You must keep intact
all notices that refer to this License and to the disclaimer of
warranties. You may not distribute, publicly display, publicly
perform, or publicly digitally perform the Work with any
technological measures that control access or use of the Work in
a manner inconsistent with the terms of this License
Agreement. The above applies to the Work as incorporated in a
Collective Work, but this does not require the Collective Work
apart from the Work itself to be made subject to the terms of
this License. If You create a Collective Work, upon notice from
any Licensor You must, to the extent practicable, remove from
the Collective Work any credit as required by clause 4(b), as
requested. If You create a Derivative Work, upon notice from any
Licensor You must, to the extent practicable, remove from the
Derivative Work any credit as required by clause 4(b), as
requested.
2. If you distribute, publicly display, publicly perform, or
publicly digitally perform the Work or any Derivative Works or
Collective Works, You must keep intact all copyright notices for
the Work and provide, reasonable to the medium or means You are
utilizing: (i) the name of the Original Author (or pseudonym, if
applicable) if supplied, and/or (ii) if the Original Author
and/or Licensor designate another party or parties (e.g. a
sponsor institute, publishing entity, journal) for attribution
in Licensor's copyright notice, terms of service or by other
reasonable means, the name of such party or parties; the title
of the Work if supplied; to the extent reasonably practicable,
the Uniform Resource Identifier, if any, that Licensor specifies
to be associated with the Work, unless such URI does not refer
to the copyright notice or licensing information for the Work;
and in the case of a Derivative Work, a credit identifying the
use of the Work in the Derivative Work (e.g., "French
translation of the Work by Original Author," or "Screenplay
based on original Work by Original Author"). Such credit may be
implemented in any reasonable manner; provided, however, that in
the case of a Derivative Work or Collective Work, at a minimum
such credit will appear where any other comparable authorship
credit appears and in a manner at least as prominent as such
other comparable authorship credit.
5. Representations, Warranties and Disclaimer
UNLESS OTHERWISE MUTUALLY AGREED TO BY THE PARTIES IN WRITING,
LICENSOR OFFERS THE WORK AS-IS AND MAKES NO REPRESENTATIONS OR
WARRANTIES OF ANY KIND CONCERNING THE WORK, EXPRESS, IMPLIED,
STATUTORY OR OTHERWISE, INCLUDING, WITHOUT LIMITATION, WARRANTIES OF
TITLE, MERCHANTIBILITY, FITNESS FOR A PARTICULAR PURPOSE,
NONINFRINGEMENT, OR THE ABSENCE OF LATENT OR OTHER DEFECTS, ACCURACY,
OR THE PRESENCE OF ABSENCE OF ERRORS, WHETHER OR NOT
DISCOVERABLE. SOME JURISDICTIONS DO NOT ALLOW THE EXCLUSION OF IMPLIED
WARRANTIES, SO SUCH EXCLUSION MAY NOT APPLY TO YOU.
6. Limitation on Liability. EXCEPT TO THE EXTENT REQUIRED BY
APPLICABLE LAW, IN NO EVENT WILL LICENSOR BE LIABLE TO YOU ON ANY
LEGAL THEORY FOR ANY SPECIAL, INCIDENTAL, CONSEQUENTIAL, PUNITIVE
OR EXEMPLARY DAMAGES ARISING OUT OF THIS LICENSE OR THE USE OF THE
WORK, EVEN IF LICENSOR HAS BEEN ADVISED OF THE POSSIBILITY OF SUCH
DAMAGES.
7. Termination
1. This License and the rights granted hereunder will terminate
automatically upon any breach by You of the terms of this
License. Individuals or entities who have received Derivative
Works or Collective Works from You under this License, however,
will not have their licenses terminated provided such
individuals or entities remain in full compliance with those
licenses. Sections 1, 2, 5, 6, 7, and 8 will survive any
termination of this License.
2. Subject to the above terms and conditions, the license granted
here is perpetual (for the duration of the applicable copyright
in the Work). Notwithstanding the above, Licensor reserves the
right to release the Work under different license terms or to
stop distributing the Work at any time; provided, however that
any such election will not serve to withdraw this License (or
any other license that has been, or is required to be, granted
under the terms of this License), and this License will continue
in full force and effect unless terminated as stated above.
8. Miscellaneous
1. Each time You distribute or publicly digitally perform the Work
or a Collective Work, the Licensor offers to the recipient a
license to the Work on the same terms and conditions as the
license granted to You under this License.
2. Each time You distribute or publicly digitally perform a
Derivative Work, Licensor offers to the recipient a license to
the original Work on the same terms and conditions as the
license granted to You under this License.
3. If any provision of this License is invalid or unenforceable
under applicable law, it shall not affect the validity or
enforceability of the remainder of the terms of this License,
and without further action by the parties to this agreement,
such provision shall be reformed to the minimum extent necessary
to make such provision valid and enforceable.
4. No term or provision of this License shall be deemed waived and
no breach consented to unless such waiver or consent shall be in
writing and signed by the party to be charged with such waiver
or consent.
5. This License constitutes the entire agreement between the
parties with respect to the Work licensed here. There are no
understandings, agreements or representations with respect to
the Work not specified here. Licensor shall not be bound by any
additional provisions that may appear in any communication from
You. This License may not be modified without the mutual written
agreement of the Licensor and You.
The asciidoc-format source to this document is available here, and corrections or updates are welcome in the form of a unified diff. Be sure to include text for the ChangeLog.