 | Level: Intermediate Martyn Honeyford (martynh@uk.ibm.com), Software Engineer, IBM UK Labs
15 Jul 2004 Obtaining an external drive is a great way to breathe new life into older hardware, or allow you to use Linux on machines on which you can't (or don't want to) alter the internal hard drives.
Say you want to use Linux in a dual-boot arrangement, but you don't
have any free space on your computer's hard drive. One solution would be
to use a "live" Linux distribution such as Knoppix, which can be run
directly from CD. This is certainly viable for occasional use, but it has
a number of serious drawbacks:
- You will still require some permanent storage for your data
files. A floppy disk may be suitable if you only work with very
small files, or a USB flash memory key may suffice for medium-sized files,
but neither of these is ideal.
- When using a "live" CD, it is at best difficult -- and at worst,
impossible -- to install your own applications, or to customize existing
applications.
- There is a performance penalty to using live distributions, most
notably on start-up while all of the devices are being detected -- but
also while running (as everything has to be loaded from CD, which is
usually much slower than from the hard drive).
To be sure, other options also exist. For instance, you could buy
another internal drive and install Linux on that. But often, you may
not have any free drive bays in your case (this is particularly true of
laptops, which typically only allow one internal hard drive).
Or, you could replace the current drive with a larger one, and install
Linux in the resulting extra space. This, however, is a very
time-intensive option, as it requires you to reinstall your existing
OS on the new drive, reinstall and reconfigure all of your applications,
and restore all of your data.
A much better solution is to purchase an external hard drive and install
Linux on that. This lets you leave your existing hardware and
software untouched and simply connect the external drive when you want
to use Linux.
Removable drive options
The range of removable devices onto which Linux may be installed ranges
from floppy drives, through USB-flash devices, on to USB/FireWire hard
drives, and more.
While it is certainly possible to install Linux on a small-capacity device
such as a 1.44MB floppy or a 32MB USB-key, these are usually (necessarily)
specialized, cut-down distributions intended for, for instance, rescuing
broken installations.
External hard drives, however, offer the most flexibility for a general
purpose Linux distribution at a reasonable cost.
External drives are available in a large number of different sizes from a
number of different manufacturers (Maxtor, Western Digital, and so on). These drives tend to consist of an external box, which holds a
standard 3-1/2 inch or 2-1/2 inch IDE drive. These drives then are typically
connected to the computer via USB or IEEE1394 (FireWire) connection.
USB comes in two main versions, 1.1 and 2.0. Version 1.1 has a maximum
transfer speed of 12 Mbit/s (megabits per second), whereas version 2.0 supports
transfer speeds of up to 480 Mbit/s. While most 2.0-compatible drives are
backwardly compatible with 1.1, it is generally advisable to avoid using
1.1 unless there is no other option (due to its slow speed).
The FireWire standard also defines a number of different possible speeds,
but in reality, whenever people say FireWire, they mean "FireWire400,"
which supports transfers up to 400 Mbit/s.
There is little to choose between USB 2.0 and FireWire in terms of speed:
although USB 2.0 has a higher quoted speed, in practice they both tend to
be comparable due to differences in the protocols. If your machine has
both, it is probably worth going for USB rather than FireWire (for reasons I'll explain later), but if it has only FireWire, then of course
you will go for that. For maximum flexibility, choose one of a number of
drives that support both USB 2.0 and FireWire (such as the one I use
later in this article).
For machines that do not have the required ports, PCI (for desktop) and
PCMCIA (for laptop), FireWire and USB 2.0 cards are now available very
inexpensively: for example, the PCMCIA FireWire card I use later in this
article was purchased for approximately 10 GBP (under $20 US).
For the purposes of this article, I have purchased a 5-1/4 inch external
drive enclosure. This is a very flexible enclosure, which is supplied with
no drive and can be filled with any standard IDE device, including 3-1/2 inch
hard drives and 5-1/4 inch IDE devices such as CD-RW/DVD-RW drives. The
enclosure has both USB 2.0 and FireWire connections.
In order to connect the enclosure to my IBM Thinkpad T30 laptop, I also
purchased a PCMCIA FireWire card (as the built-in USB ports only support
USB 1.1).
Both the enclosure and the FireWire card were relatively inexpensive
(approximately 50 GBP and 10 GBP, respectively).
For testing purposes, I fitted the enclosure with a 13GB 3-1/2 inch IDE
drive I had laying around -- for real world usage, I would buy a
larger capacity drive, which again are now very cheap (approximately 50 GBP
per gig!)
Linux support
As you may expect, Linux support for these enclosures is very good indeed.
Any device that adheres to the SBP (Serial Bus Protocol) standard for
"Mass Storage Devices" can easily be used with Linux.
In general, to enable support for these devices you will need to have a
number of things supported in your kernel (either directly compiled in or
via modules).
For both USB and FireWire, SBP device support is implemented via SCSI
emulation -- that is to say, the devices appear to Linux as though they
were SCSI disks. This is a common way to abstract storage devices within
the Linux world (for instance, IDE CD/DVD drives are typically also
connected using SCSI emulation). For this reason, the following kernel
support is required:
- SCSI support
- SCSI emulation
- SCSI disk support
In addition, the following support will be required according to the
connection method:
- For FireWire:
- IEEE1394 support
- OHCI1394 support
- RAW1394 support
- SBP-2 support
- For USB:
- (host-side) USB support
- OHCI support
- UHCI support
- USB mass-storage support
You will obviously have to have all the normal support for the rest of
your hardware (graphics cards and so on), and may require some additional
modules, depending on your exact hardware.
For instance, I am using a PCMCIA (cardbus) FireWire card, so I needed to
add:
- PCMCIA support
- cardbus support
Installation
Now that we have our nice external drive, we shall start by installing
Linux on it.
The easiest way to install Linux these days (certainly in my opinion), is
to connect all of your hardware (in my case, this consisted of plugging
in the PCMCIA FireWire card, attaching the FireWire cable to the PCMCIA
card and the drive, and turning on the drive's power switch); then boot up
your machine with the installation CD from your distribution of choice.
My distribution of choice is Gentoo (see Resources for a link), so I used the latest "Universal" x86 Live CD (2004.1). The steps required for other distributions should be
more or less comparable to those outlined here.
Once you have booted with the install CD, with a bit of luck it should
have recognized your drive. The drive should appear as a disk under
/dev/sdX, where X is a lowercase letter starting at "a." On my system, the
external drive was detected as /dev/sda, but this will vary if you have
other SCSI disks (or emulated SCSI disks); in that case, it might be /dev/sdb or some other letter. If your drive is not detected automatically, some further
steps may be required -- for instance, you might have to pass boot options
to enable FireWire or PCMCIA, or you might have to manually load some kernel
modules, or other things of that sort (see Resources for links to troubleshooting guides).
Once the drive has been recognized, it should behave exactly like an
internal hard drive as far as the rest of the installation is concerned;
so you should be able to partition it as required and install Linux as
normal.
One word of caution, however: care needs to be taken when deciding on
where to install the boot loader (usually GRUB or LILO) -- I would
recommend not installing it in the Master Boot Record (MBR) (which is usually the default).
Rather, it should be installed in the root partition (or boot partition,
if you use a separate one) of the external drive.
Now that we have Linux installed on the device, we want to boot it up.
This is where things can start to get a little tricky.
Booting
Before I discuss booting up your new drive, a little boot loader theory is
required.
Boot loaders are usually installed in the MBR of the
first hard disk in the machine. When the boot loader is invoked (the BIOS
automatically executes the code in the MBR), it usually displays a menu
of possible OSes to boot. Selecting a given OS causes it to boot up.
Two things should be noted about this scenario:
- The menu of OS choices is (usually) loaded from disk
- To boot the relevant OS, the bootloader needs to read the relevant
kernel from disk
As the above takes place before the OS has been loaded, it means that all
of the disk reading must take place by way of BIOS calls. This has a very
serious implication: namely, that in order to boot the disk directly, your
BIOS must support disks connected via FireWire or USB. This can typically
be seen as a BIOS option to boot from these types of disk. FireWire BIOS
support is currently very rare indeed, but USB support is becoming
reasonably common. Therefore, if you are using USB on a relatively recent
machine, it should be possible to boot the drive into Linux directly.
After installing GRUB in the MBR of the external drive, I was able to boot
it directly when connected via USB. Simply enter the BIOS setup utility
while booting with the disk connected. The external disk will appear as a
regular hard drive: move it so it is before the internal drive in the boot
order.
I was also able to install a boot loader in the MBR of the internal drive
and use that to boot the USB drive (where it appeared as hd1 in GRUB). If you are using FireWire, chances are
that your BIOS will not be able to boot the drive directly, and a little
more work will be required.
Luckily, due to the flexibility of Linux, there is a fairly simple
solution if you cannot boot directly (which is certainly the case in my
situation, with a PCMCIA FireWire card!) You can perform the preliminary
boot steps from a supported device such as a floppy drive, CD, USB key, or
a tiny partition on the main drive, and then use the external drive for
everything else.
Building the boot image
There are two methods we can use to boot:
- One-phase boot
The kernel boots, mounts the root filesystem, and continues
initialization by calling the initialization scripts (usually /sbin/init)
- Two-phase (initrd) boot
The kernel boots, mounts an initial ram disk (initrd), performs further
customizable initialization, then mounts the root filesystem and continues
initialization (again, usually by calling /sbin/init)
Each of these methods has its own advantages and disadvantages.
One-phase boot
In order to use the one-phase boot, we need to build a kernel that has
all the drivers needed to mount the root file system built in (any other
drivers can be built at modules that can be loaded from the root
partition during normal initialization).
If we are attempting to boot from a very small device, such as a floppy
disk, the best approach is to build a kernel with just enough drivers
built in to allow us to mount the root external file system -- and build
everything else as modules. For example, I have the SCSI support, PCMCIA
support, IEE1394, SBP, and like support built in, but everything else --
including graphics card support, networking device support, and so on --
is built as modules that are stored on the root partition (on the
external drive), rather than on the floppy.
With a simple (one-phase) boot procedure, we should be able to build a
kernel with the required support, put it on a floppy drive, install a boot
loader on the floppy (I use GRUB, but there are other options, such as
LILO), then boot with something similar to this (for GRUB):
root (fd0)
kernel (fd0)/boot/bzImage root=/dev/sda1
This almost works, except for two problems:
- Because of the fact that SBP support uses SCSI emulation, the
emulated SCSI bus need to be "rescanned" in order to detect the disk and
allow /dev/sda1 to be mounted. This scanning is performed with a couple
of simple commands. Unfortunately, however, using a one-phase boot, we
cannot run any commands until the kernel has finished booting, and the
kernel cannot finish booting until the root file system is mounted -- a
classic Catch-22 situation. Thankfully, there is a patch available for
2.4 kernels that causes the SCSI bus to be scanned on startup (see Resources for more details). By applying this
patch, I was able to have the external drive automatically detected by the
kernel during bootup with no rescanning commands required. This leads us
to the next problem.
- There is a timing window within the kernel, which means that often,
the kernel will proceed to try to mount the root device before it has had
a chance to be properly detected or initialized. Again, there is a patch
available for this (please see Resources for a
link) that simply makes the kernel wait for a short period of time on
startup, and retry if it fails to mount the root filesystem (to give the
external drive time to be recognized).
By applying these two patches, I was successfully able to build a kernel
on a bootable floppy disk, which would boot and then use the external
FireWire drive as root.
The main problem with this approach is that it requires us to patch the
kernel source -- which is at best a pain (when new kernel versions are
released) and at worst, a real problem (if the patches are not maintained
to keep in step with the other changes occurring to the kernel).
It may have occurred to you that we can avoid these two problems if our
BIOS supports USB or FireWire and we are booting directly.
Unfortunately, this is not the case: while this method uses BIOS calls to
access the disk during boot up, once the kernel begins to initialize, the
BIOS is no longer used, and the disk is accessed using the kernel drivers
-- so the same problems are encountered.
Two-phase boot
As of kernel version 2.0.X, an interesting capability was added to the
Linux kernel -- the ability to use an "initial RAM disk" (or initrd) to
give a two-phase boot process.
In a nutshell, the kernel is booted as normal; but instead of mounting the
"real" root file system, a miniature root filesystem is created in RAM,
and that is mounted. Any number of arbitrary steps can then be performed
in this initial environment before the real root is mounted and we switch
to using the real root and destroy the initial RAM disk.
This is useful in all sorts of circumstances, but for our purposes we will
simply be using our mini environment to rescan the SCSI bus, wait for the
external disk to be recognized, then switch to using this as our real root
and continue to boot.
In order to use this method, we need to create two things, a kernel and
an initrd image.
The kernel is just a regular kernel which has initrd support built in.
The initrd image is a loopback filesystem image which contains our
mini-root filesystem (this image can optionally compressed with gzip to
reduce its size).
You can check the Resources section for more
information on creating or customizing your own initrd image.
Within the initrd image, there is a file called linuxrc. This is the file
that is executed when initrd is loaded, so make sure it has execute
permission! For our purposes, the linuxrc is very simple:
Listing 1. initrd linuxrc
#!/bin/sh
REAL_ROOT=/dev/sda1
# mount the /proc filesystem
mount -t proc none /proc
#for scsi-emulation
# modprobe sd_mod
#for pcmcia
# modprobe pcmcia_core
#for FireWire
# modprobe ieee1394
# modprobe ohci1394
# modprobe raw1394
# modprobe sbp2
#for USB
# modprobe usbcore
# modprobe ohci-hcd
# modprobe uhci-hcd
# modprobe usb-storage
# loop rescanning the scsi bus + rerunning devfsd
retries=5
i=1
until [ -e $REAL_ROOT ]
do
if [ $i -gt $retries ]
then
echo "Unable to mount real root ($REAL_ROOT) - Giving up!"
/bin/ash
exit
fi
echo "Real root ($REAL_ROOT) not found, retrying ($i)"
sleep 1
echo "scsi add-single-device 0 0 0" > /proc/scsi/scsi
echo "scsi add-single-device 1 0 0" > /proc/scsi/scsi
echo "scsi add-single-device 2 0 0" > /proc/scsi/scsi
/bin/devfsd /dev -np
i=$((i+1))
done
#umount /proc as it will be remounted by the normal init process
umount /proc
#now we simply exit, and the normal boot process should continue
exit 0
|
All we are doing is loading the appropriate modules to support the
external drive: they should be uncommented as required. (I built all the
required support into my kernel, hence no modules are required.) We then
loop, rescanning the SCSI bus (by echoing a command to a special file in
the /proc pseudo-filesystem and calling devfsd)
until the root device (/dev/sda1 in my case) is present. In my case, the emulated FireWire SCSI bus in question is 1 0
0, but it doesn't hurt to try a few others -- if you know which one
you will be using, you can tailor the script. Also, if you have other SCSI
devices (or emulated SCSI devices), the drive may have a different letter
(for example, /dev/sdb1). And if you are not using the first partition on
the external drive, you will need to use a different number (for example,
/dev/sda2).
All we now need to do is copy the relevant files into the initrd image
(you can mount the uncompressed image using the mount
-o loop command). In particular, we need to ensure that we have the
linuxrc file, all of the commands used therein, and any libraries that
those commands rely on. The (unmounted) image can then be optionally
compressed.
The kernel (bzImage) and the initrd image (initrd.gz) are then copied to
the (bootable, ext3) floppy.
The final step is to install a boot loader on the floppy, and boot the
kernel with the following options: kernel bzImage
root=/dev/sda1 initrd=initrd.gz.
You should now be able to boot using the floppy: it will load the kernel
from the floppy, load the initrd image into RAM, wait for the root device
to be recognized, and then continue to boot as normal from there. From
this point onwards, the floppy can be removed.
If a floppy is not appropriate (for example, if the machine has no floppy
drive), then any device that can be booted by your BIOS can be used.
Personally, I use a little 32Mb USB key for this purpose. Alternatively,
if you don't mind altering your internal hard drive, a small partition can
be created there for more convenient booting.
Resources -
Martyn's distribution of choice is Gentoo
Linux.
-
For tips on troubleshooting installation of USB mass storage devices on
Linux, see Adding USB
mass storage devices in Fedora Core.
-
IEEE 1394 for Linux offers
information and links on FireWire on Linux.
-
Wikipedia, the free encyclopedia, has articles on FireWire and on boot loaders. For
Linux, the most common boot loaders are LILO and GNU GRUB.
-
The sbp2
hotplug patch and the SPB-Linux patches are useful for
one-phase boot operations.
-
For help in creating your own custom initrd image from scratch, see The Linux Bootdisk HOWTO.
-
"Knoppix gives bootable, one-disk Linux" (developerWorks, February 2003) introduces this one-CD, bootable Linux distro.
-
"Access
USB devices from Java applications" (developerWorks, September 2003)
examines two projects which provide APIs through which Java applications
can make use of USB devices.
-
"Burning
CDs on Linux" (developerWorks, 2003) shows how to create
your own .iso images using the full power of the command line.
- Previously on developerWorks, Martyn wrote "Connecting
KDE applications using DCOP" (developerWorks, February 2004), "Postmortem
memory profiling with PERL" (developerWorks, December 2003), and the Significant
trace series of articles (developerWorks, April 2003).
- Find more resources for Linux developers in the developerWorks Linux
zone.
- Browse for books on these and other technical topics.
About the author | | | Martyn Honeyford graduated from Nottingham University with a B.Sc. in
computer science in 1996. He has worked as a software engineer at IBM UK
Labs in Hursley, England, ever since. His current role is as a developer
in the WebSphere MQ Everyplace development team. When not working, Martyn
can usually be found either playing the electric guitar (badly) or playing
video games more than most people would consider healthy. You can contact
Martyn at martynh@uk.ibm.com.
|
Rate this page
| |
