This article describes the most common traits of embedded Linux™ distributions that people employ on x86 hardware and contrasts some of the different options frequently seen on non-x86 embedded systems.
By the time a system has booted itself to the point where it can run your application-level code, any one variant of Linux is, practically by definition, largely similar to another. However, there are several different methodologies that you can use to get the system from power-on reset to a running kernel, and beyond that point, you can construct the filesystem in which your application will run in different ways.
Each approach has its own distinct advantages and disadvantages, and a definite, two-way relationship exists between the hardware you choose to implement and the way you will structure the power-up and Initial Program Load (IPL) process. Understanding the software options available to you is a critical part of the research you must do before designing or selecting hardware.
The x86 Linux boot process
The most fundamental and obvious difference between x86 boards and embedded systems based on PPC, ARM, and others is that the x86 board will ship with one or more layers of manufacturer-supplied "black box" firmware that helps you with power-on initialization and the task of loading the operating system out of secondary storage. This firmware takes the system from a cold start to a known, friendly software environment ready to run your operating system. Figure 1 is a diagram of the typical PC boot process, with considerably more detail than you tend to find in PC-centric literature:
Figure 1. Typical start-up process for x86 Linux
For cost reasons, modern PC mainboard BIOS code is always stored compressed in flash. The only directly executable code in that chip is a tiny boot stub. Therefore, the first task on power-up is to initialize the mainboard chipset enough to get the DRAM controller working so that the main BIOS code can be decompressed out of flash into a mirror area in RAM, referred to as shadow RAM. This area is then write-protected and control is passed to the RAM-resident code. Shadow RAM is permanently stolen by the mainboard chipset; it cannot later be reclaimed by the operating system. For legacy reasons, special hardware mappings are set up so that the shadow RAM areas appear in the CPU's real-mode memory map at the locations where old operating systems like MS-DOS would expect to find them.
Keep in mind that the PC is an open architecture. This openness even extends down to firmware modules within the BIOS itself. Once the power-on initialization (POI) code has run, the next step it takes is to enumerate peripherals, and optionally install hooks provided by expansion ROMs in those peripherals. (Some of those expansion ROMs -- for instance, the video BIOS in a system that has onboard integrated video hardware -- will physically reside in the main BIOS image, but conceptually they are separate entities). The reasons the BIOS has to do this redundant initialization are:
- The main BIOS itself needs basic console services to announce messages and allow the user to override default start-up behavior and configure system-specific parameters.
- Historical issues limit the size of a user-supplied bootloader program to slightly less than 512 bytes. Since this isn't enough space to implement all the possible device drivers that might be required to access different displays and storage devices, it's necessary for the BIOS to install standardized software interfaces for all installed, recognized hardware that might be required by the bootloader.
Once all the BIOS-supported system peripherals are initialized, the main BIOS code will run through candidate boot devices (in accordance with a user-configurable preference list) looking for a magic signature word. Storage devices for IBM®-compatible PCs have historically used a sector size of 512 bytes, and therefore the BIOS only loads the first 512 bytes from the selected boot device. The operating system's installation program is responsible for storing sufficient code in that zone to bootstrap the remainder of the IPL process.
Although it would be possible to write a minimalist Linux bootloader that would fit into such a space, practical Linux bootloaders for the PC consist of two stages: a small stub that lives in the boot sector, and a larger segment that lives somewhere else on the boot medium, usually inside the partition that contains the root filesystem. LILO and grub are the best-known bootloaders for mainstream Linux installations, and SYSLINUX is a popular choice for embedded distributions.
Using a RAMdisk
The primary purpose of the bootloader is to load the operating system kernel from secondary storage into RAM. In a Linux system (x86 or otherwise), the bootloader can also optionally load an initial RAMdisk image. This is a small filesystem that resides entirely in RAM. It contains a minimal set of modules to get the operating system off the ground before mounting the primary root filesystem. The original design purpose for initial RAMdisk support in the kernel was to provide a means whereby numerous optional device drivers could be made available at boot time (potentially drivers that needed to be loaded before the root filesystem could be mounted).
You can get an idea of the original usage scenario for the RAMdisk by considering a bootable Linux installation CD-ROM. The disk needs to contain drivers for many different hardware types, so that it can boot properly on a wide variety of different systems. However, it's desirable to avoid building an enormous kernel with every single option statically linked (partly for memory space reasons, but also to a lesser degree because some drivers "fight" and shouldn't be loaded simultaneously). The solution to this problem is to link the bare minimum of drivers statically in the kernel, and to build all the remaining drivers as separately loadable modules, which are then placed in the RAMdisk. When the unknown target system is booted, the kernel (or start-up script) mounts the RAMdisk, probes the hardware, and loads only those modules appropriate for the system's current configuration.
Having said all that, many embedded Linux applications run entirely out of the initial RAMdisk. As long as you can spare the memory -- 8MB is usually more than enough -- it's a very attractive way of organizing your system. Generally speaking, this is the boot architecture I favor, for a few reasons:
- The root filesystem is always writeable. It's much less work to have a writeable root than it is to coerce all your other software to put its temporary files in special locations.
- There is no danger of exhausting flash memory erase-modify-write lifetimes or of corrupting the boot copy of the root filesystem, because the system executes entirely out of a volatile RAM copy.
- It is easy to perform integrity-checking on the root filesystem at boot time. If you calculate a CRC or other check value when you first install the root filesystem, that same value will be valid on all subsequent boots.
- (Particularly interesting to applications where the root filesystem is stored in flash) You can compress the boot copy of the root filesystem, and there is no run time performance hit. Although it's possible to run directly out of a compressed filesystem, there's obviously an overhead every time your software needs to access that filesystem. Compressed filesystems also have other annoyances, such as the inability to report free space accurately (since the estimated free space is a function of the anticipated compression ratio of whatever data you plan to write into that space).
Other x86 boot considerations
Notice a few other points from Figure 1. The first is that the color coding is meaningful. In the blue boxes, the system is running BIOS code and accessing all system resources through BIOS calls. In the green boxes, the system is running user-provided code out of RAM, but all resources are still accessed through BIOS calls. In the yellow boxes, the system is running Linux kernel code out of RAM and operating out of a RAM disk. Hardware is accessed through the Linux device driver architecture. The purple boxes are like the yellow boxes, except that the system is running out of some kind of secondary storage rather than a RAMdisk. The rules being followed in the gray box are system-specific.
You'll observe from this that there are two possible boot routes (actually, more) once the kernel has been loaded. You can load an initial RAMdisk and run entirely out of that, you can use the initial RAMdisk and then switch over to a main root filesystem on some other storage medium, or you can skip the initial RAMdisk altogether and simply tell the kernel to mount a secondary storage device as root. Desktop Linux distributions tend to use the latter design model.
Also note that there is an awful lot of redundant code here. The BIOS performs system tests and sets up a fairly complex software environment to make things cozy for operating systems like MS-DOS. The Linux kernel has to duplicate much of the hardware discovery process. As a rule, once the kernel loads, none of the ROM-resident services are used again (although there are some exceptions to this statement), yet you still have to waste a bunch of RAM shadowing that useless BIOS code.
The non-x86 Linux boot process
In contrast to the x86's complex boot process, an embedded device like the Kuro Box jumps as directly as possible into the operating system. Although there are extant standards for implementing firmware interfaces (equivalent to the PC ROM-BIOS) in PowerPC® systems, these standards are rarely implemented in embedded appliances. The general firmware construction in such a system (assuming that it is based on Linux) is that the operating system kernel, a minimal filesystem, and a small bootloader all reside in linearly-accessible flash memory.
At power-up, the bootloader initializes the RAM controller and copies the kernel and (usually) the initial RAMdisk into RAM. Flash memory is typically slow and often has a narrower data bus than other memories in the system, so it's practically unheard of to execute the kernel directly out of flash memory, although it's theoretically possible with an uncompressed kernel image.
Most bootloaders also give the user some kind of recovery interface, whereby the kernel and initial RAMdisk can be reloaded from some external interface if the flash copies are bad or missing. Off-the-shelf bootloaders used in these applications include blob, U-Boot and RedBoot, although there are others -- and there are many applications that use utterly proprietary bootloaders. Figure 2 illustrates a typical start-up flow for a non-x86 embedded Linux device:
Figure 2. Typical start-up process for PPC or ARM Linux
Observe that, as for the x86 startup process above, you have the same possible different routes once the kernel has been loaded. Also note that once control passes to the kernel, the boot process is identical to what it was on the x86. This is to be expected: the further you get in the boot process, the more the software environment is defined by the operating system's API specification rather than the vagaries of the underlying hardware.
The layout of flash memory
The exact layout of such a system in flash memory depends on two principal factors: the flash device sector size (usually in the neighborhood of 64KB), and the processor's power-on-reset behavior. A core like ARM, which starts execution at address 0, will put the bootloader at the bottom of flash. A core like x86 will need to put the bootloader at the top.
There are at least two, and generally four, entities that need to be installed in flash: the bootloader (mandatory), an optional parameter block providing nonvolatile storage for boot options, calibration data and other information, the Linux kernel itself (again, mandatory), and almost always an intial RAMdisk image. For example, a layout for a 4MB flash chip with a 64KB sector size might be as follows:
Listing 1. Typical layout of a 4MB flash chip
000000-01FFFF Bootloader (128KB) 020000-02FFFF Parameter block (64KB, probably mostly unused) 030000-1FFFFF Kernel (1.8MB) 200000-3FFFFF Initial RAMdisk image (2MB)
While it is possible to write these various segments across sector boundaries (and it is especially tempting in the case of the parameter block, which will likely be more than 99% empty), this is an extremely unwise practice and should be avoided unless you are under terribly severe flash space constraints. It is particularly vital that the bootloader should reside in a private segment that can be left write-protected. Otherwise, a failed firmware upgrade operation may leave the system entirely inoperable. Good system engineering should provide a safe fallback position from any possible user-initiated upgrade process.
The only part of this software bundle that absolutely must be preloaded at the factory is the bootloader. Once the system is startable from that boot code, you can use other (end-user-accessible) interfaces to load the kernel and RAMdisk image.
Why not do this on x86?
By the way, at this point the attentive reader may be wondering why embedded PC applications can't use a special boot ROM that simply loads the operating system kernel directly off disk (or some other medium).
The answer to this is that while it's possible to write a custom cut-down bootstrap program for a PC motherboard (see, for example, the LinuxBIOS project), the types of applications that use PC hardware tend to be using the board as a black box. Typically, the system integrator will not even have access to datasheets or schematics for the board; they can't write a bootstrap program even if they want to. Furthermore, PC operating systems are built on the assumption that lowest-common-denominator BIOS services are available, at least at boot time. In other words, it's a simple fact that the path of least resistance is so much easier than a fully custom alternative that practically nobody tries to do it the "smart" way. The inefficiencies of the multi-layer BIOS approach are lost in the noise (as it were) compared with the overall system specifications.
Having digested all the above, assuming you understand approximately how large your various software modules will be, you are well prepared to select flash and RAM sizes and layouts for a custom embedded system. Kuro Box happens to use a very uncomplicated memory architecture. It has a single 4MB linear flash chip and 64MB SDRAM. While this is the simplest design, it is not necessarily the cheapest, and you may wish to consider other alternatives if you are designing your own system.
A few alternatives
One hardware architecture that I have used with some success, and which I have also seen in a few other commercial products, is to use a very small, cheap (generally narrow-bus) OTP EPROM as the primary boot device. This chip is factory-programmed with just enough bootstrap code to load the main firmware image off a secondary storage device and check its integrity. It is very useful if you can also include a little additional intelligence so that the secondary storage device can be reloaded from some external source -- removable media, a serial port, Ethernet, USB or something else -- if the main firmware image becomes corrupted.
An attractive choice of storage device for the main image is NAND flash, which is cheaper than the linear NOR flash used by Kuro Box. NAND flash is produced in vast quantities for removable storage devices: CompactFlash cards, USB "pen disks," Secure Digital (SD) cards, MP3 players, and so on. Although it is possible, with a minimal amount of external logic, to graft NAND flash onto a normal flash/ROM/SRAM controller (such as that in the MPC8241), there are a couple of reasons why you can't simply boot directly out of the NAND flash. The first is that NAND is not guaranteed error-free; it's the host's responsibility to maintain ECC and bad sector mapping information. The second reason is that NAND flash is addressed serially; you send the chip a block number, then read the block out into RAM. Hence, you need a little boot firmware on a normal random-access PROM to do the physical-level management of the NAND. (See Resources for more on NAND and NOR.)
Note that some microcontrollers provide hardware NAND controllers that obviate the need for the little boot PROM I discussed above. The disadvantage of relying entirely on that sort of hardware is that you lose the failsafe system-recovery features that can easily be implemented in the boot PROM. However, if you're working against space or cost constraints, and your micro has the NAND control hardware, you may want to avail yourself of it. SoCs sold for cell phone applications use this sort of technology.
By now you should understand many of the design choices to be made when building your embedded Linux distribution and selecting the memories in which it will reside and run. You should also understand the way parts of your software distribution can be spread between a RAMdisk and secondary storage devices. The next article gets into the gory details of the software preloaded on the Kuro Box and sets the stage for some real development work.
- Migrating from x86 to PowerPC is the only developerWorks Power Architecture technology series on the entire Internet that will help you build your own remote-controlled robot submarine army. Missed a previous installment? Don't dismay: it's astonishingly easy to read them all now.
- Need to hack a serial port onto your Kuro box? Lewin has posted all of the details to his site.
- Tired of waiting for the BIOS to probe hardware? You may be able to load the kernel directly (developerWorks, May 2004).
- The LinuxBIOS project is a BIOS replacement for x86 PCs. It directly boots the Linux kernel and requires no secondary storage.
- Das U-Boot is a flexible and popular bootloader for numerous different hardware architectures.
- The blob bootloader (Boot Loader OBject) is a small, easy-to-use bootloader designed for ARM systems.
- Red Hat's RedBoot bootloader is actually a miniature operating system, based on the eCos product, also from Red Hat. It is very flexible and has been ported to many platforms.
- This Toshiba presentation describes some of the structure and advantages of NAND flash memories (versus NOR).
- Red Hat offers a fairly detailed x86-centric description of the Linux start-up and shutdown process.
- This useful article discusses how to write code that lives in the Master Boot Record (MBR).
- A recent developerWorks article, Standards and specs: Open Firmware -- the bridge between power-up and OS, covers the Open Firmware boot loader environment (developerWorks, October 2004).
- IBM has some related material in Linux Handbook: A Guide to IBM Linux Solutions and Resources (IBM Redbook, 2003).
- Find more Linux-related resources at the IBM developerWorks Linux zone and the developerWorks Linux on Power Architecture Developer's corner.
- Have experience you'd be willing to share with Power Architecture zone readers? Article submissions on all aspects of Power Architecture technology from authors inside and outside IBM are welcomed. Check out the Power Architecture author FAQ to learn more.
- Have a question or comment on this story, or on Power Architecture technology in general? Post it in the Power Architecture technical forum or send in a letter to the editors.
- The Power Architecture Community Newsletter includes full-length articles as well as recent news about members of the Power Architecture community and upcoming events of interest. Subscribe to the newsletter today!
- All things Power are chronicled in the developerWorks Power Architecture editors' blog, which is just one of many developerWorks blogs.
- Find more articles and resources on Power Architecture technology and all things related in the developerWorks Power Architecture technology content area.
- Download a IBM PowerPC 405 Evaluation Kit to demo a SoC in a simulated environment, or just to explore the fully licensed version of Power Architecture technology. This and other fine Power Architecture-related downloads are listed in the developerWorks Power Architecture technology content area's downloads section.