Build custom embedded Linux distributions with the Yocto Project
Open source professional-grade tools for embedded Linux development
The Yocto Project is an open source collaboration project that provides templates, tools, and methods supporting custom Linux-based systems for embedded products, regardless of the hardware architecture. For those of you who are wondering about the name, the term yocto is the smallest SI unit. As a prefix, yocto indicates 10^-24.
This article provides a step-by-step walk through on how to use the Yocto Project’s industry-standard open source tools to create a customized Linux operating system for an embedded device and to boot the operating system in a virtual machine using QEMU. The Yocto Project, a Linux Foundation-sponsored open source project funded by major hardware companies and operating systems vendors, provides industry-class tools, methods, and metadata for building Linux systems.
Two major components of the Yocto Project are maintained in conjunction with the OpenEmbedded project: BitBake, the build engine, and OpenEmbedded-Core, the core set of recipes used to run the build process. All of the project's components are described in the next section.
Introducing the Yocto Project
As a collaboration project, sometimes called an "umbrella" project, the Yocto Project incorporates many different disparate pieces of the development process. These pieces are referred to as projects within the overall Yocto Project, and they include build tools, build instruction metadata called recipes, libraries, utilities, and graphical user interfaces (GUIs).
Poky is a reference build system for the Yocto Project. It includes BitBake, OpenEmbedded-Core, a board support package (BSP), and any other packages or layers incorporated into the build. The name Poky also refers to the default Linux distribution resulting from using the reference build system, which can be extremely minimal (core-image-minimal) or a full Linux system with a GUI (core-image-sato).
You can think of the Poky build system as a reference system for the entire project—a working example of the process in action. When you download the Yocto Project, you actually download an instance of those tools, utilities, libraries, toolchain, and metadata that you can use to build the default system, as described here. That reference system and the reference distribution it creates are both named Poky. You can also use this as a starting point to create your own distribution, which of course you can name anything you like.
One item that all build systems require is a toolchain: a compiler, assembler, linker, and other binary utilities necessary for creating binary executable files for a given architecture. Poky uses the GNU Compiler Collection (GCC), but you can specify other toolchains as well. Poky uses a technique known as cross-compilation: using a toolchain on one architecture to build binary executable files for a different architecture (for example, building an ARM distribution on an x86-based system). Developers often use cross-compilation in embedded systems development to take advantage of the host system's higher performance.
The metadata set is arranged in layers, such that each layer can provide separate functionality to the layers beneath it. The base layer is OpenEmbedded-Core, or
oe-core, which provides recipes, classes, and associated functions that are common and necessary for all builds. You can then customize builds by adding new layers on top of
OpenEmbedded-Core is co-maintained by the Yocto Project and the OpenEmbedded project. One layer that separates the Yocto Project from OpenEmbedded is the
meta-yocto layer, which provides the Poky distribution configuration and a core set of reference BSPs.
The OpenEmbedded project itself is a separate open source project with (largely) interchangeable recipes and similar goals to the Yocto Project, but different governance and scope.
Board support packages
A BSP contains the essential packages and drivers necessary for building Linux for a specific board or architecture. These are often maintained by the hardware manufacturers who make the boards. BSPs are the interface between the Linux operating system and the hardware that runs it. Note that it is also possible to create BSPs for virtual machines.
BitBake is a build engine. It reads recipes and follows them by fetching packages, building them, and incorporating the results into bootable images. BitBake is co-maintained by the Yocto Project and the OpenEmbedded project.
In an effort to make embedded Linux development easier, the Yocto Project provides a few different methods for working graphically. A relatively new addition to the project is called Hob, which provides a graphical front end to BitBake and the build process. Both are under continual development, complete with community user studies.
Open source license compliance
Complying with open source licenses is an extremely important part of any Linux development effort. One goal of the Yocto Project is to make compliance as easy as possible. It is quite easy to use the Yocto Project tools to create manifests and even to build entire source repositories, as well as filtering your build process to exclude packages that use specific licenses. The project is working with the Linux Foundation on its Open Compliance Program in relation to the Software Package Data Exchange® (SPDX™) specification.
Embedded GLIBC (EGLIBC) is a variant of the GNU C Library (GLIBC) that is designed to work well on embedded systems. EGLIBC's goals include reduced footprint, configurable components, and better support for cross-compilation and cross-testing. EGLIBC is under the Yocto Project umbrella but is maintained within its own governance structure.
Application Development Toolkit
The Application Development Toolkit (ADT) enables systems developers to provide software development kits (SDKs) for the distributions they create using the Yocto Project tools, providing applications developers a way to develop against the software stacks provided by those systems developers. The ADT includes a cross-compiling toolchain, debugging and profiling tools, and QEMU emulation and support scripts. The ADT also includes an Eclipse plug-in for those who like to work with integrated development environments (IDEs).
Other tools under the Yocto Project umbrella
Several other tools under the Yocto Project banner are:
- Autobuilder: Creates continuous automated builds of the Yocto Project tools, enabling automated Quality Assurance (QA) activities.
- Cross-Prelink: Provides prelinking for cross-compilation development environments, improving performance.
- Pseudo: Emulates root access, an essential part of building a bootable final image.
- Swabber: Detects when a cross-compilation build has been contaminated by host components.
- Build Appliance: Is a virtual machine that runs Hob, enabling those who use non-Linux build hosts to see the Yocto Project process firsthand. (Note: The Yocto Project build tools are currently supported on Linux only.)
Governance and community
One vital component for any open source project is the community who develops and supports its components. The Yocto Project has an active community that is made up in part by organizations—including hardware manufacturers, operating systems and tools providers, and electronics manufacturers, thus capturing the entire spectrum of embedded development—and in part by individuals, many of whom work for those organizations.
The project is managed by a chief architect and a set of maintainers and technical leaders, who make all technical decisions. The member organizations, many of which normally compete with each other, collaborate to form an Advisory Board, whose role, as the name suggests, is advisory in nature. This board manages the resources for the project, including infrastructure, advocacy and outreach, and finance.
Build a Linux distribution
This section shows how to build a basic embedded Linux system using Poky, the reference build system. The process described here builds the reference distribution as well as all of the tools needed to build that distribution. If you prefer, you can also download pre-built binaries to avoid the need for compiling. Find details in the Yocto Project Quick Start Guide (see Related topics).
The tools themselves are only supported on Linux platforms, specifically Ubuntu, Fedora, CentOS, and openSUSE. Note that the latest release of each of these systems is frequently not yet listed as being supported, though it often works anyway, and many other Linux distributions might also work.
If your host system does not run Linux, or if you'd just like to kick the tires without committing to installing anything, you can download the Yocto Project Build Appliance. This virtual machine has the latest official Yocto Project release pre-installed. The Build Appliance can run in either VMWare Player or VirtualBox, and it boots directly to the Hob GUI. If you use the Build Appliance, most of the following steps are already accomplished and you can skip to Test drive Hob. However, I recommend you still read the intervening sections so you understand what Hob is doing. Also, for the Hob documentation on the Yocto Project website, see Related topics.
Download the Yocto Project tools
Download the Yocto Project tools with either of two methods. Make sure to use a disk that has at least 50GB free. 100GB is recommended.
- You can download a
tarfile with the latest fully tested release from the Yocto Project downloads page. Download the
tarfile and then unpack it into a directory, as in Listing 1.
Listing 1. Downloading poky
$ wget \ http://downloads.yoctoproject.org/releases/yocto/yocto-1.2/poky-denzil-7.0.tar.bz2 $ tar xjf poky-denzil-7.0.tar.bz2 $ cd poky-denzil-7.0
- You can use
gitto grab the latest release (or any specific branch), although the master branch of development might be less stable than the tested release represented in the
tarfile. Listing 2 shows the use of
gitto download the latest bits.
Listing 2. Getting poky with git
$ git clone git://git.yoctoproject.org/poky.git $ cd poky
Note that the subdirectory in this case is simply called
pokywithout a release number, as it can be updated at any time using
Initialize your environment
To initialize your working environment, follow these steps:
First, install all the required development packages from your host system's software repository. If you already develop software on your host, you might already have most of these installed. Listings 3, 4, 5, and 6 show how to install the needed packages on various distributions.
Listing 3. Installing prerequisites on Ubuntu
$ sudo apt-get install sed wget subversion git-core coreutils \ unzip texi2html texinfo libsdl1.2-dev docbook-utils fop gawk \ python-pysqlite2 diffstat make gcc build-essential xsltproc \ g++ desktop-file-utils chrpath libgl1-mesa-dev libglu1-mesa-dev \ autoconf automake groff libtool xterm libxml-parser-perl
Listing 4. Installing prerequisites on Fedora
$ sudo yum groupinstall "development tools" $ sudo yum install python m4 make wget curl ftp tar bzip2 gzip \ unzip perl texinfo texi2html diffstat openjade \ docbook-style-dsssl sed docbook-style-xsl docbook-dtds fop xsltproc \ docbook-utils sed bc eglibc-devel ccache pcre pcre-devel quilt \ groff linuxdoc-tools patch cmake \ perl-ExtUtils-MakeMaker tcl-devel gettext chrpath ncurses apr \ SDL-devel mesa-libGL-devel mesa-libGLU-devel gnome-doc-utils \ autoconf automake libtool xterm
Listing 5. Installing prerequisites on CentOS
$ sudo yum -y groupinstall "development tools" $ sudo yum -y install tetex gawk sqlite-devel vim-common redhat-lsb xz \ m4 make wget curl ftp tar bzip2 gzip python-devel \ unzip perl texinfo texi2html diffstat openjade zlib-devel \ docbook-style-dsssl sed docbook-style-xsl docbook-dtds \ docbook-utils bc glibc-devel pcre pcre-devel \ groff linuxdoc-tools patch cmake \ tcl-devel gettext ncurses apr \ SDL-devel mesa-libGL-devel mesa-libGLU-devel gnome-doc-utils \ autoconf automake libtool xterm
Listing 6. Installing prerequisites on openSUSE
$ sudo zypper install python gcc gcc-c++ libtool fop \ subversion git chrpath automake make wget xsltproc \ diffstat texinfo freeglut-devel libSDL-devel
Set your shell's environment variables using the provided script. After you run this script, your working directory is the
buildsubdirectory, the location from which you can run your build.
$ cd poky $ . ./oe-init-build-env
Examine the main configuration file (conf/local.conf). By default, the configuration file is set up to create an image for
qemux86, or a QEMU instance emulating a 32-bit x86 processor. If you have a multiprocessor host, I highly recommend you uncomment the following parallelism options to speed up your build. For now, set both of these values to 2x the number of cores on your processor (for example, 8 for a 4-core processor).
BB_NUMBER_THREADS = "8" PARALLEL_MAKE = "-j 8"
Perform an initial build
The initial build uses the host's compiler to build the cross-compilation toolchain and any other needed build tools. BitBake also needs to download all the software packages, so it can take a while. When it is finished, you can find the resulting images in the tmp/deploy/images subdirectory.
Listing 7. Initial build images
$ ls tmp/deploy/images bzImage-3.2.11+gi...1.bin bzImage-qemux86.bin core-image-minimal-qemux86-20120506194741.rootfs.ext3 core-image-minimal-qemux86-20120506194741.rootfs.tar.bz2 core-image-minimal-qemux86.ext3 core-image-minimal-qemux86.tar.bz2 modules-3.2.11-yocto-standard-r1-qemux86.tgz README_-_DO_NOT_DELETE_FILES_IN_THIS_DIRECTORY.txt
Also, look in tmp/deploy/IMAGE/license.manifest. This is a list of all the licenses used in the entire image.
Boot your new image
When the build is finished, you can boot it using QEMU emulating a 32-bit x86 machine. To do this, simply run:
$ runqemu qemux86
You can log in to the machine with the username
root, with no password. Congratulations! You have built an embedded Linux distribution.
You can make many customizations to the final image, to the packages that make up the final image, and to the process itself. The build system is written in Python and is well documented.
Test drive Hob
Hob is a GUI that helps to automate some of the processes described earlier. The goal of Hob is to make the build process more visible and easier to understand. In reality, Hob is a front end for BitBake, and there is nothing you can do in Hob that you can't also do (and script) from the command line. However, Hob shows the decision-making process graphically and outlines exactly which packages are built into the final image. Some of Hob's other features include modifying existing images, saving customized images as templates, running images using QEMU, and deploying images to a Universal Serial Bus (USB) disk for live booting on a target device. Many other features are in the works for both functionality and usability.
To run Hob, simply type
hob from your build directory, and the main screen appears. Start by choosing an architecture (in this case,
qemux86). Hob then parses available recipes, as shown in Figure 1.
Figure 1. Hob image configuration
You can then choose a base image (the earlier example built
core-image-minimal, so those binaries already exist). Choose
core-image-minimal and then click View recipes or View packages. The screen in Figure 2 shows the scrolling lists in View packages. You can select or clear check boxes to include packages or remove them from the build, respectively. Click a column header to sort by that column.
Figure 2. Hob packages
On the Packages tab, note that Python is not included in the minimal build. You can add it by simply selecting the check box for
python-2.7.2-r2.14, as in Figure 3. All the dependencies and sub-packages are immediately included as well.
Figure 3. Adding a package in Hob
Click Build image to see Hob build a new image with your choices. As the build progresses, you can watch the logs to see how things are going or click the Issues tab to see if any problems arise.
You can then run the resulting image in a QEMU emulator by simply clicking Run Image, as in Figure 4.
Figure 4. Hob image details
And now you can check to make sure Python was actually included in the build, as in Figure 5.
Figure 5. Verifying your addition
More information, tutorials, and videos are available on the Yocto Project website (see Related topics).
Learning how to use the Yocto Project tools effectively can take some time, but it is time well spent. The flexible workflow, portable configurations, and high degree of configurability of the tools enables customization at every level of the build process. The Yocto Project represents a commercial workflow—it is designed and implemented by professional embedded systems developers. As of this writing, several operating system tools manufacturers are now refocusing their products based on the project. Take the tools for a spin, and if you like what you see, join the community.
- The Yocto Project website provides official Yocto Project documentation, descriptive text, videos, conference slides and proceedings, and governance information.
- The Yocto Project Quick Start is a detailed step-by-step guide to getting started with the Yocto Project.
- The Yocto Project Build Appliance is a good way to see the Yocto Project in action without installing any software.
- The Linux Foundation SPDX work group is part of the Open Compliance Program, whose goal is to enable companies and projects to easily exchange license and component information and comply with license agreements.
- EGLIBC is a variant of the GNU C library GLIBC designed to work well on embedded systems. EGLIBC is part of the Yocto Project.
- In the developerWorks Linux zone, find hundreds of how-to articles and tutorials, as well as downloads, discussion forums, and a wealth of other resources for Linux developers and administrators.
- Evaluate IBM products in the way that suits you best: Download a product trial, try a product online, use a product in a cloud environment, or spend a few hours in the SOA Sandbox learning how to implement Service Oriented Architecture efficiently.