Build custom embedded Linux distributions with the Yocto Project

Open source professional-grade tools for embedded Linux development

The Yocto Project™ is an open source project formed with the goal of making embedded Linux® development easier and more portable across architectures. This article introduces the project and provides a step-by-step walk through on how to get started.

Jeffrey M. Osier-Mixon, Yocto Project Community Manager, Intel Corporation

Author photo - Jeffrey Osier-MixonJeff "Jefro" Osier-Mixon has worked with open source and embedded systems for over 20 years as a technical writer, developer, developer's advocate, and community manager, and he has written several articles for IBM developerWorks, including Booting Linux on the BeagleBoard-xM and Open hardware: How and why it works. Jeff currently works for Intel Corporation in the role of Community Manager for the Yocto Project, and he maintains a calendar of open source conferences through his blog.



28 August 2012

Also available in Chinese Russian Japanese Portuguese

Overview

The Yocto Project is an open source collaboration project that provides templates, tools, and methods supporting custom Linux-based systems for embedded products, regardless of the hardware architecture. For those of you who are wondering about the name, the term yocto is the smallest SI unit. As a prefix, yocto indicates 10^-24.

This article provides a step-by-step walk through on how to use the Yocto Project’s industry-standard open source tools to create a customized Linux operating system for an embedded device and to boot the operating system in a virtual machine using QEMU. The Yocto Project, a Linux Foundation-sponsored open source project funded by major hardware companies and operating systems vendors, provides industry-class tools, methods, and metadata for building Linux systems.

Two major components of the Yocto Project are maintained in conjunction with the OpenEmbedded project: BitBake, the build engine, and OpenEmbedded-Core, the core set of recipes used to run the build process. All of the project's components are described in the next section.

Introducing the Yocto Project

As a collaboration project, sometimes called an "umbrella" project, the Yocto Project incorporates many different disparate pieces of the development process. These pieces are referred to as projects within the overall Yocto Project, and they include build tools, build instruction metadata called recipes, libraries, utilities, and graphical user interfaces (GUIs).

Poky

Poky is a reference build system for the Yocto Project. It includes BitBake, OpenEmbedded-Core, a board support package (BSP), and any other packages or layers incorporated into the build. The name Poky also refers to the default Linux distribution resulting from using the reference build system, which can be extremely minimal (core-image-minimal) or a full Linux system with a GUI (core-image-sato).

You can think of the Poky build system as a reference system for the entire project—a working example of the process in action. When you download the Yocto Project, you actually download an instance of those tools, utilities, libraries, toolchain, and metadata that you can use to build the default system, as described here. That reference system and the reference distribution it creates are both named Poky. You can also use this as a starting point to create your own distribution, which of course you can name anything you like.

One item that all build systems require is a toolchain: a compiler, assembler, linker, and other binary utilities necessary for creating binary executable files for a given architecture. Poky uses the GNU Compiler Collection (GCC), but you can specify other toolchains as well. Poky uses a technique known as cross-compilation: using a toolchain on one architecture to build binary executable files for a different architecture (for example, building an ARM distribution on an x86-based system). Developers often use cross-compilation in embedded systems development to take advantage of the host system's higher performance.

Metadata set

The metadata set is arranged in layers, such that each layer can provide separate functionality to the layers beneath it. The base layer is OpenEmbedded-Core, or oe-core, which provides recipes, classes, and associated functions that are common and necessary for all builds. You can then customize builds by adding new layers on top of oe-core.

OpenEmbedded-Core is co-maintained by the Yocto Project and the OpenEmbedded project. One layer that separates the Yocto Project from OpenEmbedded is the meta-yocto layer, which provides the Poky distribution configuration and a core set of reference BSPs.

The OpenEmbedded project itself is a separate open source project with (largely) interchangeable recipes and similar goals to the Yocto Project, but different governance and scope.

Board support packages

A BSP contains the essential packages and drivers necessary for building Linux for a specific board or architecture. These are often maintained by the hardware manufacturers who make the boards. BSPs are the interface between the Linux operating system and the hardware that runs it. Note that it is also possible to create BSPs for virtual machines.

BitBake

BitBake is a build engine. It reads recipes and follows them by fetching packages, building them, and incorporating the results into bootable images. BitBake is co-maintained by the Yocto Project and the OpenEmbedded project.

Hob

In an effort to make embedded Linux development easier, the Yocto Project provides a few different methods for working graphically. A relatively new addition to the project is called Hob, which provides a graphical front end to BitBake and the build process. Both are under continual development, complete with community user studies.

Open source license compliance

Complying with open source licenses is an extremely important part of any Linux development effort. One goal of the Yocto Project is to make compliance as easy as possible. It is quite easy to use the Yocto Project tools to create manifests and even to build entire source repositories, as well as filtering your build process to exclude packages that use specific licenses. The project is working with the Linux Foundation on its Open Compliance Program in relation to the Software Package Data Exchange® (SPDX™) specification.

EGLIBC

Embedded GLIBC (EGLIBC) is a variant of the GNU C Library (GLIBC) that is designed to work well on embedded systems. EGLIBC's goals include reduced footprint, configurable components, and better support for cross-compilation and cross-testing. EGLIBC is under the Yocto Project umbrella but is maintained within its own governance structure.

Application Development Toolkit

The Application Development Toolkit (ADT) enables systems developers to provide software development kits (SDKs) for the distributions they create using the Yocto Project tools, providing applications developers a way to develop against the software stacks provided by those systems developers. The ADT includes a cross-compiling toolchain, debugging and profiling tools, and QEMU emulation and support scripts. The ADT also includes an Eclipse plug-in for those who like to work with integrated development environments (IDEs).

Other tools under the Yocto Project umbrella

Several other tools under the Yocto Project banner are:

  • Autobuilder: Creates continuous automated builds of the Yocto Project tools, enabling automated Quality Assurance (QA) activities.
  • Cross-Prelink: Provides prelinking for cross-compilation development environments, improving performance.
  • Pseudo: Emulates root access, an essential part of building a bootable final image.
  • Swabber: Detects when a cross-compilation build has been contaminated by host components.
  • Build Appliance: Is a virtual machine that runs Hob, enabling those who use non-Linux build hosts to see the Yocto Project process firsthand. (Note: The Yocto Project build tools are currently supported on Linux only.)

Governance and community

One vital component for any open source project is the community who develops and supports its components. The Yocto Project has an active community that is made up in part by organizations—including hardware manufacturers, operating systems and tools providers, and electronics manufacturers, thus capturing the entire spectrum of embedded development—and in part by individuals, many of whom work for those organizations.

The project is managed by a chief architect and a set of maintainers and technical leaders, who make all technical decisions. The member organizations, many of which normally compete with each other, collaborate to form an Advisory Board, whose role, as the name suggests, is advisory in nature. This board manages the resources for the project, including infrastructure, advocacy and outreach, and finance.


Build a Linux distribution

This section shows how to build a basic embedded Linux system using Poky, the reference build system. The process described here builds the reference distribution as well as all of the tools needed to build that distribution. If you prefer, you can also download pre-built binaries to avoid the need for compiling. Find details in the Yocto Project Quick Start Guide (see Resources).

The tools themselves are only supported on Linux platforms, specifically Ubuntu, Fedora, CentOS, and openSUSE. Note that the latest release of each of these systems is frequently not yet listed as being supported, though it often works anyway, and many other Linux distributions might also work.

If your host system does not run Linux, or if you'd just like to kick the tires without committing to installing anything, you can download the Yocto Project Build Appliance. This virtual machine has the latest official Yocto Project release pre-installed. The Build Appliance can run in either VMWare Player or VirtualBox, and it boots directly to the Hob GUI. If you use the Build Appliance, most of the following steps are already accomplished and you can skip to Test drive Hob. However, I recommend you still read the intervening sections so you understand what Hob is doing. Also, for the Hob documentation on the Yocto Project website, see Resources.

Download the Yocto Project tools

Download the Yocto Project tools with either of two methods. Make sure to use a disk that has at least 50GB free. 100GB is recommended.

  • You can download a tar file with the latest fully tested release from the Yocto Project downloads page. Download the tar file and then unpack it into a directory, as in Listing 1.
    Listing 1. Downloading poky
    $ wget \
    http://downloads.yoctoproject.org/releases/yocto/yocto-1.2/poky-denzil-7.0.tar.bz2
    $ tar xjf poky-denzil-7.0.tar.bz2
    $ cd poky-denzil-7.0
  • You can use git to grab the latest release (or any specific branch), although the master branch of development might be less stable than the tested release represented in the tar file. Listing 2 shows the use of git to download the latest bits.
    Listing 2. Getting poky with git
    $ git clone git://git.yoctoproject.org/poky.git
    $ cd poky

    Note that the subdirectory in this case is simply called poky without a release number, as it can be updated at any time using git.

Initialize your environment

To initialize your working environment, follow these steps:

  1. First, install all the required development packages from your host system's software repository. If you already develop software on your host, you might already have most of these installed. Listings 3, 4, 5, and 6 show how to install the needed packages on various distributions.

    Listing 3. Installing prerequisites on Ubuntu
         $ sudo apt-get install sed wget subversion git-core coreutils \
         unzip texi2html texinfo libsdl1.2-dev docbook-utils fop gawk \
         python-pysqlite2 diffstat make gcc build-essential xsltproc \
         g++ desktop-file-utils chrpath libgl1-mesa-dev libglu1-mesa-dev \
         autoconf automake groff libtool xterm libxml-parser-perl
    Listing 4. Installing prerequisites on Fedora
         $ sudo yum groupinstall "development tools"
         $ sudo yum install python m4 make wget curl ftp tar bzip2 gzip \
         unzip perl texinfo texi2html diffstat openjade \
         docbook-style-dsssl sed docbook-style-xsl docbook-dtds fop xsltproc \
         docbook-utils sed bc eglibc-devel ccache pcre pcre-devel quilt \
         groff linuxdoc-tools patch cmake \
         perl-ExtUtils-MakeMaker tcl-devel gettext chrpath ncurses apr \
         SDL-devel mesa-libGL-devel mesa-libGLU-devel gnome-doc-utils \
         autoconf automake libtool xterm
    Listing 5. Installing prerequisites on CentOS
         $ sudo yum -y groupinstall "development tools"
         $ sudo yum -y install tetex gawk sqlite-devel vim-common redhat-lsb xz \
           m4 make wget curl ftp tar bzip2 gzip python-devel \
           unzip perl texinfo texi2html diffstat openjade zlib-devel \
           docbook-style-dsssl sed docbook-style-xsl docbook-dtds \
           docbook-utils bc glibc-devel pcre pcre-devel \
           groff linuxdoc-tools patch cmake \
           tcl-devel gettext ncurses apr \
           SDL-devel mesa-libGL-devel mesa-libGLU-devel gnome-doc-utils \
           autoconf automake libtool xterm
    Listing 6. Installing prerequisites on openSUSE
         $ sudo zypper install python gcc gcc-c++ libtool fop \
         subversion git chrpath automake make wget xsltproc \
         diffstat texinfo freeglut-devel libSDL-devel
  2. Set your shell's environment variables using the provided script. After you run this script, your working directory is the build subdirectory, the location from which you can run your build.

    $ cd poky 
    $ . ./oe-init-build-env
  3. Examine the main configuration file (conf/local.conf). By default, the configuration file is set up to create an image for qemux86, or a QEMU instance emulating a 32-bit x86 processor. If you have a multiprocessor host, I highly recommend you uncomment the following parallelism options to speed up your build. For now, set both of these values to 2x the number of cores on your processor (for example, 8 for a 4-core processor).

    BB_NUMBER_THREADS = "8"
    PARALLEL_MAKE = "-j 8"

Perform an initial build

The initial build uses the host's compiler to build the cross-compilation toolchain and any other needed build tools. BitBake also needs to download all the software packages, so it can take a while. When it is finished, you can find the resulting images in the tmp/deploy/images subdirectory.

Listing 7. Initial build images
$ ls tmp/deploy/images
bzImage-3.2.11+gi...1.bin
bzImage-qemux86.bin
core-image-minimal-qemux86-20120506194741.rootfs.ext3
core-image-minimal-qemux86-20120506194741.rootfs.tar.bz2
core-image-minimal-qemux86.ext3
core-image-minimal-qemux86.tar.bz2
modules-3.2.11-yocto-standard-r1-qemux86.tgz
README_-_DO_NOT_DELETE_FILES_IN_THIS_DIRECTORY.txt

Also, look in tmp/deploy/IMAGE/license.manifest. This is a list of all the licenses used in the entire image.

Boot your new image

When the build is finished, you can boot it using QEMU emulating a 32-bit x86 machine. To do this, simply run:

$ runqemu qemux86

You can log in to the machine with the username root, with no password. Congratulations! You have built an embedded Linux distribution.

You can make many customizations to the final image, to the packages that make up the final image, and to the process itself. The build system is written in Python and is well documented.

Test drive Hob

Hob is a GUI that helps to automate some of the processes described earlier. The goal of Hob is to make the build process more visible and easier to understand. In reality, Hob is a front end for BitBake, and there is nothing you can do in Hob that you can't also do (and script) from the command line. However, Hob shows the decision-making process graphically and outlines exactly which packages are built into the final image. Some of Hob's other features include modifying existing images, saving customized images as templates, running images using QEMU, and deploying images to a Universal Serial Bus (USB) disk for live booting on a target device. Many other features are in the works for both functionality and usability.

To run Hob, simply type hob from your build directory, and the main screen appears. Start by choosing an architecture (in this case, qemux86). Hob then parses available recipes, as shown in Figure 1.

Figure 1. Hob image configuration
Screen capture showing image configuration screen in progress

You can then choose a base image (the earlier example built core-image-minimal, so those binaries already exist). Choose core-image-minimal and then click View recipes or View packages. The screen in Figure 2 shows the scrolling lists in View packages. You can select or clear check boxes to include packages or remove them from the build, respectively. Click a column header to sort by that column.

Figure 2. Hob packages
Screen capture of package selection screen in HOB

On the Packages tab, note that Python is not included in the minimal build. You can add it by simply selecting the check box for python-2.7.2-r2.14, as in Figure 3. All the dependencies and sub-packages are immediately included as well.

Figure 3. Adding a package in Hob
Screen capture of adding packages to HOB

Click Build image to see Hob build a new image with your choices. As the build progresses, you can watch the logs to see how things are going or click the Issues tab to see if any problems arise.

You can then run the resulting image in a QEMU emulator by simply clicking Run Image, as in Figure 4.

Figure 4. Hob image details
Screen capture of HOB showing details for an image

And now you can check to make sure Python was actually included in the build, as in Figure 5.

Figure 5. Verifying your addition
Screen capture of console showing a successful entry into the python prompt

More information, tutorials, and videos are available on the Yocto Project website (see Resources).


Conclusion

Learning how to use the Yocto Project tools effectively can take some time, but it is time well spent. The flexible workflow, portable configurations, and high degree of configurability of the tools enables customization at every level of the build process. The Yocto Project represents a commercial workflow—it is designed and implemented by professional embedded systems developers. As of this writing, several operating system tools manufacturers are now refocusing their products based on the project. Take the tools for a spin, and if you like what you see, join the community.

Resources

Learn

Get products and technologies

  • The Yocto Project website provides official Yocto Project documentation, descriptive text, videos, conference slides and proceedings, and governance information.
  • The Yocto Project Build Appliance is a good way to see the Yocto Project in action without installing any software.
  • EGLIBC is a variant of the GNU C library GLIBC designed to work well on embedded systems. EGLIBC is part of the Yocto Project.
  • Evaluate IBM products in the way that suits you best: Download a product trial, try a product online, use a product in a cloud environment, or spend a few hours in the SOA Sandbox learning how to implement Service Oriented Architecture efficiently.

Discuss

Comments

developerWorks: Sign in

Required fields are indicated with an asterisk (*).


Need an IBM ID?
Forgot your IBM ID?


Forgot your password?
Change your password

By clicking Submit, you agree to the developerWorks terms of use.

 


The first time you sign into developerWorks, a profile is created for you. Information in your profile (your name, country/region, and company name) is displayed to the public and will accompany any content you post, unless you opt to hide your company name. You may update your IBM account at any time.

All information submitted is secure.

Choose your display name



The first time you sign in to developerWorks, a profile is created for you, so you need to choose a display name. Your display name accompanies the content you post on developerWorks.

Please choose a display name between 3-31 characters. Your display name must be unique in the developerWorks community and should not be your email address for privacy reasons.

Required fields are indicated with an asterisk (*).

(Must be between 3 – 31 characters.)

By clicking Submit, you agree to the developerWorks terms of use.

 


All information submitted is secure.

Dig deeper into Linux on developerWorks


static.content.url=http://www.ibm.com/developerworks/js/artrating/
SITE_ID=1
Zone=Linux, Open source
ArticleID=831581
ArticleTitle=Build custom embedded Linux distributions with the Yocto Project
publish-date=08282012