Using IBM Platform lsmake to accelerate building Android source code

IBM Platform lsmake is a parallel GNU make packaged with IBM Platform LSF and tightly integrated with LSF.

Platform lsmake includes two components: lsmake and lsmakerm.


lsmake is a command line tool that is fully compatible with GNU make.


lsmakerm is an lsmake server that is started by lsmake. lsmakerm uses the LSF API to perform slot allocation, task execution, and monitoring. lsmake is free for LSF users; no extra license is required.

You can use the gmake -j option to build workload in parallel on multiple processors of one host. lsmake is able to further distribute tasks to processors on multiple LSF hosts and run them in parallel, which can accelerate building large projects, such as Android source code.

This article shows you how to accelerate Android source builds by using lsmake.


Preparing the Android building environment

Refer to "Initializing a Build Environment" (http://source.android.com/source/initializing.html) on the Android website to prepare the Android build environment. Refer to "Downloading the Source" (http://source.android.com/source/downloading.html) to download the Android source code.

To build Android source code on multiple hosts in parallel, you must download the source code to a shared directory.


Preparing the LSF environment

Make sure the LSF RES daemon is up and running on all compute hosts.

In the following example build environment, there are 4 virtual machines, each host has 4 cores.

lsfuser@ubun01: lsload
HOST_NAME       status  r15s   r1m  r15m   ut    pg  ls    it   tmp   swp   mem
ubun01              ok   0.0   0.0   0.1   0%   0.0   1   160   14G 1016M  1.2G
ubun02              ok   0.0   0.0   0.1   0%   0.0   1   160   14G 1016M  1.2G
ubun03              ok   0.0   0.0   0.1   0%   0.0   1   160   14G 1016M  1.2G
ubun04              ok   0.0   0.0   0.1   0%   0.0   1   160   14G 1016M  1.2G

lsfuser@ubun01: bhosts
ubun01             ok              -      4      0      0      0      0      0
ubun02             ok              -      4      0      0      0      0      0
ubun03             ok              -      4      0      0      0      0      0
ubun04             ok              -      4      0      0      0      0      0


Building Android source code using lsmake

lsmake can run either in LSF as an LSF job, or run outside of LSF. In the following examples, lsmake is submitted to LSF as an LSF job, and LSF allocates hosts for build execution.


Building Android source code on one host

Submit lsmake as LSF job:

lsfuser@ubun01: bsub -n 4 -R "span[hosts=1]" -o %J.out -cwd /share/android lsmake --no-block-shell-mode
Job <361> is submitted to default queue <normal>.

This job submission command line requests 4 slots from one host, specifies working directory /share/android, and redirects stdout to JOB_ID.out.

The --no-block-shell-mode is an lsmake option optimized for building Android source code. It can improve performance greatly for SHELL commands in your make files.

Check the job status

lsfuser@ubun01: bjobs
361  lsfuser  RUN  normal     ubun01      ubun01      * ock-mode Dec 8 12:44

lsfuser@ubun01: bjobs -l 361
Job <361>, User <lsfuser>, Project <default>, Status <DONE>, Queue <normal>, Co
                     mmand <lsmake --no-block-shell-mode>,
Mon Dec  8 12:44:40: Submitted from host <ubun01>, CWD </home/lsfuser>;
Mon Dec  8 12:44:41: Started on <ubun01>, Execution Home </home/lsfuser>, Speci
                     fied CWD </share/android>;
Mon Dec  8 12:44:43: Done successfully. The CPU time used is 18,012 seconds.

           r15s   r1m  r15m   ut      pg    io   ls    it    tmp    swp    mem
 loadSched   -     -     -     -       -     -    -     -     -      -      -
 loadStop    -     -     -     -       -     -    -     -     -      -      -
 loadSched   -
 loadStop    -

 Combined: select[type == local] order[r15s:pg]
 Effective: select[type == local] order[r15s:pg]

After the job is done, tail the output file to check whether the build was successful.

lsfuser@ubun01: tail 361.out
Install: out/target/product/generic/system/app/Gallery2.odex
Install: out/target/product/generic/system/app/Gallery2.apk
build/tools/generate-notice-files.py  out/target/product/generic/obj/NOTICE.txt  out/target/product/generic/obj/NOTICE.html "Notices for files contained in the filesystem images in this directory:" out/target/product/generic/obj/NOTICE_FILES/src
Combining NOTICE files into HTML
Combining NOTICE files into text
Installed file list: out/target/product/generic/installed-files.txt
Target system fs image: out/target/product/generic/obj/PACKAGING/systemimage_intermediates/system.img
Running:  mkyaffs2image -f out/target/product/generic/system out/target/product/generic/obj/PACKAGING/systemimage_intermediates/system.img out/target/product/generic/root/file_contexts system
Install system fs image: out/target/product/generic/system.img


Building Android source code on multiple hosts

Submit lsmake as an LSF job:

lsfuser@ubun01: bsub -n 16 -R "span[ptile=4]" -o %J.out -cwd /share/android lsmake --no-block-shell-mode

Job <362> is submitted to default queue <normal>.

This job submission command line requests 16 slots from 4 hosts.


Comparison of lsmake and gmake

gmake -j4

lsmake -j4 (1 host)

lsmake -j8 (2 hosts)

lsmake -j16 (4 hosts)





When using one host, gmake took 1 hour 25 minutes to complete building, while lsmake on 1 host introduced a little overhead. When using 2 hosts and 4 hosts, lsmake only took 49 minutes and 28 minutes respectively.


Other lsmake options

This section lists other lsmake options that are related to large build workload. Refer to the IBM Platform LSF Command Reference for complete lsmake command usage.

-a seconds

On multiple hosts, the shared file system latency can prevent new files from bing accessed on another host immediately. This option sets a delay to allow commands in a target to finish, and commands in a dependent target to wait the specified time before starting on a different host. By default, the delay is 1 second. Slower file systems require a longer delay. If the dependent target's commands start on the same execution host, there is no delay. If retries are enabled with -x, the interval between retries also depends on the delay time.


Sets the environment variables for every task sent remotely. This option can introduce extra overhead, but this is necessary when make files change or override the environment variables they inherit at startup.


Creates the data file lsmake.dat and updates it each second, tracking the number of tasks running over time. This is useful if you want to export the data to third-party charting applications.

-x num_retries

If the command fails, retries the command the specified number of times (for example, if the number of retries is 1, the command is attempted twice before exiting). This is useful to compensate for file system latency and minor errors. The interval between retries increases exponentially with each retry attempt. The time between the initial, failed attempt and the first retry is equal to 1 second by default, or equal to the buffer time specified by -a. For subsequent attempts, the interval between attempts is doubled each time.


Displays summary information after the job is done.


Performs child shell tasks without blocking. Without this parameter, blocking mode is used. You should use this option when building Android code.


NOTE: In lsmake, the -R option has different meaning from gmake. In lsmake, -R specifies resource requirements. In gmake, -R disables the built-in variable settings. So if your make file uses –R, change it to the long format option --no-builtin-variables, when building with lsmake.