Using lsmake to accelerate building Android source code
The LSF lsmake tool is a parallel GNU make that is tightly integrated with LSF.
- lsmake is a command line tool that is fully compatible with GNU make.
- lsmakerm is an lsmake server that is started by
lsmake. lsmakerm uses the LSF API to perform slot allocation,
task execution, and monitoring. lsmake is free for LSF users; no extra license is
required.
You can use the gmake -j option to build workload in parallel on multiple processors of one host. lsmake is able to further distribute tasks to processors on multiple LSF hosts and run them in parallel, which can accelerate building large projects, such as Android source code.
This article shows you how to accelerate Android source builds by using lsmake.
Preparing the Android building environment
For more information on preparing the Android environment, refer to Initializing a Build Environment (http://source.android.com/source/initializing.html) on the Android website.
For more information on downloading the Android source code, refer to Downloading the Source (http://source.android.com/source/downloading.html) on the Android website.
To build Android source code on multiple hosts in parallel, you must download the source code to a shared directory.
Preparing the LSF environment
Make sure the LSF RES daemon is up and running on all compute hosts.
In the following example build environment, there are 4 virtual machines, and each host has 4 cores.
lsfuser@ubun01: lsload
HOST_NAME status r15s r1m r15m ut pg ls it tmp swp mem
ubun01 ok 0.0 0.0 0.1 0% 0.0 1 160 14G 1016M 1.2G
ubun02 ok 0.0 0.0 0.1 0% 0.0 1 160 14G 1016M 1.2G
ubun03 ok 0.0 0.0 0.1 0% 0.0 1 160 14G 1016M 1.2G
ubun04 ok 0.0 0.0 0.1 0% 0.0 1 160 14G 1016M 1.2G
lsfuser@ubun01: bhosts
HOST_NAME STATUS JL/U MAX NJOBS RUN SSUSP USUSP RSV
ubun01 ok - 4 0 0 0 0 0
ubun02 ok - 4 0 0 0 0 0
ubun03 ok - 4 0 0 0 0 0
ubun04 ok - 4 0 0 0 0 0Building Android source code using lsmake
The lsmake tool can run either in LSF as an LSF job, or run outside of LSF. In the following examples, lsmake is submitted to LSF as an LSF job, and LSF allocates hosts for build execution.
Building Android source code on one host
Submit lsmake as an LSF job:
lsfuser@ubun01: bsub -n 4 -R "span[hosts=1]" -o %J.out -cwd /share/android lsmake --no-block-shell-mode
Job <361> is submitted to default queue <normal>.This job submission command line requests 4 slots from one host, specifies working directory /share/android, and redirects stdout to JOB_ID.out.
The --no-block-shell-mode is an lsmake option optimized for building Android source code. It can improve performance greatly for SHELL commands in your make files.
Check the job status:
lsfuser@ubun01: bjobs
JOBID USER STAT QUEUE FROM_HOST EXEC_HOST JOB_NAME SUBMIT_TIME
361 lsfuser RUN normal ubun01 ubun01 * ock-mode Dec 8 12:44
ubun01
ubun01
ubun01
lsfuser@ubun01: bjobs -l 361
Job <361>, User <lsfuser>, Project <default>, Status <DONE>, Queue <normal>, Co
mmand <lsmake --no-block-shell-mode>,
Mon Dec 8 12:44:40: Submitted from host <ubun01>, CWD </home/lsfuser>;
Mon Dec 8 12:44:41: Started on <ubun01>, Execution Home </home/lsfuser>, Speci
fied CWD </share/android>;
Mon Dec 8 12:44:43: Done successfully. The CPU time used is 18,012 seconds.
SCHEDULING PARAMETERS:
r15s r1m r15m ut pg io ls it tmp swp mem
loadSched - - - - - - - - - - -
loadStop - - - - - - - - - - -
res1
loadSched -
loadStop -
RESOURCE REQUIREMENT DETAILS:
Combined: select[type == local] order[r15s:pg]
Effective: select[type == local] order[r15s:pg]After the job is done, tail the output file to check whether the build was successful.
lsfuser@ubun01: tail 361.out
Done!
Install: out/target/product/generic/system/app/Gallery2.odex
Install: out/target/product/generic/system/app/Gallery2.apk
build/tools/generate-notice-files.py out/target/product/generic/obj/NOTICE.txt out/target/product/generic/obj/NOTICE.html "Notices for files contained in the filesystem images in this directory:" out/target/product/generic/obj/NOTICE_FILES/src
Combining NOTICE files into HTML
Combining NOTICE files into text
Installed file list: out/target/product/generic/installed-files.txt
Target system fs image: out/target/product/generic/obj/PACKAGING/systemimage_intermediates/system.img
Running: mkyaffs2image -f out/target/product/generic/system out/target/product/generic/obj/PACKAGING/systemimage_intermediates/system.img out/target/product/generic/root/file_contexts system
Install system fs image: out/target/product/generic/system.imgBuilding Android source code on multiple hosts
Submit lsmake as an LSF job:
lsfuser@ubun01: bsub -n 16 -R "span[ptile=4]" -o %J.out -cwd /share/android lsmake --no-block-shell-mode
Job <362> is submitted to default queue <normal>.This job submission command line requests 16 slots from 4 hosts.
Comparison of lsmake and gmake
The following is a comparison of build times of the various make command options for the same source code:
| Make command | Build time |
|---|---|
| gmake -j4 | 1:25:21 |
| lsmake -j4 (1 host) | 1:28:02 |
| lsmake -j8 (2 hosts) | 49:31 |
| lsmake -j16 (4 hosts) | 28>19 |
When using one host, gmake took 1 hour 25 minutes to complete building, while lsmake on 1 host introduced a little overhead. When using 2 hosts and 4 hosts, lsmake only took 49 minutes and 28 minutes respectively.
Other lsmake options
This section lists other lsmake options that are related to large build workload. For complete lsmake command usage, refer to lsmake.
- -a seconds
- On multiple hosts, the shared file system latency can prevent new files from being accessed on another host immediately. This option sets a delay to allow commands in a target to finish, and commands in a dependent target to wait the specified time before starting on a different host. By default, the delay is 1 second. Slower file systems require a longer delay. If the dependent target's commands start on the same execution host, there is no delay. If retries are enabled with -x, the interval between retries also depends on the delay time.
- -E
- Sets the environment variables for every task sent remotely. This option can introduce extra overhead, but this is necessary when make files change or override the environment variables they inherit at startup.
- -u
- Creates the data file lsmake.dat and updates it each second, tracking the number of tasks running over time. This is useful if you want to export the data to third-party charting applications.
- -x num_retries
- If the command fails, retries the command the specified number of times (for example, if the number of retries is 1, the command is attempted twice before exiting). This is useful to compensate for file system latency and minor errors. The interval between retries increases exponentially with each retry attempt. The time between the initial, failed attempt and the first retry is equal to 1 second by default, or equal to the buffer time specified by -a. For subsequent attempts, the interval between attempts is doubled each time.
- -y
- Displays summary information after the job is done.
- --no-block-shell-mode
- Performs child shell tasks without blocking. Without this parameter, blocking mode is used. You should use this option when building Android code.