IBM Platform lsmake is a parallel GNU make packaged with IBM Platform LSF and tightly integrated with LSF.
Platform lsmake includes two components: lsmake and lsmakerm.
lsmake is a command line tool that is fully compatible with GNU make.
lsmakerm is an lsmake server that is started by lsmake. lsmakerm uses the LSF API to perform slot allocation, task execution, and monitoring. lsmake is free for LSF users; no extra license is required.
You can use the gmake -j option to build workload in parallel on multiple processors of one host. lsmake is able to further distribute tasks to processors on multiple LSF hosts and run them in parallel, which can accelerate building large projects, such as Android source code.
This article shows you how to accelerate Android source builds by using lsmake.
Refer to "Initializing a Build Environment" (http://source.android.com/source/initializing.html) on the Android website to prepare the Android build environment. Refer to "Downloading the Source" (http://source.android.com/source/downloading.html) to download the Android source code.
To build Android source code on multiple hosts in parallel, you must download the source code to a shared directory.
Make sure the LSF RES daemon is up and running on all compute hosts.
In the following example build environment, there are 4 virtual machines, each host has 4 cores.
lsfuser@ubun01: lsload HOST_NAME status r15s r1m r15m ut pg ls it tmp swp mem ubun01 ok 0.0 0.0 0.1 0% 0.0 1 160 14G 1016M 1.2G ubun02 ok 0.0 0.0 0.1 0% 0.0 1 160 14G 1016M 1.2G ubun03 ok 0.0 0.0 0.1 0% 0.0 1 160 14G 1016M 1.2G ubun04 ok 0.0 0.0 0.1 0% 0.0 1 160 14G 1016M 1.2G lsfuser@ubun01: bhosts HOST_NAME STATUS JL/U MAX NJOBS RUN SSUSP USUSP RSV ubun01 ok - 4 0 0 0 0 0 ubun02 ok - 4 0 0 0 0 0 ubun03 ok - 4 0 0 0 0 0 ubun04 ok - 4 0 0 0 0 0
lsmake can run either in LSF as an LSF job, or run outside of LSF. In the following examples, lsmake is submitted to LSF as an LSF job, and LSF allocates hosts for build execution.
Submit lsmake as LSF job:
lsfuser@ubun01: bsub -n 4 -R "span[hosts=1]" -o %J.out -cwd /share/android lsmake --no-block-shell-mode Job <361> is submitted to default queue <normal>.
This job submission command line requests 4 slots from one host, specifies working directory /share/android, and redirects stdout to JOB_ID.out.
The --no-block-shell-mode is an lsmake option optimized for building Android source code. It can improve performance greatly for SHELL commands in your make files.
Check the job status
lsfuser@ubun01: bjobs
JOBID USER STAT QUEUE FROM_HOST EXEC_HOST JOB_NAME SUBMIT_TIME
361 lsfuser RUN normal ubun01 ubun01 * ock-mode Dec 8 12:44
ubun01
ubun01
ubun01
lsfuser@ubun01: bjobs -l 361
Job <361>, User <lsfuser>, Project <default>, Status <DONE>, Queue <normal>, Co
mmand <lsmake --no-block-shell-mode>,
Mon Dec 8 12:44:40: Submitted from host <ubun01>, CWD </home/lsfuser>;
Mon Dec 8 12:44:41: Started on <ubun01>, Execution Home </home/lsfuser>, Speci
fied CWD </share/android>;
Mon Dec 8 12:44:43: Done successfully. The CPU time used is 18,012 seconds.
SCHEDULING PARAMETERS:
r15s r1m r15m ut pg io ls it tmp swp mem
loadSched - - - - - - - - - - -
loadStop - - - - - - - - - - -
res1
loadSched -
loadStop -
RESOURCE REQUIREMENT DETAILS:
Combined: select[type == local] order[r15s:pg]
Effective: select[type == local] order[r15s:pg]
After the job is done, tail the output file to check whether the build was successful.
lsfuser@ubun01: tail 361.out Done! Install: out/target/product/generic/system/app/Gallery2.odex Install: out/target/product/generic/system/app/Gallery2.apk build/tools/generate-notice-files.py out/target/product/generic/obj/NOTICE.txt out/target/product/generic/obj/NOTICE.html "Notices for files contained in the filesystem images in this directory:" out/target/product/generic/obj/NOTICE_FILES/src Combining NOTICE files into HTML Combining NOTICE files into text Installed file list: out/target/product/generic/installed-files.txt Target system fs image: out/target/product/generic/obj/PACKAGING/systemimage_intermediates/system.img Running: mkyaffs2image -f out/target/product/generic/system out/target/product/generic/obj/PACKAGING/systemimage_intermediates/system.img out/target/product/generic/root/file_contexts system Install system fs image: out/target/product/generic/system.img
Submit lsmake as an LSF job:
lsfuser@ubun01: bsub -n 16 -R "span[ptile=4]" -o %J.out -cwd /share/android lsmake --no-block-shell-mode Job <362> is submitted to default queue <normal>.
This job submission command line requests 16 slots from 4 hosts.
|
gmake -j4 |
lsmake -j4 (1 host) |
lsmake -j8 (2 hosts) |
lsmake -j16 (4 hosts) |
|
1:25:21 |
1:28:02 |
49:31 |
28:19 |
When using one host, gmake took 1 hour 25 minutes to complete building, while lsmake on 1 host introduced a little overhead. When using 2 hosts and 4 hosts, lsmake only took 49 minutes and 28 minutes respectively.
This section lists other lsmake options that are related to large build workload. Refer to the IBM Platform LSF Command Reference for complete lsmake command usage.
On multiple hosts, the shared file system latency can prevent new files from bing accessed on another host immediately. This option sets a delay to allow commands in a target to finish, and commands in a dependent target to wait the specified time before starting on a different host. By default, the delay is 1 second. Slower file systems require a longer delay. If the dependent target's commands start on the same execution host, there is no delay. If retries are enabled with -x, the interval between retries also depends on the delay time.
Sets the environment variables for every task sent remotely. This option can introduce extra overhead, but this is necessary when make files change or override the environment variables they inherit at startup.
Creates the data file lsmake.dat and updates it each second, tracking the number of tasks running over time. This is useful if you want to export the data to third-party charting applications.
If the command fails, retries the command the specified number of times (for example, if the number of retries is 1, the command is attempted twice before exiting). This is useful to compensate for file system latency and minor errors. The interval between retries increases exponentially with each retry attempt. The time between the initial, failed attempt and the first retry is equal to 1 second by default, or equal to the buffer time specified by -a. For subsequent attempts, the interval between attempts is doubled each time.
Displays summary information after the job is done.
Performs child shell tasks without blocking. Without this parameter, blocking mode is used. You should use this option when building Android code.
NOTE: In lsmake, the -R option has different meaning from gmake. In lsmake, -R specifies resource requirements. In gmake, -R disables the built-in variable settings. So if your make file uses –R, change it to the long format option --no-builtin-variables, when building with lsmake.