lsmake

Runs LSF make tasks in parallel.

Synopsis

lsmake [-m "host_name [num_cores] [host_name [num_cores]] ...]"] [-a seconds] [-c num_tasks] [-E] [-G debug_level] [-T] [-u] [-V] [-x num_retries] [-y] [make_option ...] [--no-block-shell-mode] [target ...]

lsmake [-R res_req] [-j max_cores] [-a seconds] [-c num_tasks] [-E] [-G debug_level] [-T] [-u] [-V] [-x num_retries] [-y] [make_option ...] [--no-block-shell-mode] [target ...]

lsmake [-h]

Description

Runs make tasks in parallel on LSF hosts. Sets the environment variables on the remote hosts when the lsmake command starts.

By default, uses the local host, uses only one core, starts only one task in each core, processes submakes sequentially, allows 1-second buffer time to compensate for file system latency, and does not retry if the job fails. The lsmake command is a modified version of the GNU make command.

Options

-a seconds

When commands in a target finish, commands in a dependent target wait the specified time before they start on a different host. This delay allows time for the shared file system to synchronize client and server, and compensates for file system latency. By default, the delay is 1 second. Slower file systems require a longer delay.

If the dependent target's commands start on the same execution host, there is no delay.

If retries are enabled with -x, the interval between retries also depends on the delay time.

-c num_tasks

Starts the specified number of tasks concurrently on each core. If you specify too many tasks, you might overload a host.

-E

Sets the environment variables for every task sent remotely.

Setting environment variables is necessary when makefiles change or override the environment variables that they inherit at startup.

-G debug_level

Enables debugging, specify the debug level.

-j max_cores

Uses multiple cores, selecting the best available. Specify the maximum number of cores to use.

Not compatible with -m "host_name [num_cores] [host_name [num_cores]] ..." option.

Ignored if you use the bsub command to run the lsmake command.

-m "host_name [num_cores] [host_name [num_cores]] ..."

Uses the specified hosts. To use multiple cores on a host, specify the number of cores after the host name.

Not compatible with -R res_req and -j max_cores.

Ignored if you use bsub to run lsmake.

-R res_req

Uses only hosts that satisfy the specified resource requirements.

When you specify -R but not -j, uses one core on one host that satisfies the resource requirements.

If the group of hosts that match the selection string includes the submission host, the submission host is always selected. The policies that are defined by the order string affect only the other hosts.

Not compatible with the -m "host_name [num_cores] [host_name [num_cores]] ..." option.

Ignored if you use bsub to run lsmake.

-T

Enables output tagging to prefix the task ID of the sender to the parallel task output data.

-u

Creates the data file lsmake.dat and updates it each second, tracking the number of running tasks over time.

The lsmake.dat file is useful if you want to export the data to third-party charting applications.

-V

Verbose mode. Prints the names of the hosts used.

-x num_retries

If the command fails, retries the command the specified number of times. For example, if the number of retries is 1, the command is attempted twice before it exits. Setting retries is useful to compensate for file system latency and minor errors.

The interval between retries increases exponentially with each retry attempt. The time between the initial, failed attempt, and the first retry is equal to 1 second by default, or equal to the buffer time specified by -a. For subsequent attempts, the interval between attempts is doubled each time.

-y

Displays summary information after the job is done.

make_option ...

Specifies standard GNU Make options.

Note: The -j and -R options are not supported as a GNU make options, see the lsmake options -j max_cores and -R res_req. See GNU documentation for detailed descriptions of other options.

The current version of the lsmake command supports GNU Make version 3.81, which includes the following options:

-b, -m: Ignored for compatibility.
-B, --always-make: Unconditionally make all targets.
-C dir , --directory=dir: Change directory before reading the makefile.
-d: Print all debugging information.
--debug[=options]: Print basic debugging information, or specify what types of information to print (all, basic, verbose, implicit, jobs, makefile).
-e, --environment-overrides: Environment variables override makefiles.
-f file, --file=file, --makefile=file: Specify the makefile.
-h, --help: Print usage and exit.
-i, --ignore-errors: Ignore errors.
-I dir, --include-dir=dir: Search a directory for included makefiles.
-k, --keep-going: Keep going when some targets cannot be made.
-l [n], --load-average[=n], --max-load[=n]: Obsolete. Load limit.
-L , --check-symlink-times: Target file modification time considers the time stamp of symbolic links also.
-n, --just-print, --dry-run, --recon: Print instead of running.
-o file, --old-file=file, --assume-old=file: Do not remake the old file.
-p, --print-data-base: Print make's internal database.
-q, --question: Question mode, return exit status.
-r, --no-builtin-rules: Disable the built-in implicit rules.
--no-builtin-variables: Disable the built-in variable settings. The make command option -R is not supported, it conflicts with the lsmake command option -R res_req.
-s, --silent, --quiet: Silent mode, do not echo commands.
-S, --no-keep-going, --stop: Turns off the -k option.
-t, --touch: Touch targets (change modification time) instead of remaking them.
-v, --version: Print the version number of make and exit.
-w, --print-directory: Print the current directory.
--no-print-directory: Turn off -w, even if it was turned on implicitly.
-W file, --what-if=file, --new-file=file, --assume-new=file: Always consider the file to be new. Do not change modification time.
--warn-undefined-variables: Warn when an undefined variable is referenced.

--no-block-shell-mode

Run child shell tasks without blocking mode. Without this parameter, blocking mode is used. Allows the lsmake command to build customized Android 4.3 code.

target ...

Specifies targets to make.

Output with the -y option

Total Run Time: Total lsmake job run time, in the format hh:mm:ss.
Most Concurrent Tasks: Maximum number of tasks that ran simultaneously. Compare to Total Slots Allocated and Tasks Allocated per Slot to determine whether parallel execution might be limited by resource availability.
Retries Allowed: Maximum number of retries allowed. Set by the lsmake -x option.
Hosts and Number of Slots Allocated: The output is a single line that shows each name and number pair that is separated by spaces, in the format: host_name number_slots [host_name number_slots] ...
Tasks Allowed per Slot: Maximum number of tasks that are allowed per slot. Set by the lsmake -c option.
Total Slots Allocated: Total number of slots allocated. Might be limited by the lsmake -j or lsmake -m option.

Output with the -u option

The lsmake.dat file is a simple text file, consisting of two values separated by a comma. The first value is the time in the format hh:mm:s. The second value is the number of running tasks. For example:

23:13:39,2

The file is updated with a new line of information every second.

Limitations

If a submake in a makefile specifies options that are specific to the lsmake command, they are ignored. Only the command-line options are used. The resource requirements of tasks in the remote task list are not considered when lsmake dispatches tasks.