jsrun command



jsrun[option] [command]


The jsrun command is a tool within IBM® Job Step Manager (JSM) software package that is used for managing an allocation that is provided by an external resource manager. You can use this command to specify requirements for a subset of the resources that are allocated by a resource manager.

The jsrun command provides features for stdio management, signal propagation, and task termination cleanup. This command starts parallel tasks within the context of a PMIx server that allows processes to register as PMIx clients and access PMIx functions such as publishing of data and event notification, peer identification, and data exchange.

An allocation request must specify a number of CPUs. An allocation request can also specify GPU and memory resources, and can request information about how resources are selected from available sockets and nodes. A job step launch request must specify the number of tasks to start.

The following are terms that are used in JSM


Requesting resource sets
You can use the following options for the jsrun command to determine resource sets

Explicit specification of resource sets
JSM normal command line options allow the user to describe the number of CPU, GPU and memory to assign to resource sets. These normal command line options also allow JSM the freedom to choose the exact resources to use to satisfy the user's request. While the selection of resources can be further influenced by options such as, --rs_per_host, --rs_per_socket, and --latency_priority, the final selection of resources is done at JSM's discrimination. If the user requires the use of specific resources, such as when consecutive job steps require the exact same resources, there are two file-based specifications supported by JSM for requesting resources.

The first approach presented here is a legacy approach which might be deprecated in the future. This approach uses the option -U to specify a file containing the specification of the resource sets to be used by an allocation. The resource sets described in the file must describe a set of resource sets that could have been allocated by JSM using only non-explicit command line options. Specifically,each resource set must contain the same number of CPU's, GPU's and memory and a CPU or GPU can only be assigned to a single resource set.

You can use the following options for the jsrun command when consecutive job steps require the exact same resources

RS 0  {host 1 cpu 0 1 2 3 gpu 1 mem 0-4096 }
RS 1  {host 2 cpu 0 1 2 3 gpu 1 mem 0-4096 }
RS 2  {host 1 cpu 4 5 6 7 gpu 0 mem 0-3072 1-1024 }
RS 0  {host 1 cpu 0 1 2 3 gpu 1 mem 0-4096 }
RS 1  {host 2 cpu 0 1 2 3 gpu 1 mem 0-4096 }
RS 2  {host 1 cpu 4 5 6 7 gpu 0 mem 0-3072 1-1024 }

Note: -S and -U options are deprecated and may be removed in a future release. You must use the ERF file options --erf_input and --erf_output (reference to erf_format man page) which provides more features and you can easly achieve the functionality provided by the -S and -U options.

The other method for describing explicit resources is the --erf_input and --erf_output options to the jsrun command. These options also specify a file-based approach to describing the resources assigned to a job step. However, the file can be used to specify irregular or heterogeneous resource sets and resource sets with overlapping resource. In addition, the file must specify the mapping of ranks to resource sets and the binding of ranks to smt-threads within each resource set. The ERF file can also specify a mapping of ranks to applications. If a mapping of ranks to applications is not specified, then a single application is assumed and is taken from the jsrun command line.

The following options cannot be used with --erf_input --tasks_per_rs, np, --cpu_per_rs, --gpu_per_rs, --latency_priority, --memory_per_rs, --nrs, --rs_per_host, --rs_per_socket, --appfile, --allocate_only, --launch_node_task, --use_reservation, --use_resources, and --bind.

The following example uses the plane and cyclic options that have tasks that are assigned for each resource set

jsrun --launch_distribution plane2 --nrs 2 --cpu_per_rs 4 --bind packed2 --np 4
jsrun --launch_distribution cyclic --nrs 2 --cpu_per_rs 4 --bind packed2 --np 4
3  mstr  a.out
5  wrkr  b.out

You can use the same reservation in multiple lines in the file. The number of tasks that are listed in the file coordinate their use of the resources for the specified reservation. For example, if you run the jsrun -A myres --nrs 4 --rs_per_host 1 command with the following file, a single task is created on each of the four nodes of myres

2  myres  a.out
2  myres  b.out

Process management

You can use the following options for the jsrun command to manage processes that were created by the jsrun command

Environment propagation

You can use the following options for the jsrun command to set environment variables.

NoteThe following environment variables are not propagated by the jsrun command

System controls
You can use the following options for the jsrun command to control the larger software environment that run tasks: <!---

Miscellaneous options

The following are other options that you can use with the jsrun command:


  1. To run five tasks with each task on its own core, run the jsrun --np 5 a.out command.
  2. To assign one core and one GPU to each of the five tasks, run the jsrun --np 5 --nrs 5 --cpu_per_rs 1 --gpu_per_rs 1 a.out command.
  3. To run eight tasks on 16 cores and to prefer cores that are colocated on the same socket, run the jsrun --np 8 -nrs 1 --cpu_per_rs 16 --latency_priority cpu-cpu .a.out command.
  4. To run 128 ranks where every 8 ranks share a GPU, run the jsrun --np 128 --tasks_per_rs 8 --gpu_per_rs 1 --cpu_per_rs 8 ./a.out command.
  5. To run eight tasks on four nodes with each task having three GPUs and located as close as possible to the assigned GPU, run the jsrun --np 8 --nrs 8 --rs_per_host 2 --gpu_per_rs 3 --latency_priority cpu-gpu command.

See also

jskill(1), jslist(1), jswait(1)

Parent topic Job Step Manager commands