About flexible job CWD

The Current Working Directory (CWD) feature lets you create and manage the job CWD dynamically based on configuration parameters, and any dynamic patterns included in the path.

This feature is useful if you are running applications that have specific requirements for job CWD, such as copying data to the directory before the job starts running. The CWD feature ensures that this data will not be overwritten.

The CWD feature can be enabled and controlled through the following configuration parameters:

  • JOB_CWD_TTL in lsb.params and lsb.applications: Specifies the time-to-live for the CWD of a job. LSF cleans created CWD directories after a job finishes based on the TTL value.

  • JOB_CWD in lsb.applications: specifies the CWD for the job in the application profile. The path can be absolute or relative to the submission directory. The path can include dynamic directory patterns.

  • DEFAULT_JOB_CWD in lsb.params: Specifies the cluster wide CWD for the job. The path can be absolute or relative to the submission directory. The path can include dynamic patterns.

  • LSB_JOB_CWD environment variable: Specifies the directory on the execution host from where the job starts.

If the job is submitted with the -app option but without the -cwd option, and the LSB_JOB_CWD parameter is not defined, then the application profile defined in the JOB_CWD parameter will be used. If the JOB_CWD parameter is not defined in the application profile, then the value of the DEFAULT_JOB_CWD parameter is used.

For more information on these parameters, see the IBM Spectrum LSF Configuration Reference.

You can also use the bsub -cwd command option to specify the current working directory. LSF cleans the created CWD based on the time to live value set in the JOB_CWD_TTL parameter.

For more information on this command, see the IBM Spectrum LSF Command Reference.

Each specified CWD can be created as unique directory paths by using dynamic patterns. For example:

/scratch/%P will be shared for multiple jobs
/scratch/%P/%J_%I is unique

LSF creates CWD under the 700 permissions with the ownership of a submission user. If CWD creation fails, the /tmp directory is used. If the CWD path includes the user home directory and if it is not accessible on the execution host, it is replaced with the execution user home directory. If that directory is also not accessible, then /tmp is used.

When deleting a directory, LSF deletes only the last directory of the path which was created for the job. If that directly is shared by multiple jobs, data for other jobs may be lost. Therefore, it is recommended not to have shared CWD with enabled TTL.

If CWD was created for the job and then the brequeue command or the bmig command was run on the job, LSF will not delete CWD. For parallel jobs run with the blaunch command, LSF creates CWD only for the execution host and assumes that they are using a shared file system.