About flexible job output directory

The flexible job output directory feature lets you create and manage the job output directory dynamically based on configuration parameters.

This feature is useful if you are running applications that have specific requirements for job output directory, such as copying data to the directory after the job finishes. This feature ensures that this data will not be overwritten.

A job output directory can be specified through the DEFAULT_JOB_OUTDIR configuration parameter in the lsb.params file. The directory path can be absolute or relative to the submission directory and can include dynamic patterns. Once specified, the system creates the directory at the start of the job on the submission host and uses the new directory. The directory also applies to jobs that are checkpointed, migrated, requeued or rerun.

LSF checks the directories from the beginning of the path. If a directory does not exist, the system tries to create that directory. If it fails to create that directory, then the system deletes all created directories and uses the submission directory for output. LSF creates job output directory under the 700 permissions with the ownership of a submission user.

For more information on this parameter, see the IBM Spectrum LSF Configuration Reference.

You can also use the bsub -outdir output_directory command to create the job output directory. The -outdir option supports dynamic patterns for the output directory. The job output directory specified with this command option, or specified in the DEFAULT_JOB_OUTDIR parameter, also applies when using the bsub -f command to copy files between the local (submission) host and the remote (execution) host.

The following assumptions and dependencies apply to the -outdir command option:

  • The execution host has access to the submission host.

  • The submission host should be running RES or it will use EGO_RSH to run a directory creation command. If this parameter is not defined, rsh will be used. RES should be running on the Windows submission host in order to create the output directory

For more information on this command, see the IBM Spectrum LSF Configuration Reference.