Enabling custom job directories in submission templates

Edit online

When a user submits a job, the job directory is a temporary timestamped directory that is created. Job input files and output files are placed in this directory. In your submission template, you can create a field to allow users to customize the location of their own job directory. When users specify their own job directory, no temporary directory is created and users can see their job directory in Job details view.

Before you begin

Requirements:

  • Your submission template must have the status Unpublished so that changes can be made.
  • The type of field in the submission template must be Input Text or File Path Selection and must have the ID OUTPUT_FILE_LOCATION.
  • Users must have the operating system permissions of read and execute for the directories they specify. If users specify a directory for which they have no read or execute permission, job submission will fail.
  • The directory specified by the user must be an absolute path on the web server.
  • The environment variables $HOME and $USER can be used in the specified directory path.
Important: Custom job directories will not be purged when automatic data purge is performed by the system. Users will need to manually manage their custom job directories and files.

Procedure

  1. In the Workload tab, select Definitions > Templates, and click the submission template you want to modify.

    The modification window is displayed.

  2. Click Add to add a field and select the field type File Path Selection or Input Text.
  3. In the field definition, enter the field ID OUTPUT_FILE_LOCATION.
  4. Add help text for your field to indicate to users that they must have read and execute access to directories they specify or job submission will fail and any other additional information you may want to communicate.
  5. Click Save to exit the modification window.
  6. Click Save to save your changes.
  7. Select the Submission Script tab and modify your submission script.
    1. Check that your submission script has the CWD_OPT line.

      By default, all built-in submission templates have this line in the submission script.

      If you created a custom submission template, check that the submission script has this line and add it to your script if needed.

      CWD_OPT="-cwd \"$OUTPUT_FILE_LOCATION\""
    2. Add ${CWD_OPT} to the JOB_RESULT line right after the bsub command.

      For example:

      JOB_RESULT=`/bin/sh -c "bsub ${CWD_OPT} ${LSF_OPT} ${ADVANCED_OPT} ${LIMITS_OPT} ${CUSTOMIZE_OPT} ${JOB_COMMAND} 2>&1"`
      Note: Depending on your submission template, the JOB_RESULT line may differ from the example. Ensure that ${CWD_OPT} is immediately after the bsub command.
  8. Click Save to save your application.
  9. Click Publish to publish your application.
  10. Test your configuration.
    1. Select Workload and click New to submit a job to the application that you just modified.
    2. Specify the job directory location in your new field.
    3. Submit the job.

      The job directory that is created should be the one you specified.