Output and error file generation for work items in a flow

By default, output and error files are not generated for flows or individual work items.

To troubleshoot flows, however, it is useful to always generate output and error files for work items in the flow.

You can set output and error file generation in the Flow Attributes. The behavior to create output and error files is the same as using the LSF® bsub command options -o and -e.

Output and error file settings that are defined in the Flow Attributes are inherited by the following work items in the flow:
  • Jobs
  • Local jobs
  • Job arrays
  • Static subflows
  • Static flow arrays
  • Dynamic subflows
  • Dynamic flow arrays

Output and error file settings do not apply to job scripts, job array scripts, template jobs, or manual jobs.

Users can override output and error file settings that were defined in the flow in individual work items.

Default location of output and error files

By default, output and error files are generated in the working directory of the work item.

If the working directory is not specified, the location that is used for the working directory is the execution user's home directory:
  • Linux®: $HOME
  • Windows: %HOMEDRIVE%%HOMEPATH%

You can define a default working directory for flows with the parameter JS_DEFAULT_FLOW_WORKING_DIR in js.conf. Users can override the default working directory in individual work items.

Override order for output and error file generation settings

The override order for Process Manager to determine output and error file generation, location, and naming is as follows (in order of highest precedence):

  1. Use output and error file settings defined at the job level, in the Job Definition.
  2. Use output and error file settings defined in the static subflow’s Flow Attributes.
  3. Use output and error file settings defined in the dynamic subflow’s Flow Attributes:
    • If Use parent flow’s settings is selected, use settings in the parent flow’s Flow Attributes.
    • If Use inserted flow’s settings is selected, use settings in the inserted target flow’s Flow Attributes.
    • If Override parent flow’s settings and inserted flow’s settings is selected, use settings in the Dynamic subflow Flow Attributes.
  4. Use output and error file settings defined in the flow’s Flow Attributes.

Configuring output and error file generation for work items in a flow

Procedure

  1. In Flow Editor, from the menu select Action > Add Flow Attribute, and define output and error file settings.
    Field Description

    Create output files for jobs and job arrays

    Create error files for jobs and job arrays

    Select Yes to configure output and error file generation and naming pattern.

    Important: If you select to only generate output files, no error files are generated but the content of the error file is appended to the output file(same behavior as the LSF bsub -o option).
    Directory Specify a directory name or path relative to the flow's working directory. Process Manager creates specified subdirectories in the working directory if the directories do not exist.
    File Name

    Specify the naming pattern for files.

    Built-in variables you can use:

    • %u for user name
    • %t for time stamp
    • %J for job ID
    • %I for job array element
    Note: If you specify the file naming pattern to start with a path and use a slash (/), this is interpreted as an absolute path by Process Manager. For example: if your working directory is /home/user1, and you specify the file naming pattern for output files to be /test/output/output.#{JS_FLOW_FULL_NAME}.%J, the output file will be created outside of the working directory in the directory /test/output, not within the working directory.

    Default file naming pattern for output files:

    • Job: output.#{JS_FLOW_FULL_NAME}.%J
    • Local job: output.#{JS_FLOW_FULL_NAME}
    • Job array element: output.#{JS_FLOW_FULL_NAME}.%J[%I]

    Default file naming pattern for error files:

    • Job: error.#{JS_FLOW_FULL_NAME}.%J
    • Local job: error.#{JS_FLOW_FULL_NAME}
    • Job array element: error.#{JS_FLOW_FULL_NAME}.%J[%I]
  2. Click OK to save your changes.