About Process Manager terms

Jobs

A job is a program or command that is scheduled to run in a specific environment. A job can have many attributes specifying its scheduling and execution requirements. You specify the attributes of the job when you define the job in the Flow Editor. Process Manager schedules and manages jobs that run on Process Manager hosts. Process Manager uses job attributes, system resource information, and configuration settings to decide when, where, and how to run jobs. While each job is assigned a unique job name by the system, you can associate your own job names to make referencing easier.

Dependencies

A dependency describes the order in which something happens within a flow: a job (or job array or subflow) can depend on the completion of a job, job array, subflow, or event before it can run.

A dependency is shown in the Flow Editor and Flow Manager as a line with an arrow. The job at the tip of the arrow cannot run until the work item at the other end of the arrow reaches a particular condition.

A dependency is used to indicate relationships between jobs, events, alarms, and so on.

Job arrays

A job array is a group of homogeneous jobs—jobs that share the same executable and resource requirements, but have different input files, for example input1, input2, input3 and so on. You can use a job array to submit, control and monitor all of the jobs as a single unit. Each job submitted from a job array shares the same job ID as the job array and is uniquely referenced using an array index. The dimension and structure of a job array is defined when the job array is created.

Job submission script

A job submission script is a shell script or a batch file, which you can define to submit a job. You can define and submit customized job array submission script with bsub command and options. You can monitor and control the jobs that have been submitted through the customized job submission scripts. You specify the attributes of the script when you define the job submission script in the Flow Editor. Process Manager schedules and manages job submission scripts that run on the Process Manager hosts.

Job array submission script

A job array submission script is a group of submission scripts— that share the same executable and resource requirements, but have different input files, for example script1, script2, script3, and so on. You can use a customized job array submission script to control and monitor all the jobs and job arrays. Each job submitted from a job array submission script shares the same job ID as the job array submission script and is uniquely referenced using an array index. The dimension and structure of a job array submission script is defined when the job array submission script is created.

Manual jobs

A manual job is a place-holder in a flow—it marks the place in a process where some manual activity must take place before the flow can continue. Successors of a manual job cannot run until the manual job is explicitly completed.

Local jobs

A local job is a job that will execute immediately on the Process Manager host without going through LSF®. A local job is usually a short and small job.

It is not recommended to run long, computational-intensive or data-intensive local jobs as it can overload the Process Manager host.

Other Batch jobs

An Other Batch job is a job that you run on a remote batch system or workload manager that is not LSF.

When Process Manager is configured to communicate with another batch system, you will see the Other Batch Job work item enabled in Flow Editor:

In the Job Definition, you specify the command to run, other job submission options, and the user account under which the job is to run in the Other Batch System.

Flow definitions

A flow definition is a container for a group of related jobs. The flow definition describes both the jobs and their relationships to each other, as well as any dependencies the jobs have on files or dates and times. Using a flow definition, you can create a complex schedule involving many jobs, and manipulate it as a single entity. You can also use a flow definition to group jobs together that form a particular function, and imbed the flow definition as a subflow within a larger flow definition. This allows you to share and reuse common functions.

Flow definitions can be stored locally on your own machine, or within a shared file system. You can see and import flow definitions created by another user, but you cannot control running flows owned by another user unless you have administrative authority.

Flows

A flow is the particular occurrence of a flow definition that is created when the flow definition is triggered. When Process Manager creates a flow from the flow definition, it assigns each occurrence of the flow a unique ID called the flow ID.

Adhoc flows

Since version 10.1, adhoc flows have been removed from the By Definition view tab in Flow Manager. However, users can still view adhoc flow status from the By User tab or the By State tab in Flow Manager.

An adhoc flow is displayed in the tree view of the Flow Manager in each of the following ways:

Static subflows

A static subflow is a flow definition that has been copied into another flow definition. Using a static subflow within a flow is a simple method to share and reuse common routines.

Dynamic subflows

A dynamic subflow is a reference to another flow definition. The flow definition that is referenced is called a target flow. The flow that contains the dynamic subflow is called the parent flow.

You maintain the target flow separately from the parent flow. Changes made to the target flow are automatically updated in the parent flow.

Only target flows that have been submitted to Process Manager and published can be selected in a dynamic subflow.

Static flow array

A static flow array is a way to specify to run a static subflow as an array. The subflow will run as many times as the size of the array.

In addition, in the array attributes, you can define whether to run the subflow in parallel or sequentially.

Dynamic flow array

A dynamic flow array is a way to specify to run a dynamic subflow as an array. The subflow will run as many times as the size of the array.

In addition, in the array attributes, you can define whether to run the subflow in parallel or sequentially.

You maintain the target flow separately from the parent flow. Changes made to the target flow are automatically updated in the parent flow.

Only target flows that have been submitted to Process Manager and published can be selected for the dynamic flow array.

Events

An event is a change or occurrence in the system (such as the creation of a specific file, a prior job completing with a particular exit code, or simply the arrival of a file at a particular date and time) that can be used to trigger a flow or one or more jobs within a flow. Process Manager responds to the following types of events:

Time events

You use time events in Process Manager to make something happen at a specific time. You can use a time event to specify the frequency at which a repetitive job repeats, to prevent a job from running until a particular time, or to specify when to start running a flow.

You cannot create a time event without referencing a calendar, which provides the date or dates on which the time event is valid, allowing it to trigger.

You create time events using the Flow Editor.

File events

You use file events to make something happen when a file reaches a particular state. You can use a file event to trigger a flow, job or subflow: when a file arrives; when a file reaches a certain size; if a certain file exists; or any combination of these conditions.

You create file events using the Flow Editor.

Proxy events

You use proxy events to represent work items that run within another flow, or to represent another flow. You can create a dependency on the success or failure of a proxy event. You can use a proxy event to trigger a flow, or to trigger a work item within a flow.

Link events

You use link events to combine multiple dependencies into a single point in a flow diagram. You can use link events to run a job when multiple jobs complete, or you can use them to run a subflow when one of a group of jobs complete. For example, you can use an AND link event to trigger a job when all of a group of conditions are met, or you can use an OR link event to trigger a job when any one or more of a group of conditions is met.

You create link events using the Flow Editor.

Calendars

A calendar consists of a sequence of days on which the calendar is considered valid. A job is scheduled when the calendar is valid and a time of day specification is met. Calendars are defined and manipulated independently of jobs so that multiple jobs and flows can share the same calendar. Each user can maintain a private set of calendars, or use the calendars defined as system calendars. If a calendar is changed, any jobs associated with the calendar will automatically run according to the new definition. Calendars are stored within Process Manager’s private storage, and cannot be stored locally or edited outside of the Calendar Editor.

Exceptions

An exception is a specific error condition that is detected when a job does not process as expected. Process Manager detects several of these conditions.

Exception Handlers

An exception handler is a function used to respond when an exception occurs. You can use jobs or flows as exception handlers, or you can use Process Manager’s built-in exception handlers:

Kill

You can automatically kill a job, flow, or subflow if it experiences the specified exception.

Rerun

You can automatically rerun a job, flow, or subflow if it experiences the specified exception.

Alarms

An alarm is a type of built-in exception handler, used to send an email notification to key personnel or execute a script to show that an error has occurred that requires intervention.

Variables

You can use Process Manager to pass variables to and from scripts. Process Manager supports three kinds of variables:

Variable evaluator

The variable evaluator (VE) is a work item in the flow editor that allows jobs to depend on the evaluation of variable expressions instead of the traditional job exit status, time events, and file events.

This work item contains no actual jobs to run, and therefore you cannot kill, suspend, or resume it. In addition, since the work item does not perform any actual work, you cannot use the variable evaluator as a triggering event, a proxy event to start the next work item, or as a flow completion criterion.

The variable evaluator serves as an intermediate step between jobs and the validation of variable decision branches. Typically, the predecessors of a variable evaluator assign values to various user variables. When all the variables are set, the variable evaluator will evaluate all of its variable expression branches and determine which successors to start executing. If there is no predecessor to the variable evaluator, the variable evaluator will start and run to completion immediately.

File naming conventions

This guide uses UNIX file naming conventions to illustrate file names. However, if the file you are referencing is on a Windows file system, use Windows file naming conventions where applicable.