Managing the Submitted Job Tracker

The Submitted Job Tracker is an IBM® i work management feature that tracks the lifecycle of submitted jobs for a user specified set of job queues.

A user can register a job queue to be tracked by adding it to the Job Queue List (JQL). After a job queue has been registered, any jobs subsequently submitted to that job queue are tracked in a job tracking file. A user can use the information stored in the job tracking file to resubmit a job on the original node where the job was tracked or another node. The information in the job tracking file can help prepare for maintenance, recover from a node failure, or balance workload.

Submitted Job Tracker is a work management feature of the IBM i operating system 5770-SS1 and can be used without Db2® Mirror. However, there are additional capabilities when using Submitted Job Tracker in a Db2 Mirror environment. In a Db2 Mirror environment, Submitted Job Tracker automatically replicates the job tracking files between the primary and secondary nodes. Additionally, the Db2 Mirror GUI can be used to display and manage the JQL and tracked job information.

The JQL is not replicated in a Db2 Mirror environment, therefore it must be configured and managed on each node separately. Job queue objects being replicated by Db2 Mirror are not required to be tracked by Submitted Job Tracker and are not automatically added to the JQL. In order to track jobs submitted to a job queue on both nodes, the job queue must be added to the JQL on each node separately.

The job tracking file is a database file that keeps track of job submit parameters and job status for any jobs being tracked by Submitted Job Tracker. In a Db2 Mirror environment, a pair of job tracking files are used to store tracked job details for submitted jobs and can exist in SYSBAS or within a database IASP and are automatically replicated. The job tracking files are named QSBMJOBTRK/QAMRDJTS¹ and QSBMJOBTRK/QAMRDJTT². One job tracking file stores tracked job details submitted on the first node and the second job tracking file stores tracked job details submitted on the second node.

Tracking job lifecycle events

To track job lifecycle events, Submitted Job Tracker uses the following work management exit points:

Change Job (QIBM_QWT_CHGJOB)
Job Notification (QIBM_QWT_JOBNOTIFY)
Submit Job (QIBM_QWT_SBMJOB)

See the details for how Submitted Job Tracker handles each lifecycle event:

Submit Job (SBMJOB) command: When a job is submitted to a tracked job queue with SBMJOB, parameters that would be necessary to resubmit the job are stored in the job tracking file.
Change Job (CHGJOB) command or API (QWTCHGJB): When a tracked job is changed while it is on a job queue, the job tracking file is updated to reflect the changes.; When a tracked job is changed while it is active, the job tracking file is not updated.; If a tracked job is moved from a tracked job queue to one that is not tracked, then tracking ends for that job and no further updates are made to the job tracking file. However, the information for that job remains in the job tracking file until the job queue is no longer tracked or the job tracking file is cleared.; If a job is moved from an untracked job queue to one that is tracked, then the job remains not tracked.; Note: It is recommended to track all job queues involved in the lifecycle of a tracked job.
Transfer Job (TFRJOB) or Transfer Batch Job (TFRBCHJOB) command: Each time an active job is transferred to a tracked job queue, an additional entry is added to the job tracking file to represent the new routing step associated with that job. If the job queue for a routing step is not a tracked job queue, then no additional entry for that routing step is added to the job tracking file.; When the last routing step ends, all entries in the job tracking file for that job are updated to indicate the job end time. If the job queue for the last routing step is not a tracked job queue, then the job end time is not updated for any of the entries for that job in the job tracking file.; Note: It is recommended to track all job queues involved in the lifecycle of a tracked job.
Job starts: When a tracked job starts, the job tracking file is updated to indicate the job start time.
Job ends: When a tracked job ends, the job tracking file is updated to indicate the job end time and other completion details.; If a job is ended while it is on a tracked job queue that is not allocated to a subsystem, the job tracking file remains unchanged until the job queue is no longer tracked or the job tracking file is cleared.
Job schedule entries: Jobs that are scheduled to run at a future date or time using the Add Job Schedule Entry (ADDJOBSCDE) or Work with Job Schedule Entries (WRKJOBSCDE) commands are not tracked.

The job tracking file is updated asynchronously while tracked jobs encounter lifecycle events. Tracking information is gathered, processed, and stored in a job tracking file asynchronously by the Submitted Job Tracker job (QMRDBJNFY). Subsequently, tracking information in the job tracking file will always lag the real lifecycle event. The lag time should be minimal in most cases, but it is not zero. This means:

A job that is queued at the time of a node failure may not appear in the job tracking file.
A job that has started running at the time of node failure may not appear in the job tracking file or may be described in the job tracking file as still queued.
A job that has ended at the time of a node failure may be described in the job tracking file as still queued or running.

Displaying tracked job queues

The list of tracked job queues in the JQL can be displayed in the Db2 Mirror GUI by selecting Submitted Job Tracker from the left navigation setup menu, as shown in the following figure.

Figure 1. Selecting Submitted Job Tracker from setup menu

The default view lists the tracked job queues on the primary node for SYSBAS.

The tabs on the left side of the panel can be used to change which database IASP is displayed. The tabs at the top of the panel can be used to change which node is displayed. The Both Nodes tab allows you to view the tracked job queues on both nodes side-by-side, as shown in the following figure.

Figure 2. Displaying tracked job queues for both nodes

The list of job queues that have been registered to be tracked can also be displayed using the TRACKED_JOB_QUEUES view:

SELECT * FROM QSYS2.TRACKED_JOB_QUEUES;

Adding tracked job queues

The Add Job Queue to Track interface in the Db2 Mirror GUI is located above the table listing the job queues that have already been registered. To register a new job queue to be tracked with the JQL, you must specify the library name and object name of the job queue. The library and job queue do not need to be existing objects. A job retention period also must be specified. This setting specifies how long information about a tracked job should be kept after a job has ended before the information is eligible to be removed. The default value of 0 minutes indicates the job tracking information is immediately eligible for removal when a job ends.

After providing the job queue information, click the Add Job Queue button to add the job queue to be tracked on the selected node. To track the same job queue on the other node, use the tabs at the top of the panel to change nodes and then use the Add Job Queue to Track interface again to add the same job queue to the JQL on the other node.

Note: For a subsystem to track jobs, there must be at least one tracked job queue registered to Submitted Job Tracker when that subsystem starts.

Figure 3. Adding a job queue to track on the primary node

Job queues can also be registered to be tracked using the ADD_TRACKED_JOB_QUEUE procedure. For example, to add job queue APPJOBQ in library APPLIB:

CALL QSYS2.ADD_TRACKED_JOB_QUEUE(IASP_NAME => '*SYSBAS',
                                 JOB_QUEUE_LIBRARY => 'APPLIB',
                                 JOB_QUEUE         => 'APPJOBQ');

Removing tracked job queues

To remove job queues from being tracked, select the rows for the job queues to remove from the table listing the existing tracked job queues in the Db2 Mirror GUI. Right click to display the context menu and choose Remove Queue(s) from Tracking. To remove the same job queues on the other node, use the tabs at the top of the panel to change nodes and repeat the action.

Job queues can also be removed from being tracked using the REMOVE_TRACKED_JOB_QUEUE procedure. For example, to remove job queue APPJOBQ in library APPLIB:

CALL QSYS2.REMOVE_TRACKED_JOB_QUEUE(IASP_NAME         => '*SYSBAS',
                                    JOB_QUEUE_LIBRARY => 'APPLIB',
                                    JOB_QUEUE         => 'APPJOBQ');

Displaying tracked jobs

To view the list of jobs that have tracked information in a job tracking file, click the Tracked Jobs button in the top right corner of the Submitted Job Tracker display in the Db2 Mirror GUI.

Tracked jobs and their details can also be displayed using the TRACKED_JOB_INFO table function. For example, to view all tracked jobs from either node that have not completed for job queue MYJOBQ in library JQLIB:

SELECT * FROM TABLE(QSYS2.TRACKED_JOB_INFO(JOB_QUEUE_LIBRARY_FILTER => 'JQLIB',
                                           JOB_QUEUE_FILTER => 'MYJOBQ'))
  WHERE JOB_END_TIME IS NULL
  ORDER BY INTERNAL_JOB_IDENTIFIER, ROUTING_STEP;

When one of the nodes is unreachable, information about the jobs that were being tracked prior to the node being unavailable can be retrieved from the job tracking files residing on the remaining node. In the Db2 Mirror GUI, the display indicates that you are viewing Replica Data which is retrieved from the primary node.

Displaying tracked job details for unreachable node — Figure 6. Displaying tracked job details for an unreachable node

Viewing the tracked jobs and their details for the unreachable node can also be done using the same TRACKED_JOB_INFO table function with the node_name_filter parameter specifying the name of the unreachable node. For example, to view all tracked jobs that have not completed for job queue MYJOBQ in library JQLIB that did not complete on the unreachable node PROD2:

SELECT * FROM TABLE(QSYS2.TRACKED_JOB_INFO(NODE_NAME_FILTER => 'PROD2',
                                           JOB_QUEUE_LIBRARY_FILTER => 'JQLIB',
                                           JOB_QUEUE_FILTER         => 'MYJOBQ'))
  WHERE JOB_END_TIME IS NULL
  ORDER BY NODE_NAME, INTERNAL_JOB_IDENTIFIER, ROUTING_STEP;

Clearing tracked jobs details

Tracked job information is eligible to be automatically removed from a job tracking file after the job has ended, the job tracking file has been updated with a job end time value, and the job retention period has elapsed. The job retention period is specified when a job queue is registered with the JQL. The Submitted Job Tracker job (QMRDBJNFY) periodically checks for tracked job information that is eligible to be removed and clears those entries.

All tracked job information for a particular job queue is removed from a job tracking file when that job queue is removed from the JQL.

There are several situations that may cause Submitted Job Tracker to not receive a notification that a job has ended and therefore the job end time in the job tracking file did not get updated. These entries must be manually cleared from the job tracking file. The following situations can cause a tracked job to get into this state:

A node has an unplanned outage. When the node is active again, information in the job tracking file about active jobs when the outage occurred will not have a job end time value set.
An IASP is varied off or is unavailable. While the IASP is unavailable, Submitted Job Tracker is unable to update the job tracking file. When it is varied on again, entries in the job tracking file may still exist for jobs that have ended but the job end time value is not set.
A job is changed or transferred from a tracked job queue to an untracked job queue. This causes tracking to end for that job but information about it remains in the job tracking file and the job end time value is not set.
A user registers a job queue to be tracked by adding it to the JQL after starting the subsystem where the jobs will run. This can prevent Submitted Job Tracker from receiving notifications when a job ends, which results in entries in the job tracking files not being properly updated with job end time values.
A user submits a job to a tracked job queue and the job is ended before the job queue is allocated to a subsystem. Submitted Job Tracker does not receive a notification from work management that the job has ended in this case, therefore the job end time value is not set for the job in the job tracking file.
The limit of eight data queues registered for the work management job notification exit point (QIBM_QWT_JOBNOTIFY) is exceeded. This may cause Submitted Job Tracker to not receive notifications when a job starts or ends, which results in entries in the job tracking file not being properly updated.

To manually remove all tracked job information for SYSBAS or a database IASP, click to open the Actions menu above the table listing the existing tracked job queues and select Clear Tracking Details for Tracked Jobs. The following figure shows this action being selected to remove all tracked job information from the job tracking file for jobs submitted to tracked job queues in SYSBAS.

Figure 7. Clearing tracked job details on the primary node for SYSBAS

Tracked job information can also be cleared using the CLEAR_TRACKED_JOB_INFO procedure. For example, to clear all tracked job information for job queues in SYSBAS:

CALL QSYS2.CLEAR_TRACKED_JOB_INFO(IASP_NAME => '*SYSBAS');

Selectively clearing tracked job information for a specific job queue can be accomplished in the Db2 Mirror GUI by first right clicking the job queue to display the context menu and choose Remove Queue(s) from Tracking. The Add Job Queue to Track interface can then be used to register the job queue to be tracked again. Removing the job queue causes all tracked job information associated with that job queue to be cleared from the job tracking file. Using this method allows you to selectively clear some job information rather than clearing all tracked job information for SYSBAS or a database IASP.

This can also be accomplished using the QSYS2.REMOVE_TRACKED_JOB_QUEUE and QSYS2.ADD_TRACKED_JOB_QUEUE procedures. First, use the QSYS2.REMOVE_TRACKED_JOB_QUEUE procedure to stop tracking jobs submitted to the job queue. Then register the job queue to be tracked again using the QSYS2.ADD_TRACKED_JOB_QUEUE procedure. The following example makes use of this method to selectively clear all tracked job information for job queue MYJOBQ in library APPLIB before registering the job queue again:

CALL QSYS2.REMOVE_TRACKED_JOB_QUEUE(IASP_NAME         => '*SYSBAS',
                                    JOB_QUEUE_LIBRARY => 'APPLIB',
                                    JOB_QUEUE         => 'MYJOBQ'); 

CALL QSYS2.ADD_TRACKED_JOB_QUEUE(IASP_NAME         => '*SYSBAS',
                                 JOB_QUEUE_LIBRARY => 'APPLIB',
                                 JOB_QUEUE         => 'MYJOBQ');

Additionally, many filters are provided on the QSYS2.CLEAR_TRACKED_JOB_INFO procedure to restrict which entries within the job tracking files are removed. Entries can be removed that match a specific node name, job queue, or match an internal job identifier. See CLEAR_TRACKED_JOB_INFO procedure for all of the available filters that can be used to remove entries from the job tracking files.

Using tracked job details to resubmit jobs

The job tracking file contains the information necessary to resubmit a job. You can use this information to resubmit jobs that failed to complete or jobs that never started. The jobs can be resubmitted on the original node or on another node using the parameters contained within the job tracking file. This can be accomplished using the Db2 Mirror GUI or using SQL.

Here are a few examples of how you might make use of the Submitted Job Tracker:

Recovery from another node

In this example, a set of jobs is interrupted by an unplanned outage on the secondary node. Because job tracking files are replicated in a Db2 Mirror environment, the tracked job details replicated from the secondary node can be used to determine which jobs need to be resubmitted on the primary node.

On the secondary node:
1. Register job queues to be tracked using the techniques described in Adding tracked job queues.
2. Start the subsystem.
3. Submit jobs to the tracked job queues.
4. After some time, the secondary node experiences an unplanned outage.
On the primary node after the outage:
1. Use the techniques described in Displaying tracked jobs to display the list of jobs that were tracked on the secondary node. If the secondary node is unavailable because of an outage, the display indicates that you are viewing Replica Data retrieved from the primary node.
2. To resubmit a job using the Db2 Mirror GUI, select it from the list of tracked jobs on the secondary node and right click to display the context menu and choose Submit Job.
  Figure 8. Resubmitting a tracked job after an unplanned outage
3. The parameter values necessary to resubmit the job are retrieved from the tracked job details and loaded into the resubmit job panel. The parameter values can be updated before the job is submitted.
On the secondary node after it becomes available:
1. Remove jobs that were resubmitted on the primary node from their job queue on the secondary node before the associated subsystem starts.
  Note: This only applies to job queues that reside in SYSBAS. Job queues in a database IASP are cleared when the IASP is varied on.
2. Use the techniques described in Clearing tracked jobs details to remove tracked job information for jobs that were successfully resubmitted on the primary node.
3. Start the subsystem.

Database IASP maintenance

In this example, a user determines that some maintenance is needed for a database IASP that has queued jobs in a job queue that resides on that IASP. The sequence of steps shown in this example ensures that existing running jobs are allowed to run to completion and shows how to resubmit queued jobs after the maintenance has finished and the database IASP is available again. Because job queues in an IASP are cleared when the IASP is varied on, the job tracking details captured in the job tracking file must be used to resubmit jobs that were queued prior to the maintenance.

Register all job queues to be tracked using the techniques described in Adding tracked job queues.
Note: This must be done before submitting any jobs to the job queue. Submitted Job Tracker does not track any job details for jobs already on the job queue at the time the queue is registered.
Start the subsystem.
Submit jobs to the tracked job queues.
After some time, database IASP maintenance is needed.
Hold the tracked job queues to prevent new jobs from starting.
Wait for running jobs complete.
Vary off the IASP and perform the maintenance.
After the maintenance is complete, vary on the IASP. The job queues are cleared during IASP vary on.
Use the techniques described in Displaying tracked jobs to display the list of jobs that were tracked for the database IASP.
To resubmit a job, select the job from the list of tracked jobs for the database IASP in the Db2 Mirror GUI and right click to display the context menu and choose Submit Job.
Figure 9. Resubmitting a tracked job after IASP maintenance
Release the job queue.

Restoring Submitted Job Tracker data

Restoring the JQL: The JQL, which is a database file named QSYS/QAMRDJQL, can be restored from save media. This will replace the existing list of tracked job queues with the list of tracked job queues from the save media. When the QAMRDJQL file is restored, each job queue on the restored list of tracked job queues is ensured to be properly registered with Submitted Job Tracker. If the restore causes a job queue to no longer be tracked, then the job tracking file will be cleared of entries for that job queue.
Restoring job tracking files: The job tracking files, which are database files named QSBMJOBTRK/QAMRDJTS and QSBMJOBTRK/QAMRDJTT³, cannot be restored from save media. If an attempt is made to restore these system objects, they will not be restored and the existing objects are left unchanged. If other objects are included in the restore request, they will be processed normally.

¹ For each database IASP, the QAMRDJTS job tracking file exists in library QSJTnnnnn.

² For each database IASP, the QAMRDJTT job tracking file exists in library QSJTnnnnn.

³ For each database IASP, the job tracking files exist in library QSJTnnnnn.