DELAY - Delay Report

The Delay report allows you to determine which system resources are causing delays for jobs or job groups, and to what extent the jobs are delayed.

The report gives you information about job delay for every type of delay that RMF monitors. This includes processor delay (PRC), device delay (DEV), storage delay (STR), subsystem delay (SUB), operator delay (OPR), and enqueue delay (ENQ). RMF provides a detail report for each of these delays except OPR. Operator delay includes message, mount, and quiesce requests. SUB is divided into an HSM, JES, and XCF detail report. The names of the detail reports correspond to the names that appear in the Delay report.

How to request this report

To request the Delay report, select 1 from the Primary Menu, and then select 4 on the Overview Report menu (shown in Figure 1) or enter the following command using the format:
DELAY [job_class,service_class]
For example, to get a Delay report for TSO service class TSOPRIME, enter:
DELAY T, TSOPRIME

Contents of the report

Figure 1. DELAY Report
                         RMF V2R4  Delay Report                Line 1 of 58
Command ===>                                                 Scroll ===> HALF

Samples: 120     System: MVS1  Date: 05/11/19  Time: 12.00.00  Range: 120 Sec

            Service     WFL USG DLY IDL UKN ---- % Delayed for ---- Primary
Name     CX Class    Cr  %   %   %   %   %  PRC DEV STR SUB OPR ENQ Reason

*SYSTEM                  49   1   1  62  36   0   0   0   0   0   0
*TSO                     56   1   1  95   2   0   0   0   1   0   0
*BATCH                   39   2   4   0  94   1   0   0   4   0   0
*STC                     40   0   1  51  48   0   0   0   0   0   0
*ASCH                         0   0   0   0   0   0   0   0   0   0
*OMVS                         0   0   0 100   0   0   0   0   0   0
*ENCLAVE                      0   0   0   0   0 N/A   0 N/A N/A N/A
JES2     S  SYSSTC        0   0   1   0  99   0   1   0   0   0   0 SCLSP4
BMAI     T  PRDTSO   S   16   9  66  13   4   0   1   0  65   0   0 HSM
HSM      S  STCCMD   C   30  26  62   0  23   0   1   0   0  50  11 Mount
HIRW2    B  BATCHMED     35   6  14   0   1   2   1   0  12   0   0 HSM
TCPNET   SO SYSSTC   SC  60   3   2   0  97   2   0   0   0   0   0 NET
*MASTER* S  SYSTEM       67   3   2   0  95   0   2   0   0   0   0 M00202

The graphic form of this report shows the percent of time that each user spent delayed for the above resources.

Field descriptions

Table 1. Fields in the DELAY Report
Field Heading Meaning
Name Name of the job, job group or enclave.

The enclave names, starting with the letters ENC, and belonging to class E, are created dynamically by RMF. You cannot use the names to track a particular enclave through different time ranges. However, the enclave token is used when combining multiple set-of-samples, so that data are combined only for the same individual enclaves, thus providing consistent data. N/A is shown if the value does not apply to enclaves.

CX Abbreviation for the address space types as follows:
S
Started task
T
TSO
B
Batch
A
ASCH
O
OMVS
?
Data is missing or not valid.
Or it can indicate an enclave:
E
Enclave

For summary entries, this field is blank.

An O as second character indicates that an OMVS process exists for this address space.

Service Class The name of the service class that a specified job has been running in.

If a job changes its service class during the report interval, RMF displays eight asterisks (********) instead of the service class name. If the service class is not available, RMF displays eight dashes (--------).

Cr An indication whether WLM managed the address space as storage critical and/or CPU critical during the report interval.
C
CPU critical
S
Storage critical
SC
Both storage and CPU critical
WFL % The workflow percentage of the job or job group. Address space workflow (%) shows the formula used to calculate this value.
USG % The using percentage for the job or job group. Address space using (%) shows the formula used to calculate this value.
DLY % The delay percentage for the job or job group. See Address space delay (%) for more information.
IDL % The idling percentage for a job or job group. Jobs in terminal wait, timer wait, or waiting for job selection by JES are in an idling state if they are not using the processor or devices and are not delayed for any monitored reason.
Jobs classified as in terminal wait meet all of the following conditions:
  • They are not found using any monitored resource
  • They are not found delayed for any monitored reason
  • They are swapped out
  • They are in terminal wait
  • They are waiting for a user ready indication before being swapped in.
Jobs classified as in timer wait meet all of the following conditions:
  • They are not using or delayed for a monitored resource.
  • Their address space is waiting for a timer.

The idling percentage of an address space can vary from 0 to 100%, where 0% indicates that the user is not idling during the report interval, and 100% represents a job that is idle at every sample.

The idling percentage for an address space during a refresh period is calculated as follows:
         # Idle Samples
IDL % =  --------------- * 100
           # Samples
Idle samples
The number of samples that show the job in an idle state.
The idling percentage for a group of address spaces during a range period is calculated as follows:
               SUM Idle Samples
IDL % = -------------------------------- * 100
        # Samples * Avg # Address Spaces
Note: The value reported might include some delay for a non-monitored resource.
UKN % RMF considers jobs that are not delayed for a monitored resource, not using a monitored resource, or not in an idling state to be in an unknown state.
Examples of address spaces in an unknown state are:
  • Idle address spaces that use a non-monitored mechanism for determining when they are active. Most system tasks (STC) show up as unknown when they are idle.
  • Address spaces waiting for devices other than DASD or tape.

The unknown state percentage for an address space can vary from 0 to 100%, where 0% indicates that the state was always known during the report interval and 100% represents a job in an unknown state throughout the report interval.

% Delayed for The percentage that each defined resource contributes to the overall delay of the job or job group.

The overall delay value DLY % may exceed the sum of the reported resource delay values, because there are other resources which contribute to the overall delay, such as WLM capping delay.

If the percentages add up to more than DLY %, there is an overlap of delay states.

The defined resources that can delay the job or job group are as follows:
PRC
The job or job group has ready work on the dispatching queue, but it is not being dispatched.
DEV
The job or job group is delayed for a DASD or tape.
STR
The job or job group is waiting for a COMM, LOCL, SWAP, XMEM, HIPR or VIO page, or is on the out/ready queue. See the Storage Delay report.
SUB
The job or job group is delayed for a JES, HSM, or XCF subsystem request.
OPR
The job or job group is delayed by a message or a mount request or a quiesce. Quiesce means that the operator has quiesced the address space. A quiesced address space can show unexpected data:
  • A swappable address space will be swapped out, thus it can be OUTR and show storage delays.
  • A non-swappable address space will get lowest priority, thus it can show CPU delay, paging delay, or other delays, and even some USG % from time to time depending on the load on the system.
ENQ
The job or job group is waiting to use an enqueued (reserved) resource.
Primary Reason Reported only for a specific job, this field provides additional information about the primary reason for the delay. The contents depend on the resource having the largest % Delayed for value.
If the resource with the maximum delay is:
PRC
This field contains the name of the job that used the processor most frequently while the reported job was delayed.
STR
This field identifies the cause of the largest percentage of delay:
COMM
common storage paging (includes shared pages)
LOCL
local storage paging (includes shared pages)
VIO
virtual I/O paging
SWAP
swap-in delay
OUTR
swapped out and ready
XMEM
cross memory address space
HIPR
standard hiperspace paging delays
DEV
This field contains the volume serial number of the device that the reported job was most frequently delayed for.
SUB
This field contains either JES, HSM, or XCF depending on which subsystem is causing the most delay.
OPR
This field contains Message if most of the delay was due to a message or Mount if most of the delay was due to a mount request.
The field can contain QUIESCE if the operator quiesced the address space. A quiesced address space can show unexpected data:
  • A swappable address space will be swapped out, thus it can be OUTR and show storage delays.
  • A non-swappable address space will get lowest priority, thus it can show CPU delay, paging delay, or other delays, and even some USG % from time to time depending on the load on the system.

Cursor-sensitive control on this field gives you the Quiesce delay variation of the Job Delay report.

ENQ
This field contains the major name of the resource most responsible for the delay.
*ENCLAVE
One or more enclaves are holding the processor.
RG-Cap
The job is delayed due to WLM resource capping. That means that
  • either the resource group for which the job is running, has used up its CPU service specified in the WLM policy,
  • or the work for which the job is running is overachieving its goal. So this work may be capped in order to divert its resources to run discretionary work (see also section 'Using Discretionary Goals' in z/OS MVS Planning: Workload Management).

Monitor III Utility fields

You can use the Monitor III Utility to customize the DELAY report. In addition to the delays previously described, you can use the Utility to have the following delay percentages shown.

Table 2. Additional Fields in the DELAY Report
Field Heading Meaning
% Delayed for
  • JES delay percentage
  • HSM delay percentage
  • XCF delay percentage
  • Operator mount delay percentage
  • Operator message delay percentage
  • Operator quiesce delay percentage
  • WLM resource group capping delay percentage

Cursor-sensitive control on the Delay report

To see all delays for a particular class or summary line (*SYSTEM, *TSO, *BATCH, *STC, *ASCH or *OMVS), use cursor-sensitive control on any name starting with an asterisk ('*') under the name column or on any value in the CX or Service Class columns, to display a subset of the Delay report for that group.

When you use cursor-sensitive control on the *ENCLAVE summary line, you are shown a subset of individual enclave names.

Using cursor-sensitive control on an enclave name displays a pop-up panel that shows information you extracted from the WLM Enclave Classification Data (ECD) control block. You can use this information to identify the transactions that are processed in the enclave. See Enclave Classification Attributes for an example.

To see all jobs using or delayed for processor, use cursor-sensitive control on any indicator under USG % to display either the Processor Delays or the Device Delays report, depending on which is contributing more to the delay.

To investigate which jobs or resources are contributing to a delay, use cursor-sensitive control on any indicator under DLY % or % Delayed for to display the related resource report or job delay report.

Report options

Figure 2. DELAY Report Options Panel
                       RMF Delay Report Options: DELAY              Line 1 of 1
 Command ===>                                                  Scroll ===> HALF

 Change or verify parameters. To exit press END.
 All changes (except for Summary and Criterion specification) will apply to
 DELAY, DEV, ENQ, HSM, JES, PROC, PROCU, STOR, STORC, STORF, STORM and XCF.

   Class         ===> ALL       Classes: ALL TSO BATCH STC ASCH OMVS
   Service class ===> *ALL      *ALL or one of available service classes below
   Summary       ===> NO        Class summary lines on DELAY report (YES NO)
   Criterion     ===> 0         Minimum delay to include job in DELAY report

   Jobs          ===> NO        View job selection/exclusion panel next (YES NO)

                            Available Service classes
 GPMSERVE   OE         OMVSKERN   STCDEF     TSODEF     SYSTEM     SYSSTC

The parameters that you specify on this panel (except Summary and Criterion) affect all job-oriented detail delay reports.

Class
The class of jobs for which you want delay and common storage data reported. For Class, you can request:
  • T or TSO
  • B or BATCH
  • S or STC for started task
  • A or ALL for all jobs in the system
  • AS or ASCH for ASCH address spaces
  • O or OMVS

Your selection for Class applies to all delay and common storage reports and is saved across sessions in the current option set.

Service Class
The service class for which you want data reported. For Service Class, you can specify any of the available service classes listed under Available Service Classes.

If the service class you want is not listed, it was not active during the current report interval. If you specify the service class, it will appear on the report when it is available.

Your selection applies to all delay and common storage reports and is saved across sessions in the current option set.

Summary
Summary allows you to specify whether you want summary lines for the DELAY report.

To produce one summary line for all jobs in the system and one summary line for each class (TSO, BATCH, STC, ASCH or OMVS), enter ALL for Class and YES for Summary.

To only produce a summary line for one class, group or service class, enter the name for Class and YES for Summary.

Your selection for Summary applies only to the DELAY report and is saved across sessions in the current option set.

Criterion
The value (from 0% to 100%) that RMF compares to each job's computed delay value in deciding whether to include the job in the DELAY report.

RMF displays all jobs whose delay values meet or exceed the Criterion.

The value that you specify for Criterion applies only to the DELAY report and is saved across sessions in the current option set.

Jobs
A YES for JOBS displays the name of all the active jobs in the Class, Group or Service class you specified and any jobname that you previously selected or excluded.

You can use this list to view active jobs in the system and to select and exclude jobs from your report.

Available Service classes
The list of available service classes includes all of the service classes that belong to the Class you specified and that had any activity during the current report interval.

Press the END key to make these values active for the session.

Job Selection/Exclusion Option panel

If you select YES for Jobs on the Delay Report Options panel, RMF displays a Job Selection/Exclusion panel shown in Figure 3.

Figure 3. DELAY Report Job Selection/Exclusion Panel
                      RMF Delay Report Options: DELAY             Line 1 of 77
Command ===>                                                  Scroll ===> HALF

Select (S), exclude (X), or fill-in jobs for report. Press END.

Sel  Jobname    Sel  Jobname    Sel  Jobname    Sel  Jobname    Sel  Jobname
     ________        ________        ________        ________        ________
     ________        ________        ________        ________        ________
 S   *ALL            *MASTER*        520252          EFIBERC         ALISONW
     ALLOCAS         ALPERTA         ALTER2          AMSAQTS         AMYH
     ANDREA          ANDREW          ANN             ARTHUR          ARTI
     ASTER2          AUXCFTH         AULT            BARBARA         BARBIE
     BCOVEN          BEENA           BERNIEP         BERRZA          BETHC

The Job Selection/Exclusion panel allows you to select or exclude specific jobs from your delay reports.

The panel lists:
  • Active jobs in the class and group specified on the Delay Report Options panel.
  • All jobs that you previously selected or excluded, selection codes appear to the left of jobs previously selected or excluded.

To select a job for your delay reports, type an s to its left, under SEL; to exclude a job, type an x to its left. (You can select *ALL for all jobs in the specified class and group and then exclude specific jobs. Similarly, you can exclude *ALL and then select specific jobs.)

To select several jobs with similar names, use an asterisk ('*') as a "wild card" character under Jobname. For example: to request a report for all jobs starting with A, specify 's' under Sel, 'a*' under Jobname and ensure that there is an 'x' beside *ALL.

You can also specify multiple wild card entries, for example, to list all jobs starting with A and all jobs starting with BK, specify:
Sel  Jobname    Sel  Jobname
 S   A*______    S   BK*_____
     ________        ________
 X   *ALL

To select or exclude a job that is not listed, enter the job name in the top row and the appropriate selection code to its left.

All the jobs might not fit on this panel. Use PF8 and PF7 to scroll through the remaining job names.