Diagnosing problems in z/OS System Logger

Extended waits by the CICS® log manager can be caused by problems within z/OS® System Logger or other areas of z/OS.

Procedure

Follow these steps to diagnose problems in z/OS System Logger.

Step 1. Look at console messages and dumps

Look out for:

Outstanding WTOR messages
IXGxxx messages
Allocation, catalog and HSM error messages
IO errors for log stream data sets and LOGR couple data sets
IXCxxx messages, indicating problems with the coupling facility structure or couple data sets.
1C5 abends, and other abends from the IXGLOGR address space.

Log stream data sets are of the form IXGLOGR.stream_name.Annnnnnn. The high level qualifier (IXGLOGR) may be different if the HLQ parameter was specified when the log stream was defined.

Explanations of z/OS System Logger reason codes, which are shown in CICS and z/OS messages and traces, are in the IXGCON macro and in z/OS MVS Programming: Assembler Services Reference ABE-HSP.

Step 2. Check GRS resource contention

Check GRS resource contention by displaying GRS enqueues and latch usage on all machines in the sysplex.

Issue either of the following z/OS commands:

RO *ALL,D GRS,C
RO *ALL,D GRS,RES=(SYSZLOGR,*)

The RO *ALL phrase means that the command goes to all systems in the sysplex.

A normal response looks like what is shown in Figure 1.

Figure 1. Example of a normal response to

RO *ALL,D
GRS,C

   D GRS,C
 
   ISG02 0I 12.06.49 GRS STATUS 647
   NO ENQ CONTENTION EXISTS
   NO LATCH CONTENTION EXISTS
 
   D GRS,RES=(SYSZLOGR,*)
 
   ISG020I 14.04.28 GRS STATUS 952
   NO REQUESTORS FOR RESOURCE  SYSZLOGR *

A response showing GRS contention looks like what is shown in Figure 2. You may also see latch set name SYS.IXGLOGER_MISC.

Figure 2. Example of a response showing GRS contention

   D GRS,C
 
   ISG020I 12.06.31 GRS STATUS 619
   LATCH SET NAME:  SYS.IXGLOGER_LCBVT
   CREATOR JOBNAME: IXGLOGR   CREATOR ASID: 0202
     LATCH NUMBER:  7
       REQUESTOR  ASID  EXC/SHR    OWN/WAIT
       IXGLOGR    0202  EXCLUSIVE  OWN
       IXGLOGR    0202  SHARED     WAIT
 
   D GRS,RES=(SYSZLOGR,*)
 
   ISG020I 19.58.33 GRS STATUS 374
   S=STEP    SYSZLOGR  91
   SYSNAME        JOBNAME         ASID     TCBADDR   EXC/SHR  OWN/WAIT
   MV26      MSLDELC1           002F       008F6370 EXCLUSIVE   OWN
   S=STEP    SYSZLOGR  93
   SYSNAME        JOBNAME         ASID     TCBADDR   EXC/SHR  OWN/WAIT
   MV26      MSLWRTC1           002E       008DED90 EXCLUSIVE   OWN
   MV26      MSLWRTC1           002E       008DB990 EXCLUSIVE   WAIT
   MV26      MSLWRTC1           002E       008DB700 EXCLUSIVE   WAIT
   MV26      MSLWRTC1           002E       008F60C8 EXCLUSIVE   WAIT
   S=SYSTEMS SYSZLOGR LPAYROL.TESTLOG.TLOG1
   SYSNAME        JOBNAME         ASID     TCBADDR   EXC/SHR  OWN/WAIT
   MV27      IXGLOGR            0011       008F7398 EXCLUSIVE   OWN
   MV26      IXGLOGR            0011       008F7398 EXCLUSIVE   WAIT

This shows which tasks (that is, z/OS TCBs) have exclusive enqueues on the log streams, and which tasks are waiting for them. It is quite normal for enqueues and latches to be obtained, occasionally with contention. They are indications of a problem only if they last for more than a minute or so.

Long term enqueuing on the SYSZLOGR resource can be a sign of problems even if there is no contention.

You can choose to display only those log streams exclusively enqueued on by CICS jobs in the sysplex. Issue the following z/OS command:

   D GRS,RES=(DFHSTRM,*)

A typical response to this command looks like this:

Figure 3. Response to D GRS,RES=(DFHSTRM,*)

   ISG020I 14.51.28 GRS STATUS 541
   S=SYSTEMS DFHSTRM  PAYROL.CICSVR.DFHLGLOG
   SYSNAME        JOBNAME         ASID     TCBADDR   EXC/SHR  OWN/WAIT
   MV29      PAYROL91           0042       007D9108   SHARE     OWN
   MV29      PAYROL93           0044       007D9138   SHARE     OWN
   S=SYSTEMS DFHSTRM  PAYROL.FWDRECOV.UTL3
   SYSNAME        JOBNAME         ASID     TCBADDR   EXC/SHR  OWN/WAIT
   MV29      PAYROL91           0042       007D9108   SHARE     OWN
   MV29      PAYROL93           0044       007D9138   SHARE     OWN
   S=SYSTEMS DFHSTRM  PAYROL.IYK8ZET1.DFHJ02
   SYSNAME        JOBNAME         ASID     TCBADDR   EXC/SHR  OWN/WAIT
   MV29      PAYROL91           0042       007D9108   SHARE     OWN
   S=SYSTEMS DFHSTRM  PAYROL.IYK8ZET1.DFHLOG
   SYSNAME        JOBNAME         ASID     TCBADDR   EXC/SHR  OWN/WAIT
   MV29      PAYROL91           0042       007D9108 EXCLUSIVE   OWN
   S=SYSTEMS DFHSTRM  PAYROL.IYK8ZET1.DFHSHUNT
   SYSNAME        JOBNAME         ASID     TCBADDR   EXC/SHR  OWN/WAIT
   MV29      PAYROL91           0042       007D9108 EXCLUSIVE   OWN
   S=SYSTEMS DFHSTRM  PAYROL.IYK8ZET3.DFHJ02
   SYSNAME        JOBNAME         ASID     TCBADDR   EXC/SHR  OWN/WAIT
   MV29      PAYROL93           0044       007D9138   SHARE     OWN
   S=SYSTEMS DFHSTRM  PAYROL.IYK8ZET3.DFHLOG
   SYSNAME        JOBNAME         ASID     TCBADDR   EXC/SHR  OWN/WAIT
   MV29      PAYROL93           0044       007D9138 EXCLUSIVE   OWN
   S=SYSTEMS DFHSTRM  PAYROL.IYK8ZET3.DFHSHUNT
   SYSNAME        JOBNAME         ASID     TCBADDR   EXC/SHR  OWN/WAIT
   MV29      PAYROL93           0044       007D9138 EXCLUSIVE   OWN

Step 3. Check coupling facility structure and couple data set status

Display the z/OS System Logger couple data set status by issuing the following z/OS command:

D XCF,CPL,TYPE=LOGR

A normal response looks like this:

Figure 4. Example of a normal response to

D
XCF,CPL,TYPE=LOGR

   D XCF,CPL,TYPE=LOGR
   IXC358I  14.47.51  DISPLAY XCF 391
   LOGR COUPLE DATA SETS
   PRIMARY    DSN: SYS1.SYSPLEX2.SEQ26.PLOGR
              VOLSER: P2SS05     DEVN: 230D
              FORMAT TOD         MAXSYSTEM
              12/20/95 09:25:48          8
   ALTERNATE  DSN: SYS1.SYSPLEX2.SEQ26.ALOGR
              VOLSER: P2SS06     DEVN: 2C10
              FORMAT TOD         MAXSYSTEM
              12/20/95 09:27:45          8
   LOGR IN USE BY ALL SYSTEMS

If the response shows that LOGR is not in use by all systems, there may be a problem to investigate. Look for IXCxxx messages which might indicate the cause of the problem and issue the following command to attempt reconnection to the couple data set:

SETXCF CPL,TYPE=(LOGR),PCOUPLE=(couple_dataset_name)

Display all structures with Failed_persistent connections by issuing the following z/OS command:
```
D XCF,STR,STRNM=*,STATUS=FPCONN
```
z/OS System Logger should resolve any failed connections.

Step 4. Check the log stream status

To display information about the status of CICS log streams, a batch job should issue this IXCMIAPU command:

LIST LOGSTREAM NAME(streamname) DETAIL(YES)

You can use wildcards to select multiple log streams. For example, the following job produces a report on the system log streams for CICS region IYLX4:

Figure 5. Sample JCL

   //IYLXLIST JOB NOTIFY=WILLIN,MSGCLASS=A
   //LOGLIST  EXEC PGM=IXCMIAPU
   //SYSPRINT DD SYSOUT=A,DCB=RECFM=FBA
   //SYSIN    DD *
     DATA TYPE(LOGR) REPORT(NO)
     LIST LOGSTREAM NAME(WILLIN.IYLX4.DFH*) DETAIL(YES)

Figure 6 shows a typical response to this command, with system logs streams for CICS region IYXL4.

Figure 6. Example output produced by the LIST LOGSTREAM NAME command

   LOGSTREAM NAME(WILLIN.IYLX4.DFHLOG) STRUCTNAME() LS_DATACLAS()
           LS_MGMTCLAS() LS_STORCLAS() HLQ(IXGLOGR) MODEL(NO) LS_SIZE(0)
           STG_MGMTCLAS() STG_STORCLAS() STG_DATACLAS() STG_SIZE(0)
           LOWOFFLOAD(40) HIGHOFFLOAD(85) STG_DUPLEX(YES) DUPLEXMODE(UNCOND)
           RMNAME() DESCRIPTION() RETPD(0) AUTODELETE(NO)
           DASDONLY(YES)
           MAXBUFSIZE(64000)
       LOG STREAM ATTRIBUTES:
         User Data:
          0000000000000000000000000000000000000000000000000000000000000000
          0000000000000000000000000000000000000000000000000000000000000000
  
      LOG STREAM CONNECTION INFO:
         SYSTEMS CONNECTED: 1
         SYSTEM    STRUCTURE         CON CONNECTION  CONNECTION
        NAME      VERSION           ID  VERSION     STATE
        --------  ----------------  --  ----------  ----------
        MV28      0000000000000000  00  00000000    N/A

       LOG STREAM DATA SET INFO:
         DATA SET NAMES IN USE: IXGLOGR.WILLIN.IYLX4.DFHLOG.<SEQ#>
         Ext.   <SEQ#>    Lowest Blockid    Highest GMT        Highest Local
        -----  --------  ----------------  -----------------  ---------------
       *00001  A0000007  0000000000496BAB  07/18/97 08:29:13  07/18/97 09:29:
 
       NUMBER OF DATA SETS IN LOG STREAM: 1 
      POSSIBLE ORPHANED LOG STREAM DATA SETS:
         DATA SET NAMES:
        --------------------------------------------
        IXGLOGR.WILLIN.IYLX4.DFHLOG.A0000037
        IXGLOGR.WILLIN.IYLX4.DFHLOG.A0000404
 
        NUMBER OF POSSIBLE ORPHANED LOG STREAM DATA SETS: 2

 LOGSTREAM NAME(WILLIN.IYLX4.DFHSHUNT) STRUCTNAME() LS_DATACLAS()
           LS_MGMTCLAS() LS_STORCLAS() HLQ(IXGLOGR) MODEL(NO) LS_SIZE(0)
           STG_MGMTCLAS() STG_STORCLAS() STG_DATACLAS() STG_SIZE(0)
           LOWOFFLOAD(0) HIGHOFFLOAD(80) STG_DUPLEX(YES) DUPLEXMODE(UNCOND)
           RMNAME() DESCRIPTION() RETPD(0) AUTODELETE(NO)
           DASDONLY(YES)
           MAXBUFSIZE(64000)
       LOG STREAM ATTRIBUTES:
         User Data:
          0000000000000000000000000000000000000000000000000000000000000000
          0000000000000000000000000000000000000000000000000000000000000000
 
       LOG STREAM CONNECTION INFO:
         SYSTEMS CONNECTED: 1
         SYSTEM    STRUCTURE         CON CONNECTION  CONNECTION
        NAME      VERSION           ID  VERSION     STATE
        --------  ----------------  --  ----------  ----------
        MV28      0000000000000000  00  00000000    N/A
 
      LOG STREAM DATA SET INFO:
         DATA SET NAMES IN USE: IXGLOGR.WILLIN.IYLX4.DFHSHUNT.<SEQ#>
         Ext.   <SEQ#>    Lowest Blockid    Highest GMT        Highest Local
        -----  --------  ----------------  -----------------  ---------------
       *00001  A0000000  0000000000001F1E  07/16/97 12:52:22  07/16/97 13:52:
     
       NUMBER OF DATA SETS IN LOG STREAM: 1
       POSSIBLE ORPHANED LOG STREAM DATA SETS:
         NUMBER OF POSSIBLE ORPHANED LOG STREAM DATA SETS: 0

If you are using coupling facility log streams, the IXCMIAPU LIST STRUCTURE NAME(structname) DETAIL(YES) command is useful in finding the status of CICS log stream structures. For further information about these commands, see z/OS MVS Setting Up a Sysplex.

Step 5. Review SMF and RMF statistics

SMF 88 log stream statistics records and RMF coupling facility usage reports are useful for analyzing problems that are affecting performance.

Increasing the amount of coupling facility storage allocated to a structure, or the size of a staging data set, might improve the performance of both z/OS System Logger and CICS.