OA61459: ABEND0DC RSN08 EOM ABEND1C5 RSN00070021 MSGIXG063I MSGIXG062A  MSGIEF402I RRS LATCH CONTENTION

APAR status

• Closed as unreproducible in next release.

Error description

• The WAS Liberty Server terminated with EOM (end of memory)
  ABEND0DC RSN08, reported via msgIEF402I:
  IEF402I JOBNAME FAILED IN ADDRESS SPACE 0123
          SYSTEM ABEND S0DC - REASON CODE 08
  
  Some time later, System Logger issued ABEND1C5 RSN00070021 =
  TRX_OffloadNotProceding, reported via msgIXG063I:
  IXG063I LOGGER ABENDED AND REQUESTED AN SVC DUMP
          WHILE PROCESSING
          LOGSTREAM: ATR.gname.DELAYED.UR
          STRUCTURE: RRSLOG1
          GROUP: PRODUCTION
          MODULE=IXGF1TRX,ABEND=S01C5,
          REASON=00070021
  
  This ABEND was issued by System Logger when it was unable to
  make progress clearing the staging dataset being used for
  duplexing of the ATR.gname.DELAYED.UR (DELAYED UR) log stream
  after log stream data had been offloaded.  Later, when the
  staging dataset filled to capacity, System Logger was no longer
  able to accept IXGWRITE requests for the DELAYED UR log stream,
  since the IXGWRITE data could not be duplexed.
  
  Because IXGWRITE requests for the DELAYED UR log stream were not
  able to proceed, RRS could no longer perform logpoints during
  transaction syncpoint processing.  RRS latch contention was
  observed against its DELAYED UR log stream latch and various UR
  latches (SYS.ATRURCPO.nnnnnnnn) representing the hung syncpoint
  transactions.
  
  Example RRS Contention Display Output:
  (Note: Output truncated and modified for readability.)
  
  ISG343I 23.57.56 GRS STATUS
  NO ENQ RESOURCE CONTENTION EXISTS
  NO REQUESTS PENDING FOR ISGLOCK STRUCTURE
  
  LATCH SET NAME:  SYS.ATR.gname.DELAYED.UR
  CREATOR JOBNAME: RRS       CREATOR ASID: 002A
    LATCH NUMBER:  0
  REQUESTOR ASID EXC/    OWN/ WORKUNIT TCB ELAPSED
                 SHR     WAIT              TIME
  JOBNAME   0124 SHARED  OWN  009AB328 Y   1:26:28
  JOBNAME   0124 SHARED  OWN  009BC280 Y   1:26:28
  
  LATCH SET NAME:  SYS.ATRURCPO.00000001
  CREATOR JOBNAME: RRS       CREATOR ASID: 002A
    LATCH NUMBER:  21
  REQUESTOR ASID EXC/    OWN/ WORKUNIT TCB ELAPSED
                 SHR     WAIT              TIME
  JOBNAME   0124 EXCLUSVE OWN 009BBAD0  Y  1:28:28
  JOBNAME   0124 SHARED  WAIT 0099C580  Y  1:01:20
    LATCH NUMBER:  38
  REQUESTOR ASID EXC/    OWN/ WORKUNIT TCB ELAPSED
                 SHR     WAIT              TIME
  JOBNAME   0124 EXCLUSVE OWN 009B25B0  Y  1:30:38
  JOBNAME   0124 SHARED  WAIT 009A8580  Y  1:03:30
  
  These RRS latch resources are originally held under the
  application jobname requesting sycnpoint commit, as shown in
  this contention display.  When the latches held are associated
  with IXGWRITE processing, as in this case, RRS will assume
  ownership of the syncpoint thread and associated latches if the
  address space holding the latches terminates.  As a result,
  subsequent contention displays will show the same latches held,
  but, instead, the latch holders will reflect the ownership
  change to RRS.
  
  In order to clear the latch contention, RRS was FORCEd.  During
  FORCE processing, disconnect processing for the RRS DELAYED UR
  log stream was initiated, as expected.  System Logger detected
  and reported outstanding transactions associated with RRS:
  
  IXG062A LOGGER CLEANUP NOT PROGRESSING FOR RRS,
          3 ASYNC EVENTS NOT DONE. REPLY TO
          CONTINUE TERMINATION.
  
  After the WTOR reply was entered, Logger successfully completed
  disconnection processing for the RRS connection to the DELAYED
  UR log stream.  Upon RRS restart, the DELAYED UR log stream
  connection was established, and RRS resumed normal processing.
  
  KNOWN IMPACT:
  From a System Logger perspective, I/O to a staging dataset
  initiated during IXGWRITE processing will remain outstanding if:
  1) an ABEND0DC RSN08 or another EOM termination occurs in an
  address space that is running in cross memory mode to Logger and
  2) the address space in EOM termination is not the same as the
  address space that has established the log stream connection and
  3) the EOM termination occurs after Logger has given control to
  Media Manager to duplex write a log stream record to the staging
  dataset but before the I/O is successfully initiated.
  
  Under these conditions, the outstanding I/O will prevent System
  Logger from clearing the staging dataset.  As a result, when the
  staging dataset fills to capacity, System Logger will not be
  able to accept any IXGWRITE requests for that log stream, since
  it will not be able to provide the required log stream data
  duplexing.
  
  VERIFICATION STEPS:
  This APAR describes a flow of events and impact to the system
  using the lens of a potential scenario where it could occur.
  Any scenario that meets the criteria described by these
  verification steps can considered a match for the problem
  described by this APAR.
  
  1. Verify an ABEND0DC RSN08 or other EOM termination has
  occurred in an address space where log stream IXGWRITE requests
  are being run in a cross-memory environment to System Logger.
  
  2. Verify the address space entering EOM termination differs
  from the address space connected to the log stream associated
  with the IXGWRITE request.  For example, a TCB in the WAS
  Liberty Server is running an application that invokes RRS.  RRS
  processing running under the WAS TCB then initiates an IXGWRITE
  request to the DELAYED UR log stream, using an already
  established connection by RRS to the log stream.
  
  3. Verify staging dataset duplexing is being used for the log
  stream target of the IXGWRITE request.
  
  4. In a dump of the System Logger address space, for the
  IxgStdVB associated with the log stream staging dataset, verify
  there is at least one IxgStdVB_CI_Array Entry with values
  IxgStdVB_Block_Id = 00000000 00000000 and IxgStdVB_CI_IOini = 1.
  
  5. In the dump of the System Logger address space, verify the
  IXGWRITE RQE associated with the IxgStdVB_CI_Array Entry
  identified in step #4 was initiated prior to the ABEND0DC RSN08
  or other EOM termination event.
  

Local fix

• BYPASS/CIRCUMVENTION:
  IBM does not recommend using staging dataset duplexing for the
  ATR.gname.DELAYED.UR and ATR.gname.MAIN.UR log streams, as they
  are written to during transaction processing under the
  transaction threads.  The I/O cost of staging dataset duplexing
  can present a negative performance impact. 
  
  Following the IBM recommendation to avoid staging dataset
  duplexing of these two RRS log streams will circumvent the
  specific scenario described by this APAR.
  
  Generally, the problem described by this APAR can be avoided by
  refraining from using staging dataset duplexing for a log stream
  that is the target of IXGWRITE requests running under a HOME
  ASID that differs from the log stream connecting ASID.
  
  RECOVERY ACTION:
  1) Disconnect from the log stream by either stopping the
  application that owns the connection to the log stream to cause
  a log stream disconnect or using an application function to
  cause a log stream disconnect.
  
  2) If the connection to the log stream remains after attempting
  a normal log stream disconnect, issue:
  SETLOGR FORCE,DISCONNECT,LSN=logstream_name
  
  3) Restart the application, which will drive a reconnection to
  the log stream, or use an application function to cause a log
  stream connect.
  

Problem summary

• ****************************************************************
  * USERS AFFECTED: Users of HBB77B0 and above supported         *
  *                 releases using System Logger Services        *
  ****************************************************************
  * PROBLEM DESCRIPTION: MSGIXG063I ABEND1C5 RSN00070021         *
  *                      as a result of an outstanding I/O       *
  *                      to a staging dataset                    *
  ****************************************************************
  * RECOMMENDATION:                                              *
  ****************************************************************
  While attempting to write a log stream record to the staging
  dataset, EOM termination occurred in an address space that
  was running in cross memory mode to Logger, resulting in
  Logger failing to clear the staging dataset (MSGIXG063I).
  

Problem conclusion

Temporary fix

Comments

• This APAR is being closed UR1 with the awareness of the
  submitting customer.  This means that a fix to this APAR is
  expected to be delivered from IBM in a release (if any) to be
  available within the next 36 months.
  

APAR Information

• APAR is sysrouted FROM one or more of the following:

• APAR is sysrouted TO one or more of the following:

Fix information

• Fixed component name

  SYSTEM LOGGER

• Fixed component ID

  5752SCLOG

Applicable component levels