OA65914: RSN0C IXLM2LRS GLOBAL ENQ CONTENTION IXL055I IXL053E ABEND026 RSN090C0053

Obtain the fix for this APAR.

APAR status

• Closed as program error.

Error description

• During a shutdown of DB2 when entering the phase of DBM1
  termination a large number of requests are being made to GRS for
  DEQ processing. This results in a DEQ PURGE for a greater than
  60000 SYSDSN and SYSVSAM GLOBAL resources.
  For each of these global resources we need to clean up the lock
  byte on the ISGLOCK structure. This results in a burst of
  activity to the coupling facility with a high percentage of the
  request going ASYNC. Backend completion for each of these ASYNC
  requests requires an SRB dispatch to drive the GRS completion
  exit. The burst of backend completions results in local lock
  contention for storage obtains of x'A000' bytes in GRS private
  LSQA by IXLM2LRS. This may result in ABEND878 RSN0C in GRS and
  global enqueue contention at a Sysplex-wide scope. The system
  incurring the ABEND878 RSN0C needs to be partitioned out of the
  Sysplex to resolve the contention.
  
  
  
  KNOWN IMPACT:
  System outage
  Sysplex wide global contention
  CF subchannel delays
  
  VERIFICATION STEPS:
  1. Check for ABEND878 RSN0C at IXLM2LRS+x'F4' for x'A000' bytes
  in subpool 205(x'CD') in GRS.
  2. Run IP VERBX GRSTRACE 'SYSTEMS JOBNAME(xxxxDBM1) DETAIL'
  report and look for 'pending QORE:'.
  3. Get the QWB@ from the QORE and examine the following fields:
  #totalstarglobalqores...+088 Total number of ENQs that need to
  be cleaned up
  qoreentrycount..........+08C
  qorelockingcount........+090 total # of IXLLOCK requests started
  qorecompletecount.......+094 total # of IXLLOCK requests
  completed
  4. Verify in either RMF MON III data or SMF74.4 records there is
  a high rate of request going ASYNC to ISGLOCK structure in the
  timeframe of the problem.
  
  ADDITIONAL SYMPTOMS:
  With function CFSTRQMON enabled it may result in IXL055I and
  IXL053E and following SVC DUMP with ABEND026 RSN090C0053 is
  taken. XES will issue these messages when we consistently see
  100 waiting requests queued for over 1000 microseconds over a 24
  second time period.
  

Local fix

• BYPASS/CIRCUMVENTION:
  We have found in some cases increasing the number of
  PATHs/subchannels could prevent or lessen the impact.
  

Problem summary

• ****************************************************************
  * USERS AFFECTED:                                              *
  * Users of JBB6602 and above, PTFs will be                     *
  * provided for HBB77D0 and above.                              *
  ****************************************************************
  * PROBLEM DESCRIPTION:                                         *
  * ABEND878 RSN0C in GRS and global                             *
  * enqueue contention at a Sysplex-wide                         *
  * scope following DB2 shutdown.                                *
  ****************************************************************
  * RECOMMENDATION:                                              *
  ****************************************************************
  During a shutdown of DB2 when entering the phase of DBM1
  termination a large number of requests are being made to GRS for
  DEQ processing. This results in a DEQ PURGE for greater than
  60000 SYSDSN and SYSVSAM GLOBAL resources.
  
  For each of these global resources, GRS needs to clean up the
  lock byte on the ISGLOCK structure. This results in a burst of
  activity to the coupling facility with a high percentage of the
  request going ASYNC. Backend completion for each of these ASYNC
  requests requires an SRB dispatch to drive the GRS completion
  exit. The burst of backend completions results in local lock
  contention for storage obtains of x'A000' bytes in GRS private
  LSQA by IXLM2LRS. This may result in ABEND878 RSN0C in GRS and
  global enqueue contention at a Sysplex-wide scope.
  

Problem conclusion

• GRS DEQ purge processing is changed to throttle the number of
  ASYNC IXLLOCK requests being processed at the same time.
  When the threshold is reached new IXLLOCK requests will be
  halted and resumed after the total outstanding ASYNC requests
  goes below the minimum threshold. This will reduce the number
  of completion SRBs active at the same time, which will reduce
  storage usage in GRS.
  

Temporary fix

Comments

• ×**** PE24/12/17 FIX IN ERROR. SEE APAR OA67329  FOR DESCRIPTION
  ×**** PE24/12/17 FIX IN ERROR. SEE APAR OA67329  FOR DESCRIPTION
  

APAR Information

• APAR is sysrouted FROM one or more of the following:

• APAR is sysrouted TO one or more of the following:

  UJ96195 UJ96203

Modules/Macros

• ISGTDFMT ISGVCX   ISGWDR   ISGWDEQ  ISGGVT   ISGTTRAR ISGXSTAX
  ISGWENQ  ISGDMOTH ISGGRS00 ISGWRP   ISGTFMTT ISGTTRAC
  

Fix information

• Fixed component name

  GRS

• Fixed component ID

  5752SCSDS

Applicable component levels

• R7D0 PSY UJ96203

     UP24/11/06 P F411 ¢

• R7E0 PSY UJ96195

     UP24/11/06 P F411 ¢

Fix is available

• Select the PTF appropriate for your component level. You will be required to sign in. Distribution on physical media is not available in all countries.