OA47615: ABEND0C4-4 ABEND0C4 PIC4 BPXTXROP LATCH CONTENTION MOUNT LATCH FILE SYSTEM LATCH DEADLOCK FLKSOTCB

Obtain the fix for this APAR.

APAR status

• Closed as program error.

Error description

• Customer received two abend0C4-4 in csect BPXTXROP:
  
  IEA794I SVC DUMP HAS CAPTURED:
  DUMPID=001 REQUESTED BY JOB (OMVS    )
  DUMP TITLE=COMPON=BPX,COMPID=SCPX1,ISSUER=BPXMIPCE,
  MODULE=BPXTXROP+0A06,ABEND=S00C4,REASON=00000004
  .
  (this abend was in the recovery flow following the initial
  abend0C4-4)
  IEA794I SVC DUMP HAS CAPTURED:
  DUMPID=002 REQUESTED BY JOB (OMVS    )
  DUMP TITLE=COMPON=BPX,COMPID=SCPX1,ISSUER=BPXMIPCE,
  MODULE=BPXTXROP+4096,ABEND=S00C4,REASON=00000004
  
  The abends are caused by an invalid FlksOTCB pointer resulting
  from the double use of a Pprt structure.  We held the FILE
  SYSTEM latch as part of the normal flow at this point.  The
  initial abend should cause recovery processing for BPXTXROP to
  clean up any resources (latches) that it might hold.  The
  subsequent abend in recovery caused us to not clear information
  about the FILE SYSTEM latch from the Flks.
  
  The next work processed by OMVS TCB 008D4028 then picked up the
  residual latch request information from the Flks for the FILE
  SYSTEM latch.  OMVS TCB 008D4028 began processing a new request
  that validly required the MOUNT latch, resulting in the
  deadlock.  Note the ASID (010C) of the FILE SYSTEM latch holder
  in the 'ip analyze resoucee' output below.
  
  Following these abends, msgBPXM056E was seen:
  
  *BPXM056E UNIX SYSTEM SERVICES LATCH CONTENTION DETECTED
  
  A 'D OMVS,W' command showed MOUNT latch contention:
  
  BPXO063I
  OMVS     0010 ACTIVE
  MOUNT LATCH ACTIVITY:
      USER   ASID     TCB           REASON
  ---------------------------------------------
   HOLDER:
    OMVS     0010  008D2E88     FileSys Unmount
       TIME: 2015/04/16.16.43.55.
       IS DOING: FS Latch Wait-Latch 846
       FILE SYSTEM: OMVS.HOME.DB4ADM
  WAITER(S):
  OMVS     0010  008D4028     FileSys Unmount
  
  
  Review of the dump shows (from ip analyze resource) a deadlock:
  
  RESOURCE #0025:
    NAME=SYS.BPX.A000.FSLIT.FILESYS.LSN  ASID=0010  Latch#=2
  
  RESOURCE #0025 IS HELD BY:
   JOBNAME=OMVS      ASID=0010  TCB=008D2E88
    DATA=EXCLUSIVE  RETADDR=AD6DCA60  REQID=0010000013C75770
  
  RESOURCE #0025 IS REQUIRED BY:
   JOBNAME=OMVS      ASID=0010  TCB=008D4028
    DATA=EXCLUSIVE  RETADDR=AD6E341E  REQID=0010000013C74DD0
  
  RESOURCE #0026:
     NAME=SYS.BPX.A000.FSLIT.FILESYS.LSN  ASID=0010
     Latch#=846:(ID NOT AVAILABLE)
  
  RESOURCE #0026 IS HELD BY:
    JOBNAME=OMVS      ASID=0010  TCB=008D4028
     DATA=SHARED     RETADDR=AD6C0BA6  REQID=010C000013C74DD0
                                             ****
  
  RESOURCE #0026 IS REQUIRED BY:
    JOBNAME=OMVS      ASID=0010  TCB=008D2E88
     DATA=EXCLUSIVE  RETADDR=AD5CB986  REQID=0010000013C75770
  
  An IPL was required to break the latch deadlock.
  
  
  Verification steps:
  Review PPRT of the abending thread.
  - in the Flks, FlksReqIDAsid will not match the kernel asid.
  - PprtSyscallCode may reflect x'00000065'
  - if sysomvs ctrace is active, you may see a BPX1MPC call in the
     syscall history for the failing TCB
  - FlksOTCB does not match STCBOTCB for the failing OMVS task
  

Local fix

• FINREVERSAL OA45440
  

Problem summary

• ****************************************************************
  * USERS AFFECTED: All users of z/OS UNIX System Services for   *
  *                 HBB7780 and HBB7790                          *
  ****************************************************************
  * PROBLEM DESCRIPTION: ABEND0C4 or other unpredictable ABENDs  *
  *                      due to double use of a PPRT or a KSER.  *
  *                      .                                       *
  *                      ABENDEC6 out of SMFRE because of a      *
  *                      double free of a FUPT or a KSER or      *
  *                      a PPRT.                                 *
  ****************************************************************
  * RECOMMENDATION:                                              *
  ****************************************************************
  During process termination, PRTRM calls FSPTX to do file
  system cleanup for EndOfProcess ( Pix#EEOP ), including
  freeing the FUPT.  When we get back from the FSPTX call, the
  OAPBFUPT is cleared.
  .
  There are two issues here:
  - if an abend occurs in FSPTX after the FUPT is freed, the
   OAPBFUPT will still be set.  If we end up in EOM, RRTRM will
   call PRTRM with an indicator that the FUPT needs to be freed.
   This will lead to ABENDEC6 RSN0E0705CC attempting to free the
   FUPT again.
  - if an abend occurs in FSPTX and we percolate to BPXPRTRM
   9-er, we will call pseudodubdown to free the pseudo control
   blocks, including the pseudo PPRT.  If an ABEND then occurs
   and we end up in EOM, RRTRM will call PRTRM indicating the
   FUPT needs to be freed.  If the FUPT was not freed yet, then
   we go ahead and free the FUPT.  We complete process
   termination and call pseudodubdown.  We free the PPRT which
   had already been freed.  If the PPRT was already reallocated,
   then the PPRT can end up being use by multiple processes.
  

Problem conclusion

• FSPTX is updated to clear the OapbFupt prior to freeing it.
  This will prevent double frees of the FUPT.
  PRTRM is updated to clear the OapbFUPT in the 9er.  This way,
  if we end up in EOM after calling pseduodubdown, then PRTRM
  (under EOM) will not attempt to free the pseduo blocks again.
  

Temporary fix

Comments

APAR Information

• APAR is sysrouted FROM one or more of the following:

• APAR is sysrouted TO one or more of the following:

  UA77931 UA77932

Modules/Macros

• BPXFSPTX BPXPRTRM
  

Fix information

• Fixed component name

  OPENMVS SYS SRV

• Fixed component ID

  5695SCPX1

Applicable component levels

• R780 PSY UA77931

     UP15/07/08 P F507

• R790 PSY UA77932

     UP15/07/08 P F507

Fix is available

• Select the PTF appropriate for your component level. You will be required to sign in. Distribution on physical media is not available in all countries.