OA54865: ABEND878 RC10 RC0C OR OTHER OUT OF PRIVATE STORAGE CONDITION IN BATCH CICS REGION. IEWPLMH EYECATCHERS IN SP229.

Obtain the fix for this APAR.

APAR status

• Closed as program error.

Error description

• Customer receives an "out of z/OS storage" condition such as
  ABEND878 RC10 or RC0C in a batch CICS region.  This typically
  occurs when the initiator that the batch CICS job is running in
  has been up for an extended period of time, and the CICS region
  has been recycled under that initiator a number of times.
  
  This problem stems from an orphaning of z/OS Loader storage
  which occurs when code does a directed load of a program object
  with deferred classes.  In this environment SP229 KEY0 storage
  under system RCT TCB grows over time and across recycles of the
  CICS region. This orphaned storage contains eyecatchers such as:
  IEWPLMH, IEWPRAT, IEWPRDT, IEWLIDX, IEWPSEGM, IEWPDSIT, C_WSA,
  and IEWPGSTB.
  
  The reason for this orphaning of storage is related to how
  directed loads vs "regular" loads are handled, and how the
  presence of modules with deferred classes influences the
  processing behind directed loads in particular.
  
   - For the case of a "regular" load, the Loader component
     builds the control blocks it needs when the load is issued,
     leaves them in place while the module is in use, and will
     clean up the module/structures implicitly when the owning
     TCB terminates.
   - For the case of a directed load (also known as an explicit
     load), the issuer of the load request is responsible for
     cleaning up.  The loader does not retain any loader control
     blocks past the time that the load is completed.  CICS does
     directed loads.
   - For the case of a directed load of a module with deferred
     classes (segments), loader must keep certain control blocks
     around to represent the deferred class in case it needs to be
     instantiated at a later time.  Loader sets up these control
     blocks to be owned by the system RCT TCB.  There is no
     interface for the issuer of the directed load to use to
     free up these control blocks on termination, and because
     the storage is owned by the system RCT TCB which does not
     terminate when a batch CICS region ends, the Loader storage
     does not get implicitly freed upon CICS termination.
  
     Modules with deferred classes have become more prevalent with
     the COBOL V5 compiler.
  
  This problem can be seen in batch environments other than CICS
  if the job in question is doing directed loads of objects with
  deferred classes, and is being recycled without the initiator
  being drained.  Note that this APAR does not address a case
  where the storage described above is accumulating within a
  single instance of a batch job.
  
  ADDITIONAL SYMPTOMS: ABEND04E when issued for an out of storage
  situation.  Possibly an ABEND80A.
  
  VERIFICATION STEPS:
  
  A VSM analysis of the out of storage condition shows that
  storage is exhausted above and below the line.  Subpools that
  contain CICS DSAs are naturally large and account for most of
  the storage usage, but the problem is with the size of SP229
  key0 storage owned by the RCT TCB, the first TCB within the
  address space.  In that storage there are many IEWPLMH
  eyecatchers, as well as IEWPRAT, IEWPRDT, IEWLIDX, IEWPSEGM,
  IEWPDSIT, C_WSA, and IEWPGSTB eyecatchers.  This system-owned
  storage has been orphaned on shutdown of the batch CICS regions
  and therefore has accumulated across various runs of CICS.
  region.
  
  From an abend/SLIP dump of the out of storage condition, or from
  a console dump of a tight on storage condition, do the
  following:
  
  1) To identify the address of the RCT TCB, do:
  
     IP SUMM FORM   for the job in question,
  
     then do:  FIND 'TCB:' to locate the first TCB.  This is the
     RCT TCB.
  
  2) While in the SUMMARY FORMAT report, do: FIND IEFIIC  to
     verify that this is a batch job.  If no hits, then this
     APAR is not your problem.
  
  3) Do: IP VERBX VSMDATA 'NOG SUM ASID(nn)'  where nn is the
     decimal ASID number of the address space in question. Go to
     the bottom and page backwards until you come to the "Local
     Subpool Usage Summary".  Locate the lines for the RCT TCB.
     Here is an example, although you will have more lines in your
     report and they will not be contiguous:
  
     TCB/OWNER   SP#   KEY      BELOW    ABOVE    TOTAL
     ---------   ---   ---      -----    -----    -----
       9FD520    229    0           0  A0FD000  A0FD000
       9FD520    230    0        3000   791000   794000
       9FD520    230    1        3000    37000    3A000
  
     Note that SP229 Key0 is particularly large.
  
  4) Locate the description of storage in that subpool,
     formatted higher in the report.  (Doing a find on the
     subpool total can be a good way to locate the summary
     line, for example, FIND A0FD000 FIRST .  The summary
     line is at the end of the section so scroll back a few
     pages and IP LIST the storage indicated by the ADDR and
     SIZE fields of several sample DQEs.  If this APAR is your
     problem, you will see IEWLPLMH eyecatchers at the start of
     the storage, and other IEW eyecatchers appearing after that.
     Notably you will see C_WSA eyecatchers, which represent
     deferred classes.
  
  5) FIND 'LOCAL STORAGE MAP' in the VSMDATA report.  Note
     the ELSQA bottom.  FIND 'Key Information' in the
     VSMDATA report.  Note the EHIAL field.
  
     IP LIST xxxxxxxx LEN(X'yyyyyyyy')
  
     where xxxxxxxx is the ELSQA bottom and yyyyyyyy is the
     EHIAL field.   Do: FIND IEWLFMD ALL   against this
     output and note the count that is returned.  It should
     be a number in the 1000's if there has been significant
     growth due to this APAR.
  
  6) Spot check some IEWLFMD's. IEWLFMD+x'47' bit X'10' being
     on indicates this module was loaded via directed load.
  
  If the above steps check out, then you are likely experiencing
  the problem described by this APAR.
  

Local fix

• Recycling the initiator will clean up the orphaned storage.
  

Problem summary

• ****************************************************************
  * USERS AFFECTED: Users running z/OS HBB77A0.                  *
  ****************************************************************
  * PROBLEM DESCRIPTION: Abend878 RC0C or RC10 SP229 out of      *
  *                      storage condition in a batch CICS       *
  *                      region.                                 *
  ****************************************************************
  * RECOMMENDATION:                                              *
  ****************************************************************
  The out of storage condition is due to Loader storage not
  being cleaned up for a directed load of a program object with
  deferred classes, thus leaving the storage orphaned.
  Eventually SP229 Key0 private storage gets filled up with
  IEW eyecatchers.
  
  This seems to occur more frequently in a batch CICS job
  running under an initiator and the job has run for an extended
  period of time, including having the CICS region recycled
  numerous times.
  

Problem conclusion

• Add function to Loader to perform cleanup of Loader related
  storage for directed load of program objects with deferred
  segments.
  
  This problem is already addressed in z/OS HBB77B0.
  

Temporary fix

Comments

APAR Information

• APAR is sysrouted FROM one or more of the following:

• APAR is sysrouted TO one or more of the following:

  UA96916

Modules/Macros

• IEWLCLDX IEWLCLNP
  

Fix information

• Fixed component name

  LOADER

• Fixed component ID

  5752SCLDR

Applicable component levels

• R7A0 PSY UA96916

     UP18/08/01 P F807

Fix is available

• Select the PTF appropriate for your component level. You will be required to sign in. Distribution on physical media is not available in all countries.