II10752: ICF CATALOG PERFORMANCE PROBLEMS - CASPERFM

APAR status

• Closed as canceled.

Error description

• Catalog CAS informational infoapar
  This APAR is created to document what information is required to
  report a Catalog Perfomance problem when running a supported
  version of DFSMS supplied with z/OS. Current supported versions
  of z/OS are 1.12 and higher releases.
  .
  Refer to the list of items below, following them step by step.
  .
  1. Document the following specific items for each of these:
       a. What information indicated an increase in CPU usage ?
       b. Was this increase noticeable immediately after upgrading
          to a new release of z/OS ?
       c. What release level was used as base for comparison ?
       d. What are the PUT levels of the base ?
       e. Were any CHANGES made in the catalog environment ?
     a. What information indicated an increase in CPU usage?
     b. Was this increase noticeable immediately after upgrading
        to a new release of MVS, DFSMS/MVS, OS/390 or z/OS ?
     c. What release level was used as base for comparison?
     d. What are the PUT levels of the base?
     e. Were any CHANGES made in the catalog environment?
        - any new shared catalogs, or old catalogs that are now
          shared
        - VLF or ISC changes catalogs
  NOTE: SMF record type41 subtype3 gives details on VLF
        hit ratios and trimming for each VLF class.
        based on this info VLF documentation may suggest
        increasing the VLF MAXVIRT value for the IGGCAS
        VLF CLASS. Refer to VLF documentation for details.
        - STRNO changes to catalogs
        - CATMAX or TASKMAX changes
        - any other relevant changes to the system environment
     f. Have there been any increases in workload on the system
        (i.e.,new applications,more TSO users,more batch workload)
     g. Have you noticed any increases in CPU consumption for
        other system address spaces (such as GRS,SMS,MASTER,JES) ?
     h. Are there specific timeframes of increased Catalog CPU
        usage?
     i. Are there specific jobs or users affected?
     j. Are multi-system GRS or MIM being used for sharing?
     k. Is this affecting overall system performance ?
     l. Do you have any DASD UCBs specified as LOCANY=YES in HCD ?
        If so, when were the UCBs put above the line ?
     If the answers to these questions point to an overall
     increase in CAS CPU consumption (as opposed to specific users
     or jobs) then proceed to step 3 and add the keyword CASPERFM
     to the scratch pad. Otherwise, refer to "Possible Causes and
     Solutions" discussed further in this apar.
  .
  2. Provide comparison data of CAS CPU consumption between the
     base (previous) release and new DFSMS release. This
     information should represent total weekly or monthly CPU
     consumption for CAS broken down by TCB and SRB.  If using
     RMF, this is easiest if the Catalog address space is in a
     report performance group. You can use SLR (EPDM or
     Performance Monitor for MVS), MICS, MXG or other similar
     tools.
  .
  3. Provide comparison data of CAS storage size between the base
     or previous release and new release of DFSMS.
  .
  4. Provide comparison data of CAS I/O rates between the base
     or previous release and new release of DFSMS.
  .
  5. If OMEGAMON is available, then do an INSPECT of CAS drilling
     down on a few of the active TCBs to a CSECT level and the
     offsets within those CSECTs that are consuming CPU. TMON/MVS
     also has a similiar capability as may other monitors.
  .
  Options for further Diagnosis:
  1. GTF Trace the User and Catalog Address Space for:
     DSP,SVC=(19,20,26,48,56,99) and set the following SLIP.
     SLIP SET,IF,JOBNAME=jjjjj,ACTION=TRACE,LPAMOD=(IGC0002F,xxx),
          TRDATA=(STD,REGS,
          12R?+278?,+14,                    <===CAMLST
          12R?+278?+4?,+2C,                 <===ENTRY NAME
          12R?+270?,+1C,                    <===CTGPL
          11R?+CD4?,+1000,                  <===GFL workarea
          12R?+270?+4?,+2C,                 <===ENTRY NAME
          12R?+260,+263),END                <===CATMAN RC/RSN
          Where: jjjjj = the batch jobname or TSO userid
                 xxx = the offset of procedure RESMCNTL in module
                       IGC0002F which is X'728' for all current
                       releases of DFSMS (z/OS 1.11 thru 1.13 at
  .                    PTF level UA60462, UA60463, UA60464)
   Start GTF with the following options:
      TRACE=SLIP,DSP,SVCP,JOBNAMEP
      WHEN PROMPTED FOR A REPLY, ENTER:
     R XX,SVC=(19,20,26,48,56,99),JOBNAME=(JJJJ,CATALOG),END.
        Added: DSP to GTF Trace 1/5/1998 DJD
  .
  2. Examine the time from the start of the SVC26 until the
     instruction fetch entry (see step 1). The time for that
     should be in the range of 10-15 milliseconds.
  .
  3. CCW Trace the device where the Catalog resides. Multiple
     reads to the VVDS are normal whereas continually rereading
     the same BCS record is not.
  .
  Possible Causes and Solutions:
  1. If it all possible convert from GRS ring to GRS star. The
     performance of GRS star is much better than GRS ring.
  .
  2. If you must use GRS ring check to see if the GRS Residency
     Time Value (RESMIL) parameter is set too high. Refer to
     MVS Planning: Global Resource Serialization for details but
     but this should be set at 0 or OFF. Note that this only
     applies to a GRS Ring configuration
  .
  3. The service class for CATALOG or GRS has been changed from
     the default of SYSTEM.
  .
  4. CATMAX is too low. Issue an F CATALOG,ALLOCATED to see how
     many Catalogs are open and allocated. Increase CATMAX above
     the number of active Catalogs.
  .
  5. Catalog STRNO is too low. This is indicated by queue wait
     times on SYSZRPLW/catname. Consult RMF ENQ delay reports
     to determine specific catalogs that might need to have
     STRNO bumped. For a highly active Catalog, the default value
     of 2 should be increased to 5 or 7. Issue:
         LISTCAT ENT(catname) ALL CAT(catname)
     to find the value it has been set to.
  .
  6. VLF or ISC not active. F CATALOG,VLF and F CATALOG,ISC can
     activate these options.
  .
  7. Catalog Enqueue and Dequeue Resources
  .
    The resource SYSIGGV2 MUST be converted to a SYSTEMS enqueue
    when using catalogs in ECS mode. Failure to do so will result
    in damage to catalogs in ECS mode.
    This is applicable to all levels of DFSMS that support ECS
    (HDZ11F0, HDZ11G0, HDZ11H0, HDZ11J0, HDZ11K0, HDZ1180,
    HDZ1190, HDZ1A10, HDZ1B10, HDZ1C10 HDZ1D10).
                                                             .
  The resources SYSZVVDS and SYSVTOC MUST be treated the same
  way, either both converted or both excluded to prevent deadlocks
                                                                 .
  See info apar II14297 for more discussion on Catalog and GRS
  resources.
  
  .
  8.  ISGAUDIT use.  Due to the high number of enqueues and
  done by the catalog address space, if ISGAUDIT is being used
  and the SYSZVVDS and/or SYSIGGV2 names are not being filtered
  out, it is possible to see high CPU numbers attributed to the
  Catalog Address Space as a result of monitoring these.  These
  can be removed from the monitor list to correct this.
  .
  9. The dispatching priority for CAS should be 'FF'.
  .
  10. Ensure there are no PER Slip Traps enabled, especially if
      more than one address space is involved.
  .
  11. Check to see if an IBM product called SMARTBATCH is
       installed. (10/98)
  
  12. If you are using MIM exits please see GRS apar OW56028 and
      OA01861.  z/OS 1.2 and higher releases have reported
      GRS/Catalog performance problems using GRS/MIM exits without
      these apars being installed.
      CA PTFS TD45048 and TD45064 address thisproblem with the
      GRS PTFs for the above GRS apars.
      See MIM info apar QI24715 that provides detail from a MIM
      perspective.
  .
  13.  WTO Buffer Shortage: This is indicated by the presence
       of message IEA405E.  When this occurs, the system issues
       a GQSCAN macro, which requires the CEDQ lock - this is a
       normal lock required by ENQ processing.  Therefore due
       to the high number of ENQs required by normal SVC26
       processing, catalog processing slows down and CPU usage
       appears high for the catalog address space.
  
  14. For users of MIM, MIM apar TD46021 should be installed.
  
  15. Beginning with HDZ11G0 and in subsequent versions of DFSMS
      I/O statistics for catalogs and the Catalog Address Space
      will appear differently than earlier releases. Prior to
      z/OS 1.3 VSAM did the I/O to VSAM data sets, including
      catalogs. Starting with HDZ11G0 VSAM uses Media Manager to
      do all I/O. Prior to HDZ11G0 VSAM specifically omits the
      collection of Start-I/O or block counts when accessing a
      catalog. Media Manager does not differentiate between I/O to
      catalog or another type of data set. You may now see higher
      I/O counts for Catalog Address Space I/O requests. The
      actual I/O rates have not changed, simply the reporting of
      them.
                                                              .
  16. To improve IDCAMS EXPORT processing of catalogs, specify
      the BUFND, BUFNI and BUFNO parameters. To specify BUFND and
      BUFNI you will need to use the INFILE parameter for EXPORT.
      Sample JCL is below:
  //EXPRTCAT EXEC PGM=IDCAMS
  //SYSPRINT DD   SYSOUT=*
  //INCAT    DD   DSN=MY.CATALOG,DISP=SHR,
  //  AMP=('BUFND=XXX','BUFNI=YYY')
  //OUTCAT   DD   DSN=MY.EXPORTED.CATALOG,DISP=(NEW,CATLG),
  //  UNIT=SYSDA,SPACE=(CYL,(10,10)),BUFNO=ZZ
  //SYSIN    DD   *
    EXPORT MY.CATALOG -
    INFILE(INCAT) -
    OUTFILE(OUTCAT) -
    TEMPORARY
  /*
    For BUFND (XXX) use the number of CI's per CA for data
  component of the catalog. For BUFNI, compute the number of
  index records by dividing the High Used RBA of the index
  component by the index component CISIZE and add a value of 5 to
  10 to that calculation. For BUFNO (ZZ) use a value in the range
  of 30 to 40.
  CATKEYS: CATINFO CATPERF
  

Local fix

Problem summary

Problem conclusion

Temporary fix

Comments

• Closed for internet viewing
  

APAR Information

• APAR is sysrouted FROM one or more of the following:

• APAR is sysrouted TO one or more of the following:

Fix information

Applicable component levels