OA48280: TSO ADDRESS SPACE HANGS WITH ENQUEUE CONTENTION FOR MAJOR NAME SYSZTSOC MINOR IKJOBTAINMSG SCOPE=STEP

Obtain the fix for this APAR.

APAR status

• Closed as program error.

Error description

• TSO batch job executing a REXX EXEC hangs with Enqueue
  contention on SYSZTSOC IKJOBTAINMSG.   The REXX EXEC
  establishes an EMCS CONSOLE that is defined to receive HARDCOPY
  (ie. SYSLOG) from every system in the SYSPLEX (MSCOPE=*ALL).
  This generated a large volume of message traffic to this EMCS
  console since there were 13 systems in the sysplex.  The
  REXX EXEC then issues a command (SETOMVS) followed by a
  GETMSG to retrieve messages.
  
  The error is seen when the system is constrained causing any
  delay to the command response expected by the REXX EXEC.  The
  EMCS dataspace quickly fills up with messages from all other
  systems in the sysplex and becomes suspended.  IKJCNANT will
  start to clear the dataspace by receiving messages from the
  dataspace so new messages can arrive.  The amount of storage
  required by the IKJCNANT task trying to deal with the sysplex
  message traffic causes it to run out of private storage
  resulting in an abend878 'out of private storage' condition in
  the TSO address space. IKJCNANT receives a return code x'18'
  from the MCSOPMSG GETMSG request indicating a system problem
  so IKJCNANT attempts to deactivate the console, cleanup and
  abort the REXX exec.
  
  IKJCNANT calls IKJCNASR to deactivate the console without
  releasing the SYSZTSOC IKJOBTAINMSG enqueue then waits
  in IKJCNASR+X'272' because there is an outstanding GETMSG
  request (CONSOLE_NUMBER_OF_REQUESTS is equal to 0001).  The
  GETMSG request will never complete (i.e. the count will never
  go to 0) because it is waiting for the SYSZTSOC IKJOBTAINMSG
  enqueue.  If the address space remains in this state (with an
  active console listening to all hardcopy message with
  MSCOPE=*ALL) then the system can eventually fail with
  abend878-10 in the CONSOLE address space with CONSOLEs running
  out of private storage due to excessive message traffic.
  
  Additional symptoms:
  CONSOLE ABEND878 RC10
  IEE286I CONSOLE DELAYED MESSAGE BUFFER FULL
  IXC431I GROUP SYSMCS MEMBER JOB CONSOLE ASID 000A STALLED AT
  
  Verification Steps in dump of hung user:
  1.  IP RESOURCE ANALYSIS should show:
      RESOURCE #0034:
      NAME=MAJOR=SYSZTSOC MINOR=IKJOBTAINMSG SCOPE=STEP
      RESOURCE #0034 IS HELD BY:
      JOBNAME=JOBA MASID=0096  TCB=00990E88
      RESOURCE #0034 IS REQUIRED BY:
      JOBNAME=JOBA MASID=0096  TCB=009AE438
  2.  TCB 00990E88 will be in a WAIT in IKJCNASR+x'272'
      (HTE7790 level)  Reg9 at time of WAIT will point
      to a control block with the ID of CONSOLE.
      CONSOLE+x'5C' will contain 80000001
      TCB 009AE438 will be waiting for the ENQUEUE
  

Local fix

• Specify MSCOPE=*  instead of MSCOPE=*ALL to reduce the
  amount of message traffic
  .
  A ++APAR (AA48280) is available for HTE7780 (V1R13) and
  HTE7790 (V2R1) from L2 support.
  

Problem summary

• ****************************************************************
  * USERS AFFECTED: All users of z/OS TSOE CONSOLE command       *
  ****************************************************************
  * PROBLEM DESCRIPTION: TSO Address Space may hang with ENQ     *
  *                      contention on Majorname='SYSZTSOC' and  *
  *                      Minorname='IKJOBTAINMSG' during a TSO/E *
  *                      EMCS CONSOLE session.                   *
  ****************************************************************
  * RECOMMENDATION:                                              *
  ****************************************************************
  TSO batch job executing a REXX EXEC hangs with Enqueue
  contention on SYSZTSOC IKJOBTAINMSG. (This is referred to
  as the cache enqueue.)  The REXX EXEC establishes an EMCS
  CONSOLE that is defined to receive HARDCOPY (ie. SYSLOG) from
  every system in the SYSPLEX (MSCOPE=*ALL).  This generated a
  large volume of message traffic to this EMCS console since
  there were 13 systems in the sysplex.  The REXX EXEC then
  issues a command (SETOMVS) followed by a GETMSG to retrieve
  messages.
  -
  The error is seen when the system is constrained causing any
  delay to the command response expected by the REXX EXEC.  The
  EMCS dataspace quickly fills up with messages from all other
  systems in the sysplex and becomes suspended.  IKJCNANT will
  start to clear the dataspace by receiving messages from the
  dataspace so new messages can arrive.  The amount of storage
  required by the IKJCNANT task trying to deal with the sysplex
  message traffic causes it to run out of private storage
  resulting in an abend878 'out of private storage' condition in
  the TSO address space. IKJCNANT receives a return code x'18'
  from the MCSOPMSG GETMSG request indicating a system problem
  so IKJCNANT attempts to deactivate the console, cleanup and
  abort the REXX exec.
  -
  IKJCNANT calls IKJCNASR to deactivate the console without
  releasing the SYSZTSOC IKJOBTAINMSG enqueue then waits
  in IKJCNASR+X'272' because there is an outstanding GETMSG
  request (CONSOLE_NUMBER_OF_REQUESTS is equal to 0001).  The
  GETMSG request will never complete (i.e. the count will never
  go to 0) because it is waiting for the SYSZTSOC IKJOBTAINMSG
  enqueue at IKJCNAGW+x'168' under the EXEC task.  If the
  address space remains in this state (with an active console
  listening to all hardcopy message with MSCOPE=*ALL) then the
  system can eventually fail with abend878-10 in the CONSOLE
  address space with CONSOLEs running out of private storage
  due to excessive message traffic.
  

Problem conclusion

• IKJCNANT will release the console cache enqueue before trying
  to deactivate a console.
  

Temporary fix

• *********
  * HIPER *
  *********
  SEE LOCAL FIX:
  SPECIFY MSCOPE=*  INSTEAD OF MSCOPE=*ALL TO REDUCE THE
  AMOUNT OF MESSAGE TRAFFIC
  

Comments

APAR Information

• APAR is sysrouted FROM one or more of the following:

• APAR is sysrouted TO one or more of the following:

  UA78520 UA78521 UA78522

Modules/Macros

• IKJCNANT
  

Fix information

• Fixed component name

  TSO/E SCHEDULAR

• Fixed component ID

  566528502

Applicable component levels

• R7A0 PSY UA78520

     UP15/08/19 P F508

• R780 PSY UA78521

     UP15/08/19 P F508

• R790 PSY UA78522

     UP15/08/19 P F508

Fix is available

• Select the PTF appropriate for your component level. You will be required to sign in. Distribution on physical media is not available in all countries.