PH68097: CSQ0216E OR CSQ0215E WITH REASON 014CE006

APAR status

• Closed as program error.

Error description

• The certificates for AMS policies expired, so this message was
  issued:
  
  CSQ0215E cpf CSQ0CPUT Message protection failed, return code 8,
  reason 0335302F
  
  Some hours later these messages were seen:
  
  CSQ0216E cpf CSQ0CGET Message unprotection failed, return code
  8, reason 014CE006
  CSQ0215E cpf CSQ0CPT1 Message protection failed, return code 12,
  reason 014CE006
  
  The reason above means:
  
  014CE006 Insufficient memory is available.
  
  Explanation
  There is not enough memory available to allocate a required
  control block or data element.
  
  User response
  Increase the memory available to the application and then retry
  the request. Contact your service representative if the error
  persists.
  
  Even though the expired certificate issue was resolved and
  refresh of the AMS policy was attempted, the following messages
  were received:
  
  CSQ0608E cpf CSQ0DCNS Failed to load policy configuration,
  MQRC=2071 (MQRC_STORAGE_NOT_AVAILABLE)
  CSQ0636I cpf CSQ0DCNS POLICY refresh failed
  CSQ0642I cpf CSQ0DCNS REFRESH command failed
  
  The dump shows that when the application attempted to put to a
  protected queue and failed due to the expired certificate
  (CSQ0CPUT Message protection failed, return code 8, reason
  0335302F), it ended/disconnected without closing the queue.
  
  During deallocation processing, CSQMCALH calls CSQ0CALH to
  request AMS frees control blocks associated with the handle in
  the xxxxAMSM address space, however this request fails because
  deallocation processing is running on an SRB, but CSQ0CALH
  expects a TCB, resulting in the control blocks being leaked.
  
  Over time the repeated failures caused the leaked storage to
  exhaust available storage in the xxxxAMSM address space, leading
  to the later errors.
  

Local fix

Problem summary

• ****************************************************************
  * USERS AFFECTED: All users of IBM MQ for z/OS Version 10      *
  *                 Release 0 Modification 0.                    *
  ****************************************************************
  * PROBLEM DESCRIPTION: Storage leak in the xxxxAMSM address    *
  *                      space resulting in varying symptoms -   *
  *                      these can include:                      *
  *                      - Messages CSQ0215E / CSQ0216E          *
  *                        reporting AMS operations have failed  *
  *                        with reason 014CE006                  *
  *                      - Message CSQ0607E reporting            *
  *                        "Insufficient storage available"      *
  *                      - Performance degradation               *
  *                      - Abnormal queue manager termination    *
  *                        following AMS Refresh                 *
  ****************************************************************
  During implicit disconnect processing all open handles for a
  connected task are closed. When the handle was for an AMS
  protected queue, a request is queued to the AMSM address space
  to free resources in the AMSM address space related to the
  handle.
  However when this occurs on an SRB (for example, during
  deallocation processing), the request fails and the resource are
  not freed.
  

Problem conclusion

• When running on an SRB CSQMCALH is changed to queue the request
  to the AMSM address space using a service task to ensure
  successful completion.
  

Temporary fix

Comments

APAR Information

• APAR is sysrouted FROM one or more of the following:

• APAR is sysrouted TO one or more of the following:

  UO10349

Modules/Macros

• CSQ0ESTP CSQ0STRT CSQMCALH
  

Fix information

• Fixed component name

  IBM MQ Z/OS V10

• Fixed component ID

  5656MQA00

Applicable component levels