PH61431: AFTER RRS HAS BEEN CANCELED, AN INTEGRATION SERVER ON Z/OS MAY APPEAR TO BE HUNG AND EVENTUALLY ABEND.

APAR status

• Closed as program error.

Error description

• If RRS is ended, the Integration Node will lose its connection
  to the queue manager and the Integration Servers will stop their
  message flows before shutting down.
  
  The Integration Node will then monitor for the availability of
  the MQ queue manager and once its connection to its queue
  manager is re-established, the Node will restart
  
  the Servers.
  
  
  However, it is possible that an Integration Server may appear
  to be hung if it has not been able to stop a message flow,
  though even in this case, we do see the Server is eventually
  terminated via an abend after multiple iterations of the
  BIP2111E '~ImbDataFlowManager;lockForUpdate;delay' message.
  
  
  Initially IIB logs the following expected MQ related errors
  (which may be repeated many times for each deployed MQInput
  flow):
  
  +BIP2628W (Msg 1/2) <NODE> <IS> <thd_id> EXCEPTION CONDITION
  DETECTED ON INPUT NODE '<Message_Flow>.MQ Input'.
  +BIP2606E (Msg 2/2) <NODE> <IS> <thd_id> UNABLE TO GET MESSAGE
  FROM WEBSPHERE MQ QUEUE 'MQ.QUEUE.NAME': MQCC=2; MQRC=2195; NODE
  '<Message_Flow>.MQ Input'.
  
  
  An attempt to publish an integration node status message then
  fails and message BIP9283S is logged:
  
  
  +BIP2152I (Msg 1/1) <NODE> <IS> <thd_id> CONFIGURATION MESSAGE
  RECEIVED FROM INTEGRATION NODE.
  +BIP2153I (Msg 1/1) <NODE> <IS> <thd_id> ABOUT TO 'Stop' AN
  INTEGRATION SERVER.
  +BIP7094W (Msg 1/1) <NODE> <IS> <thd_id> EXCEPTION CONDITION
  DETECTED WHILE PUBLISHING AN INTEGRATION NODE STATUS MESSAGE.
  +BIP2121E (Msg 1/2) <NODE> <IS> <thd_id> THE THREAD BOOTSTRAP
  CODE CAUGHT AN UNHANDLED EXCEPTION ON THREAD NUMBER
  '12345678910' WITH NAME ''.
  +BIP9283S (Msg 2/2) <NODE> <IS> <thd_id> UNABLE TO SET CONTEXT
  WITH MQ ON QUEUE MANAGER '<QMGR>': MQCC=2; MQRC=2195.
  
  The expected behaviour at this point is that the deployed
  message flows will be stopped and the Integration Server ended
  whilst the Integration Node monitors for the availability of the
  MQ queue manager. But if a message flow cannot be stopped, the
  shut down of the Integration Server is delayed for over an hour
  with the following messages logged every 5 minutes:
  
  +BIP2111E (Msg 1/2) <NODE> <IS> 0 INTEGRATION NODE INTERNAL
  ERROR: DIAGNOSTIC INFORMATION
  '&#126;ImbDataFlowManager;lockForUpdate;delay'.
  +BIP2242E (Msg 2/2) <NODE> <IS> 0 MESSAGE FLOW '<Message_Flow>'
  COULD NOT BE RECONFIGURED BECAUSE IT REMAINED BUSY FOR LONGER
  THAN THE TIMEOUT PERIOD OF '300000'
  
  Eventually, after multiple iterations of the BIP2111E/BIP2242E
  messages, the Integration Server will be terminated and an abend
  with the following stack will be generated:
  
  
  ImbDumpAndTerminate(int,const void*)
  ImbAbend::terminateProcessInternal(const void*,bool)
  ImbAbend::logSystemErrorAndTerminateProcessInternal(const ch
  ImbDataFlowManager::¯ImbDataFlowManager()
  ImbDeployedResourceGroup::clearDataFlowList()
  ImbDeployedResourceGroup::clearDataFlowResources()
  ImbDataFlowApplication::clearApplicationContents()
  ImbDataFlowApplication::¯ImbDataFlowApplication()
  ImbExecutionGroupDirector::terminateInner()
  ImbPersistentResource::terminate()
  ImbMain::start(ImbStartParameters&)
  

Local fix

Problem summary

• ****************************************************************
  USERS AFFECTED:
  All users of IBM Integration Bus on z/OS
  
  
  Platforms affected:
  z/OS
  
  ****************************************************************
  PROBLEM DESCRIPTION:
  If RRS is terminated, the Integration Node will lose its
  connection to its queue manager and logs the following messages
  stating it is unable to set a context with its queue manager:
  
  +BIP2121E (Msg 1/2) <thd_id> THE THREAD BOOTSTRAP CODE CAUGHT AN
  UNHANDLED EXCEPTION ON THREAD NUMBER '12345678910' WITH NAME ''.
  +BIP9283S (Msg 2/2) <thd_id> UNABLE TO SET CONTEXT WITH MQ ON
  QUEUE MANAGER '': MQCC=2; MQRC=2195.
  
  The node will then shut down its integration servers whilst it
  attempts to re-establish a connection with the queue manager.
  The servers will attempt to stop their message flows before
  shutting down. However, it is possible that if a flow has not
  completed a clean shut down, the server will take an extended
  period of time to terminate and will appear to be hung, all the
  while logging multiple iterations of the BIP2111E/BIP2242E
  messages as below:
  
  +BIP2111E (Msg 1/2) 0 INTEGRATION NODE INTERNAL ERROR:
  DIAGNOSTIC INFORMATION
  '&#126;ImbDataFlowManager;lockForUpdate;delay'.
  +BIP2242E (Msg 2/2) 0 MESSAGE FLOW '<Message_Flow>' COULD NOT BE
  RECONFIGURED BECAUSE IT REMAINED BUSY FOR LONGER THAN THE
  TIMEOUT PERIOD OF '300000'
  
  Eventually the the Integration Server will be terminated and an
  abend with the following stack will be generated:
  
  ImbDumpAndTerminate(int,const void*)
  ImbAbend::terminateProcessInternal(const void*,bool)
  ImbAbend::logSystemErrorAndTerminateProcessInternal(const ch
  ImbDataFlowManager::¯ImbDataFlowManager()
  ImbDeployedResourceGroup::clearDataFlowList()
  ImbDeployedResourceGroup::clearDataFlowResources()
  ImbDataFlowApplication::clearApplicationContents()
  ImbDataFlowApplication::¯ImbDataFlowApplication()
  ImbExecutionGroupDirector::terminateInner()
  ImbPersistentResource::terminate()
  ImbMain::start(ImbStartParameters&)
  

Problem conclusion

• The product has been modified to resolve the delay terminating
  the Integration Server component.
  
  ---------------------------------------------------------------
  The fix is targeted for delivery in the following PTFs:
  
  Version    Maintenance Level
  v10.1      10.1.0.4
  
  The latest available maintenance can be obtained from:
  http://www-01.ibm.com/support/docview.wss?rs=849&uid=swg27006041
  
  If the maintenance level is not yet available,information on
  its planned availability can be found on:
  http://www-1.ibm.com/support/docview.wss?rs=849&uid=swg27006308
  ---------------------------------------------------------------
  

Temporary fix

Comments

APAR Information

• APAR is sysrouted FROM one or more of the following:

• APAR is sysrouted TO one or more of the following:

Fix information

• Fixed component name

  INT BUS FOR Z/O

• Fixed component ID

  5655AB110

Applicable component levels