IT31603: A DEADLOCK CAN OCCUR IN AN INTEGRATION SERVER WHEN STOPPING AN APPLICATION AS PART OF AN ADMINISTRATION ACTION

APAR status

• Closed as program error.

Error description

• A timing window while processing an application shutdown request
  can cause a deadlock wit the following stack:
  
  #0 0x00007fbdfe659cf2 in pthread_cond_timedwait@@GLIBC_2.3.2 ()
  from /usr/lib64/libpthread.so.0
  #1 0x00007fbe025a307f in ImbSemaphore::wait(long) () from
  /opt/EAsmbl_image/ace-11.0.0.3/server/lib/libCommonServices.so
  #2 0x00007fbe025bdfb3 in ImbThreadPool::~ImbThreadPool() ()
  from
  /opt/EAsmbl_image/ace-11.0.0.3/server/lib/libCommonServices.so
  #3 0x00007fbe01850a37 in ImbDataFlowManager::onStop(bool) ()
  from
  /opt/EAsmbl_image/ace-11.0.0.3/server/lib/libMessageServices.so
  #4 0x00007fbe01a558dd in
  ImbResource::stateTransition(ImbResource::ObjectState,
  ImbConfigObject*, bool) () from /opt/EAsmbl_image/ace-11.0.0.3/s
  erver/lib/libMessageServices.so
  #5 0x00007fbe01a56b9f in ImbResource::stop(bool,
  ImbResource::FailureAction) () from
  /opt/EAsmbl_image/ace-11.0.0.3/server/lib/libMessageServices.so
  #6 0x00007fbe0187e49d in
  ImbDeployedResourceGroup::stopFlows(bool) () from
  /opt/EAsmbl_image/ace-11.0.0.3/server/lib/libMessageServices.so
  #7 0x00007fbe018a35aa in ImbDataFlowApplication::onStop(bool) ()
  from
  /opt/EAsmbl_image/ace-11.0.0.3/server/lib/libMessageServices.so
  #8 0x00007fbe01a558dd in
  ImbResource::stateTransition(ImbResource::ObjectState,
  ImbConfigObject*, bool) () from
  /opt/EAsmbl_image/ace-11.0.0.3/server/lib/libMessageServices.so
  
  This occurs because the flow thread pool is waiting for all the
  message flow threads to terminate, but the termination request
  has not been fully sent and acknowledged by all the message flow
  threads. Some message flows in the deployed applications may
  continue to process messages while in this hung state.
  

Local fix

• The Integration Server must be manually restarted.
  

Problem summary

• ****************************************************************
  USERS AFFECTED:
  All users of IBM App Connect Enterprise 11.0.0.4 and earlier.
  
  
  Platforms affected:
  MultiPlatform
  
  ****************************************************************
  PROBLEM DESCRIPTION:
  A timing window while processing an application shutdown request
  can cause a deadlock wit the following stack:
  
  #0 0x00007fbdfe659cf2 in pthread_cond_timedwait@@GLIBC_2.3.2 ()
  from /usr/lib64/libpthread.so.0
  #1 0x00007fbe025a307f in ImbSemaphore::wait(long) () from
  /opt/EAsmbl_image/ace-11.0.0.3/server/lib/libCommonServices.so
  #2 0x00007fbe025bdfb3 in ImbThreadPool::~ImbThreadPool() ()
  from
  /opt/EAsmbl_image/ace-11.0.0.3/server/lib/libCommonServices.so
  #3 0x00007fbe01850a37 in ImbDataFlowManager::onStop(bool) ()
  from
  /opt/EAsmbl_image/ace-11.0.0.3/server/lib/libMessageServices.so
  #4 0x00007fbe01a558dd in
  ImbResource::stateTransition(ImbResource::ObjectState,
  ImbConfigObject*, bool) () from /opt/EAsmbl_image/ace-11.0.0.3/s
  erver/lib/libMessageServices.so
  #5 0x00007fbe01a56b9f in ImbResource::stop(bool,
  ImbResource::FailureAction) () from
  /opt/EAsmbl_image/ace-11.0.0.3/server/lib/libMessageServices.so
  #6 0x00007fbe0187e49d in
  ImbDeployedResourceGroup::stopFlows(bool) () from
  /opt/EAsmbl_image/ace-11.0.0.3/server/lib/libMessageServices.so
  #7 0x00007fbe018a35aa in ImbDataFlowApplication::onStop(bool) ()
  from
  /opt/EAsmbl_image/ace-11.0.0.3/server/lib/libMessageServices.so
  #8 0x00007fbe01a558dd in
  ImbResource::stateTransition(ImbResource::ObjectState,
  ImbConfigObject*, bool) () from
  /opt/EAsmbl_image/ace-11.0.0.3/server/lib/libMessageServices.so
  
  This occurs because the flow thread pool is waiting for all the
  message flow threads to terminate, but the termination request
  has not been fully sent and acknowledged by all the message flow
  threads. Some message flows in the deployed applications may
  continue to process messages while in this hung state.
  

Problem conclusion

• The product will no longer hang on application shutdown if a
  message flow thread does not terminate quickly.
  
  ---------------------------------------------------------------
  The fix is targeted for delivery in the following PTFs:
  
  Version    Maintenance Level
  v11.0      11.0.0.5
  
  The latest available maintenance can be obtained from:
  http://www-01.ibm.com/support/docview.wss?rs=849&uid=swg27006041
  
  If the maintenance level is not yet available,information on
  its planned availability can be found on:
  http://www-1.ibm.com/support/docview.wss?rs=849&uid=swg27006308
  ---------------------------------------------------------------
  

Temporary fix

Comments

APAR Information

• APAR is sysrouted FROM one or more of the following:

• APAR is sysrouted TO one or more of the following:

Fix information

• Fixed component name

  APP CONNECT ENT

• Fixed component ID

  5724J0550

Applicable component levels