IT09442: IBM MQ V8 JMS APP THAT CREATES/CLOSES MANY CONNECTIONS TO A QMGR CAUSES HIGH CPU, MEMORY UTILIZATION.

APAR status

• Closed as program error.

Error description

• When IBM MQ v8 JMS application creates/closes many
  connections to a queue manager, there is gradual CPU usage
  increase over time, and an increase in the JVM memory usage.
  This causes serious impact on the JVM and GC.
  
  Over 30 day period the system/CPU load (as reported by Linux)
  is increasing, from around 0.1-0.5 to 3-5%.
  
  Memory allocation increase, Every time full GC is performed,
  memory goes down, but memory allocation increases, and as a
  consequence, frequency of (full) GCs is increasing.
  
  Once the system is under increased load (even though it's
  idle really) performing a particular user action/request can
  push the system over the line and it then does not recover.
  

Local fix

Problem summary

• ****************************************************************
  USERS AFFECTED:
  This issue affects users of:
  
  - The IBM MQ V8 allclient JAR file.
  - The IBM MQ V8 classes for JMS.
  - The IBM MQ V8 classes for JMS OSGi bundle.
  - The IBM MQ V8 resource adapter.
  
  who have JMS applications that:
  
  - Connect to a queue manager using either IBM MQ messaging
  provider normal mode, or IBM MQ messaging provider normal mode
  with restrictions, using the CLIENT transport.
  - And connect using an IBM MQ channel instance that has the
  SHARECNV attribute set to a value greater than 1.
  - And create lots of short-lived JMS connections, JMS sessions
  or JMSContexts.
  
  
  Platforms affected:
  MultiPlatform
  
  ****************************************************************
  PROBLEM DESCRIPTION:
  When an IBM MQ classes for JMS application connects to a queue
  manager in either IBM MQ messaging provider normal mode, or IBM
  MQ messaging provider normal mode with restrictions, using the
  CLIENT transport, it can make use of the "conversation sharing"
  functionality provided by the queue manager. This allows
  multiple conversations between the application and the queue
  manager to use the same underlying channel instance (or TCP/IP
  connection, if you prefer to think of it in those terms). The
  maximum number of conversations that can take place over a
  single channel instance is specified by the IBM MQ channel
  property SHARECNV. The default value is 10.
  
  For JMS applications, every JMS Connection starts a conversation
  with the queue manager. Similarly, every JMS Session created by
  an application also starts a conversation with the queue
  manager. Each JMSContext starts two conversations with a queue
  manager.
  
  Every conversation that is started over a channel instance has a
  unique number (or identifier), which increments by 2 - the first
  conversation over a specified channel instance is given the
  number 1, then second conversation over the same channel
  instance has the number 3, and so on. These conversation
  identifiers are generated by the queue manager. If the WebSphere
  MQ classes for JMS are required to start a new conversation on a
  new channel instance, then that conversation will have the
  identifier 1.
  
  The IBM MQ classes for JMS maintain internal lists containing
  information about the conversations that are currently active
  over a channel instance. When a new conversation is started,
  details of that conversation are added to the internal list for
  the appropriate channel instance. When a conversation ends, the
  entry for that conversation is removed from the relevant list.
  
  These lists were defined as arrays, which was keyed off of the
  conversation identifier. When a new conversation was created on
  the channel instance, some information about that conversation
  was put into the array at a position specified by the identifier
  for that conversation.
  
  If an application created lots of short lived conversations over
  a single channel instance, then the following behaviour
  occurred:
  
  - A JMS Connection was created. This resulted in a new channel
  instance being created, as well as a new conversation being
  started. This conversation had identifier 1, and so went into
  position 1 in the array for the channel instance.
  - A short-lived JMS Session was created. A new conversation was
  started with the queue manager, which had conversation
  identifier 3. The array was resized to contain 3 elements, and
  details about this conversation went into position 3 in the
  array.
  - When the JMS Session was closed, the conversation was ended
  and details about it were removed from position 3 in the array.
  - A second short-lived JMS Session was created and a new
  conversation was started with the queue manager. This had
  conversation identifier 5. The array was resized to contain 5
  elements, and the details about the new conversation went into
  position 5 in the array.
  - When this JMS Session was closed, the conversation was ended
  and details about it were removed from position 5 in the array.
  
  If we move forward in time a bit now, the following behaviour
  was seen:
  
  - A short-lived JMS Session was created from the original JMS
  connection (which had conversation identifier 1). A new
  conversation was started with the queue manager, which had
  conversation identifier 505. The array was resized to contain
  505 elements, and information about this conversation went into
  position 505 in the array.
  - When the JMS Session was closed, the conversation was ended
  and details about it were removed from position 505 in the
  array.
  
  And moving forward a bit further in time:
  
  - A short-lived JMS Session was created from the original JMS
  connection (which had conversation identifier 1) and a new
  conversation was started with the queue manager. This had
  conversation identifier 5005. The array was resized to contain
  5005 elements, and details about this conversation went into
  position 5005 in the array.
  - When this JMS Session was closed, the conversation was ended
  and details about it were removed from position 5005 in the
  array.
  
  And again:
  
  - A short-lived JMS Session was created from the original JMS
  connection (which had conversation identifier 1). A new
  conversation was started with the queue manager, which had
  conversation identifier 50005. The array was resized to contain
  50005 elements, and information about the new conversation went
  added to the array in position 50005.
  - When the JMS Session was closed, the conversation was ended
  and details about it were removed from position 50005 in the
  array.
  
  This resulted in the Java Runtime Environment creating a bigger
  array each time a new conversation was started, even though only
  a few elements in the array were actually being used......
  Because of the size of the array, the memory usage of the Java
  Runtime Environment that the application was running in kept
  growing and shrinking, which resulted in high CPU usage as the
  Java Runtime Environment had to perform lots of garbage
  collection to free up enough memory to hold the array.
  

Problem conclusion

• The IBM MQ classes for JMS have been updated to use sets to
  store information about the conversations associated with the
  object, rather than an array. When a new conversation is
  started, an entry is added to the set. The entry will be removed
  (and the size of the set decremented) when the conversation
  finishes. This ensures that the size of the set will never be
  larger than the maximum number of conversations that can take
  place over the channel instance (as defined by the IBM MQ
  channel attribute SHARECNV).
  
  ---------------------------------------------------------------
  The fix is targeted for delivery in the following PTFs:
  
  Version    Maintenance Level
  v8.0       8.0.0.4
  
  The latest available maintenance can be obtained from
  'WebSphere MQ Recommended Fixes'
  http://www-1.ibm.com/support/docview.wss?rs=171&uid=swg27006037
  
  If the maintenance level is not yet available information on
  its planned availability can be found in 'WebSphere MQ
  Planned Maintenance Release Dates'
  http://www-1.ibm.com/support/docview.wss?rs=171&uid=swg27006309
  ---------------------------------------------------------------
  

Temporary fix

Comments

APAR Information

• APAR is sysrouted FROM one or more of the following:

• APAR is sysrouted TO one or more of the following:

Fix information

• Fixed component name

  WMQ BASE MULTIP

• Fixed component ID

  5724H7251

Applicable component levels

• R800 PSY

     UP