IBM Support

Resolving JMSException due to com.ibm.mq.MQException: MQJE001: Completion Code 2, Reason 2009

Troubleshooting


Problem

The IBM WebSphere MQ Reason Code 2009 (MQRC_CONNECTION_BROKEN) may occur when an application tries to connect to a WebSphere MQ queue manager. Often this occurs when the Application Server tries to use an MQ connection is QCF pool.

Symptom

Here are some examples of errors that are caused by Reason Code 2009:

The following exception was logged javax.jms.JMSException:
MQJMS2008: failed to open MQ queue
com.ibm.mq.MQException: MQJE001: Completion Code 2, Reason 2009

javax.jms.JMSException: MQJMS2005: failed to create MQQueueManager for 'mynode:WAS_mynode_server1'
at com.ibm.mq.jms.services.ConfigEnvironment.newException(ConfigEnvironment.java:556)
at com.ibm.mq.jms.MQConnection.createQM(MQConnection.java:1736)
...
com.ibm.mq.MQException: MQJE001: An MQException occurred: Completion Code 2, Reason 2009
MQJE003: IO error transmitting message buffer
at com.ibm.mq.MQManagedConnectionJ11.<init>(MQManagedConnectionJ11.java:239)
...

WMSG0019E: Unable to start MDB Listener MyMessageDrivenBean, JMSDestination
jms/MyQueue : javax.jms.JMSException: MQJMS2005: failed to create
MQQueueManager for 'mynode:WAS_mynode_server1'
at com.ibm.mq.jms.services.ConfigEnvironment.newException(ConfigEnvironment.java:556)
at com.ibm.mq.jms.MQConnection.createQM(MQConnection.java:1736)

...

Cause

There are two possible scenarios. One is allocating a new QCF connection. The other is when the Application Server allocates a free connection from its connection pool, but the connection is no longer active, that is, broken. The connection may be broken for a number of different reasons; the 2009 reason code indicates that something prevented a successful connection to the Queue Manager. The most common causes for 2009 are the following:

1. A firewall has terminated the connection.
2. An IOException caused the socket to be closed.
3. An explicit action caused the socket to be closed by one end.
4. The queue manager is offline.
5. The maximum number of channels allowed by the queue manager are already open.
6. A configuration problem in the Queue Connection Factory (QCF).

Resolving The Problem


Preventing the firewall from terminating connections
Configure the Connection Pool and Session Pool settings for the QCF that is configured in WebSphere Application Server so that WebSphere can remove connections from the pool before they are dropped by the firewall. Change the value of Min Connections to 0 and set the Unused Timeout to half the number of seconds as the firewall timeout. For example, if the firewall times out connections after 15 minutes (900 seconds), set the Unused Timeout to 450 seconds.

Configuring to minimize the possibility of an IOException
On a UNIX® system, configure the TCP stanza of the qm.ini for your queue manager to contain this entry:
KeepAlive=YES
This setting causes TCP/IP to check periodically that the other end of the connection is still available. If it is not, the channel is closed.

Also follow the instructions in Tuning operating systems in the WebSphere Application Server Info Center. These will have you set the operating system configuration for TCP/IP to try to prevent sockets that are in use from being closed unexpectedly. For example, on Solaris, you will set the TCP_KEEPALIVE_INTERVAL setting on the WebSphere MQ machine. This should be set to be less than the firewall timeout value. If you do not set the TCP_KEEPALIVE_INTERVAL to be lower than the firewall timeout, then the keepalive packets will not be frequent enough to keep the connection open between WebSphere Application Server and MQ.

NOTE: You must be sure that the firewall is configured to allow keepalive packets to pass through. A connection broken error could be caused by the firewall not letting the keepalive packets through.

An explicit action can cause this
An action such as stopping the queue manager or restarting the queue manager would also cause Reason Code 2009. There have been some MQ defects that could result in unexpected 2009 errors. Go to the MQ support page to see if there are any known APARs that apply to your environment that have this Reason Code as a symptom.

The maximum number of channels has been reached
This could be due to the number of channels for the JMS provider not being large enough, or there could be some errors occurring that are causing channels to not close, so that they cannot be reused. For additional information, refer to these technotes, MQ Manager Stops Responding To JMS Requests. Also, WebSphere Application Server and MQ do not agree on the number of JMS connections.

A QCF Configuration problem
This problem could also occur because of a QCF configuration problem. If the Queue Manager, Host, Port, and Channel properties are not set correctly (for example, the MQ Channel name is case-sensitive), then a Reason Code 2009 would occur when an application uses the QCF to try to connect to the queue manager.

Other best practices

  1. Set the Purge Policy of the QCF Connection Pool and Session Pool to EntirePool. The default value is FailingConnectionOnly. When the Purge Policy is set to EntirePool, the WebSphere connection pool manager will flush the entire connection pool when a fatal connection error, such as Reason Code 2009, occurs. This will prevent the application from getting other bad connections from the pool.
  2. If the Reason Code 2009 error occurs when a message-driven bean (MDB) tries to connect to the queue manager, configure the MAX.RECOVERY.RETRIES and RECOVERY.RETRY.INTERVAL properties so that the message listener service will retry the connection. See Message listener service custom properties for more information on these properties.
  3. If you are not using an MDB, but the Reason Code 2009 error occurs for an application that sends messages to a queue, the application should have logic to retry the connection when the error occurs. See, Designing an enterprise application to use JMS, for information on how to program your application to use a JMS connection. Also, see Messaging troubleshooting tips for additional details.

Internal Use Only

Refer to these document for info on enabling traces to diagnose the problem.
http://l3.hursley.ibm.com/cgi-bin/ViewPRB.pl?trace
http://l3.hursley.ibm.com/cgi-bin/ViewPRB.pl?1410

Here are links to potential fixes
the IC42636 bundle from http://l3.hursley.ibm.com/cgi-bin/ViewPRB.pl?standard/2053.PRB might be the right patch.

[{"Product":{"code":"SSEQTP","label":"WebSphere Application Server"},"Business Unit":{"code":"BU004","label":"Hybrid Cloud"},"Component":"Java Message Service (JMS)","Platform":[{"code":"PF002","label":"AIX"},{"code":"PF010","label":"HP-UX"},{"code":"PF012","label":"IBM i"},{"code":"PF016","label":"Linux"},{"code":"PF027","label":"Solaris"},{"code":"PF033","label":"Windows"}],"Version":"9.0;8.5.5;8.0;7.0","Edition":"Base;Network Deployment"},{"Product":{"code":"SSNVBF","label":"Runtimes for Java Technology"},"Business Unit":{"code":"BU004","label":"Hybrid Cloud"},"Component":"Java SDK","Platform":[{"code":"","label":""}],"Version":"","Edition":""}]

Document Information

Modified date:
15 June 2018

UID

swg21226703