APAR status
Closed as program error.
Error description
When using the WebSphere Application Server 8.5.5 WebSphere MQ messaging provider, if an activation specification encounters an issue while either connecting or reconnecting to an IBM MQ queue manager, then the WebSphere MQ resource adapter (the component of the application server that handles all of the communication with IBM MQ) will write an error message to the application server's SystemOut.log file containing: - The top level exception that was encountered during the connection or reconnection attempt, along with its associated Java stack trace. - Details of all of the linked exceptions associated with the top level exception, and their associated Java stack trace. An example of this is shown below: CWSJY0003W: MQJCA4023: Startup reconnection failed for ActivationSpec 'javax.jms.Queue:jms/ivt/IVTQueue@QM1 <-528266751>'. Exception details: ' Message : com.ibm.msg.client.jms.DetailedIllegalStateException: JMSWMQ0018: Failed to connect to queue manager 'QM1' with connection mode 'Client' and host name 'localhost(1414)'. Check the queue manager is started and if running in client mode, check there is a listener running. Please see the linked exception for more information. Class : class com.ibm.msg.client.jms.DetailedIllegalStateException Stack : com.ibm.msg.client.wmq.common.internal.Reason.reasonToException( Reason.java:474) <Java Stack Trace> Caused by [1] --> Message : com.ibm.mq.MQException: JMSCMQ0001: WebSphere MQ call failed with compcode '2' ('MQCC_FAILED') reason '2059' ('MQRC_Q_MGR_NOT_AVAILABLE'). Class : class com.ibm.mq.MQException Stack : com.ibm.msg.client.wmq.common.internal.Reason.createException(Re ason.java:204) <Java Stack Trace> Caused by [2] --> Message : com.ibm.mq.jmqi.JmqiException: CC=2;RC=2059;AMQ9204: Connection to host 'localhost(1414)' rejected. [1=com.ibm.mq.jmqi.JmqiException[CC=2;RC=2059;AMQ9524: Remote queue manager unavailable. [3=SYSTEM.DEF.SVRCONN]],3=localhost(1414),5=RemoteConnection.ana lyseErrorSegment] Class : class com.ibm.mq.jmqi.JmqiException Stack : com.ibm.mq.jmqi.remote.api.RemoteFAP.jmqiConnect(RemoteFAP.java: 2099) <Java Stack Trace> Caused by [3] --> Message : com.ibm.mq.jmqi.JmqiException: CC=2;RC=2059;AMQ9524: Remote queue manager unavailable. [3=SYSTEM.DEF.SVRCONN] Class : class com.ibm.mq.jmqi.JmqiException Stack : com.ibm.mq.jmqi.remote.impl.RemoteConnection.analyseErrorSegment (RemoteConnection.java:3936) <Java Stack Trace> Caused by [4] --> Message : java.net.ConnectException: Connection refused (Connection refused) Class : class java.net.ConnectException Stack : java.net.PlainSocketImpl.socketConnect(PlainSocketImpl.java:-2) <Java Stack Trace> These messages will potentially consist of a large number of lines. If an activation specification tries to reconnect to the queue manager, and continuously fails, then the application server's SystemOut.log file will quickly fill up with these messages.
Local fix
Problem summary
**************************************************************** USERS AFFECTED: This issue affects users of: - The IBM MQ 9.1 resource adapter - The IBM MQ 9.2 resource adapter - The WebSphere Application Server V8.5.5 WebSphere MQ messaging provider - The WebSphere Application Server V9 traditional IBM MQ messaging provider who have activation specifications that consume messages from an MQ destination, and are making use of the reconnection functionality provided the IBM MQ resource adapter. Platforms affected: MultiPlatform **************************************************************** PROBLEM DESCRIPTION: WebSphere Application Server 8.5.5 embeds the WebSphere MQ 7.1 resource adapter, which is used to handle all of the communication between the WebSphere Application Server WebSphere MQ messaging provider and an IBM MQ queue manager. When using this version of the application server, if an activation specification had been configured to consume messages from a JMS destination hosed on an IBM MQ queue manager, and that activation specification: - Either failed to connect to the queue manager when it started up. - Or became disconnected from the queue manager while it was running. then the WebSphere MQ resource adapter would wait for a period of time before trying to connect to the queue manager again. If the reconnection attempt failed, then the WebSphere MQ resource adapter would write a message, similar to the example shown below, to the application server's SystemOut.log file: CWSJY0003W: MQJCA4023: Startup reconnection failed for ActivationSpec '<activation specification>'. Exception details: ' Message : com.ibm.msg.client.jms.DetailedIllegalStateException: JMSWMQ0018: Failed to connect to queue manager '<queue manager>' with connection mode '<transport>' and host name '<hostname>(<port>)'. Check the queue manager is started and if running in client mode, check there is a listener running. Please see the linked exception for more information. Class : class com.ibm.msg.client.jms.DetailedIllegalStateException Stack : com.ibm.msg.client.wmq.common.internal.Reason.reasonToException( Reason.java:474) <Java Stack Trace> Caused by [1] --> Message : com.ibm.mq.MQException: JMSCMQ0001: WebSphere MQ call failed with compcode '2' ('MQCC_FAILED') reason '2059' ('MQRC_Q_MGR_NOT_AVAILABLE'). Class : class com.ibm.mq.MQException Stack : com.ibm.msg.client.wmq.common.internal.Reason.createException(Re ason.java:204) <Java Stack Trace> Caused by [2] --> Message : com.ibm.mq.jmqi.JmqiException: CC=2;RC=2059;AMQ9204: Connection to host '<hostname>(<port>)' rejected. [1=com.ibm.mq.jmqi.JmqiException[CC=2;RC=2059;AMQ9524: Remote queue manager unavailable. [3=<channel>],3=<hostname>(<port>),5=RemoteConnection.analyseErr orSegment] Class : class com.ibm.mq.jmqi.JmqiException Stack : com.ibm.mq.jmqi.remote.api.RemoteFAP.jmqiConnect(RemoteFAP.java: 2099) <Java Stack Trace> Caused by [3] --> Message : com.ibm.mq.jmqi.JmqiException: CC=2;RC=2059;AMQ9524: Remote queue manager unavailable. [3=<channel>] Class : class com.ibm.mq.jmqi.JmqiException Stack : com.ibm.mq.jmqi.remote.impl.RemoteConnection.analyseErrorSegment (RemoteConnection.java:3936) <Java Stack Trace> Caused by [4] --> Message : java.net.ConnectException: Connection refused (Connection refused) Class : class java.net.ConnectException Stack : java.net.PlainSocketImpl.socketConnect(PlainSocketImpl.java:-2) <Java Stack Trace> It would then wait for a period of time before trying to reconnect. If this subsequent attempt failed, then the resource adapter would write another message to the application server's SystemOut.log file before waiting and trying to reconnect again. Depending on the reason for the failure, the MQJCA4023 messages written to the SystemOut.log file could be in excess of 100 lines long. If this occurred, then the SystemOut.log file would potentially fill up quite quickly and wrap, losing useful information. In addition to this, the size of the MQJCA4023 messages would make it harder to spot smaller messages in the SystemOut.log file reporting other errors and warnings. In later versions of the IBM MQ resource adapter (including the IBM MQ 9 resource adapter embedded within WebSphere Application Server V9 traditional), the MQJCA4023 message was removed and a new MQJCA4029 message added indicating when an activation specification was being reconnected. An example of this message is shown below: MQJCA4029: Activation specification reconnection attempt '<number>' of '<number>'. Waiting '<time>' milliseconds before attempting to reconnect the endpoint '<activation specification>' [This was done as part of the work for APAR IT25910]. However, the new MQJCA4029 error did not provide any indication of why the activation specification had either been disconnected or had failed to connect to a queue manager.
Problem conclusion
Two changes have been made to resolve this issue: Change 1: ------------------------ The WebSphere MQ 7.1 resource adapter, used by the WebSphere Application Server V8.5.5 WebSphere MQ messaging provider, has been updated so it will write: - A new MQJCA4037 message to the application server's SystemOut.log file when an activation specification fails to reconnect to a queue manager, which contains details of the linked exception. - Followed by a new MQJCA4029 message indicating that the WebSphere MQ resource adapter will pause for a period of time before trying to reconnect the activation specification. If the WebSphere MQ resource adapter subsequently tries to reconnect the activation specification, and the reconnection attempt fails with the same underlying exception as before, then it will just write another MQJCA4029 message to the application server's SystemOut.log file before pausing again. However, if a subsequent reconnection attempt fails for a different reason to the previous one, the WebSphere MQ resource adapter will write a new MQJCA4037 message to the application server's SystemOut.log file containing details of the failure, followed by an MQJCA4029 message. After the MQJCA4029 message has been written, the WebSphere MQ resource adapter will pause before trying to reconnect again. An example of these messages are shown below: MQJCA4037: Reconnection failed for ActivationSpec '<activation specification>'. Exception details: ' Message : com.ibm.msg.client.jms.DetailedIllegalStateException: JMSWMQ0018: Failed to connect to queue manager '<queue manager>' with connection mode '<transport>' and host name '<hostname>(<port>)'. Check the queue manager is started and if running in client mode, check there is a listener running. Please see the linked exception for more information. Class : class com.ibm.msg.client.jms.DetailedIllegalStateException Stack : com.ibm.msg.client.wmq.common.internal.Reason.reasonToException( Reason.java:474) <Java Stack Trace> Caused by [1] --> Message : com.ibm.mq.MQException: JMSCMQ0001: WebSphere MQ call failed with compcode '2' ('MQCC_FAILED') reason '2059' ('MQRC_Q_MGR_NOT_AVAILABLE'). Class : class com.ibm.mq.MQException Stack : com.ibm.msg.client.wmq.common.internal.Reason.createException(Re ason.java:204) <Java Stack Trace> Caused by [2] --> Message : com.ibm.mq.jmqi.JmqiException: CC=2;RC=2059;AMQ9204: Connection to host '<hostname>(<port>)' rejected. [1=com.ibm.mq.jmqi.JmqiException[CC=2;RC=2059;AMQ9524: Remote queue manager unavailable. [3=<channel>],3=<hostname>(<port>),5=RemoteConnection.analyseErr orSegment] Class : class com.ibm.mq.jmqi.JmqiException Stack : com.ibm.mq.jmqi.remote.api.RemoteFAP.jmqiConnect(RemoteFAP.java: 2099) <Java Stack Trace> Caused by [3] --> Message : com.ibm.mq.jmqi.JmqiException: CC=2;RC=2059;AMQ9524: Remote queue manager unavailable. [3=<channel>] Class : class com.ibm.mq.jmqi.JmqiException Stack : com.ibm.mq.jmqi.remote.impl.RemoteConnection.analyseErrorSegment (RemoteConnection.java:3936) <Java Stack Trace> Caused by [4] --> Message : java.net.ConnectException: Connection refused (Connection refused) Class : class java.net.ConnectException Stack : java.net.PlainSocketImpl.socketConnect(PlainSocketImpl.java:-2) <Java Stack Trace>'. CWSJY0003W: MQJCA4029: Activation specification reconnection attempt '<number>' of '<number>'. Waiting '<timeout>' milliseconds before attempting to reconnect the endpoint '<activation specification>'. Change 2: ------------------------ The IBM MQ 9.1 and 9.2 resource adapter have been updated so that they will log a new MQJCA4037 message if an activation specification failed to reconnect to a queue manager after it had been disconnected. If subsequent reconnection attempts fail with the same underlying exception as before, then the IBM MQ resource adapters will not write any additional MQJCA4037 messages to the application server's log file. They will just write an MQJCA4029 message to the log file, indicating that the IBM MQ resource adapter will pause before trying to reconnect the activation specification again. However, if a reconnection attempt fails due to a different exception, then a new MQJCA4037 message will be written to the log file, providing details of why the reconnection attempt failed, followed by an MQJCA4029 message. An example of the new MQJCA4037 message is shown below: MQJCA4037: Reconnection failed for ActivationSpec '<activation specification>'. Exception details: ' Message : com.ibm.msg.client.jms.DetailedIllegalStateException: JMSWMQ0018: Failed to connect to queue manager '<queue manager>' with connection mode '<transport>' and host name '<hostname>(<port>)'. Check the queue manager is started and if running in client mode, check there is a listener running. Please see the linked exception for more information. Class : class com.ibm.msg.client.jms.DetailedIllegalStateException Stack : com.ibm.msg.client.wmq.common.internal.Reason.reasonToException( Reason.java:474) <Java Stack Trace> Caused by [1] --> Message : com.ibm.mq.MQException: JMSCMQ0001: WebSphere MQ call failed with compcode '2' ('MQCC_FAILED') reason '2059' ('MQRC_Q_MGR_NOT_AVAILABLE'). Class : class com.ibm.mq.MQException Stack : com.ibm.msg.client.wmq.common.internal.Reason.createException(Re ason.java:204) <Java Stack Trace> Caused by [2] --> Message : com.ibm.mq.jmqi.JmqiException: CC=2;RC=2059;AMQ9204: Connection to host '<hostname>(<port>)' rejected. [1=com.ibm.mq.jmqi.JmqiException[CC=2;RC=2059;AMQ9524: Remote queue manager unavailable. [3=<channel>],3=<hostname>(<port>),5=RemoteConnection.analyseErr orSegment] Class : class com.ibm.mq.jmqi.JmqiException Stack : com.ibm.mq.jmqi.remote.api.RemoteFAP.jmqiConnect(RemoteFAP.java: 2099) <Java Stack Trace> Caused by [3] --> Message : com.ibm.mq.jmqi.JmqiException: CC=2;RC=2059;AMQ9524: Remote queue manager unavailable. [3=<channel>] Class : class com.ibm.mq.jmqi.JmqiException Stack : com.ibm.mq.jmqi.remote.impl.RemoteConnection.analyseErrorSegment (RemoteConnection.java:3936) <Java Stack Trace> Caused by [4] --> Message : java.net.ConnectException: Connection refused (Connection refused) Class : class java.net.ConnectException Stack : java.net.PlainSocketImpl.socketConnect(PlainSocketImpl.java:-2) <Java Stack Trace>'. Caused by: 'CC=2;RC=2538;AMQ9213: A communications error for 'TCP' occurred. [1=java.net.ConnectException[Connection refused (Connection refused)],3=connnectUsingLocalAddress,4=TCP,5=Socket.connect]'. Caused by: 'Connection refused (Connection refused)'. '. --------------------------------------------------------------- The fix is targeted for delivery in the following PTFs: Version Maintenance Level v9.1 LTS 9.1.0.12 v9.2 LTS 9.2.0.7 v9.3 LTS 9.3.0.1 v9.x CD 9.3.1 The latest available maintenance can be obtained from 'WebSphere MQ Recommended Fixes' http://www-1.ibm.com/support/docview.wss?rs=171&uid=swg27006037 If the maintenance level is not yet available information on its planned availability can be found in 'WebSphere MQ Planned Maintenance Release Dates' http://www-1.ibm.com/support/docview.wss?rs=171&uid=swg27006309 ---------------------------------------------------------------
Temporary fix
Comments
APAR Information
APAR number
IT39919
Reported component name
MQ WINDOWS V7
Reported component ID
5724H7220
Reported release
710
Status
CLOSED PER
PE
NoPE
HIPER
NoHIPER
Special Attention
NoSpecatt / Xsystem
Submitted date
2022-02-09
Closed date
2022-06-07
Last modified date
2022-09-08
APAR is sysrouted FROM one or more of the following:
APAR is sysrouted TO one or more of the following:
Fix information
Fixed component name
MQ WINDOWS V7
Fixed component ID
5724H7220
Applicable component levels
[{"Business Unit":{"code":"BU059","label":"IBM Software w\/o TPS"},"Product":{"code":"SSFKSJ","label":"WebSphere MQ"},"Platform":[{"code":"PF025","label":"Platform Independent"}],"Version":"7.1","Line of Business":{"code":"LOB45","label":"Automation"}}]
Document Information
Modified date:
08 September 2022