APAR status
Closed as program error.
Error description
WebSphere Application Server 9.0.x and IBM MQ are installed on separate machines. An IBM MQ messaging provider activation specification has been defined within the application server, and is configured to connect to a queue manager using the BINDINGS_THEN_CLIENT transport. While the activation specification is processing messages, the application server is stopped unexpectedly. When the application server restarts, it tries to recover the XA transactions that the activation specification was involved in at the time that the application server stopped. However, the transaction recovery fails and the following message is written to the application server's SystemOut.log file: WTRN0141I: Recovered transaction (tid=2044193076) commit of xid 0000017A3E7525A40000000179D3A864F0C16576CBF488D9DEBDBED0AFA31020 65E959CB:0000017A3E7525A40000000179D3A864F0C16576CBF488D9DEBDBED 0AFA3102065E959CB000000010000000000000000000000000001 with XAResource: cells/LAPTOP-R910IGN3Node18Cell/resources.xml#J2CResourceAdapter _1612870301746 resulted in XAER_RMFAIL
Local fix
Update the transport of MQ connection factory or Activation Spec to 'Client'.
Problem summary
**************************************************************** USERS AFFECTED: This issue affects users of the WebSphere Application Server 9.0.x IBM MQ messaging provider, who: - Have the application server installed on a different machine to an IBM MQ queue manager. - And have activation specifications defined within the application server that are configured to connect to the queue manager using the default transport type of BINDINGS_THEN_CLIENT. Platforms affected: MultiPlatform **************************************************************** PROBLEM DESCRIPTION: When creating an IBM MQ messaging provider activation specification in WebSphere Application Server, the default transport mode is BINDINGS_THEN_CLIENT. When an activation specification which is configured to use this transport mode is started, then the MQ resource adapter (the component of WebSphere Application Server that handles all of the communication with IBM MQ queue managers) will initially try to connect to the specified queue manager using the BINDINGS transport. If the connection attempt fails, then the MQ resource adapter will try to connect to it using the CLIENT transport. If that attempt also fails, then the MQ resource adapter will throw a JMSException back to the application server and the activation specification will not start. Now, if a message-driven bean (MDB) application is: - Deployed into WebSphere Application Server. - And is configured to use an IBM MQ messaging provider activation specification to consume messages from a queue hosted on a queue manager. - And is also configured to use XA transactions. then whenever a transaction is started, the application server will store a reference to the activation specification to its transaction logs. If the XA transaction needs to be recovered for some reason, then the application server will use the details of the activation specification stored in its transaction logs to reconnect to the queue manager and then recover the transaction. When the issue reported in this APAR occurred, the application server had stopped unexpectedly: - After an xa_commit() had been flowed to the queue manager to complete an XA transaction involving an activation specification that was configured to use the BINDINGS_THEN_CLIENT transport. - And before the application server had forced a "deletion record" to its transaction logs. When the application server restarted, the following steps occurred: - The application server checked its transaction logs and found an entry for the XA transaction. - It then called into the MQ resource adapter, asking it to use the activation specification definition associated with the XA transaction to connect to the queue manager and retry the xa_commit() operation. - Initially, the MQ resource adapter tried to connect to the queue manager using the BINDINGS transport. This attempt failed with the following JMSException: JMSFMQ6312: An exception occurred in the Java(tm) MQI. The Java(tm) MQI has thrown an exception describing the problem. See the linked exception for further information. ... Caused by: com.ibm.mq.jmqi.JmqiException: CC=2;RC=2495;AMQ8568: The native JNI library 'mqjbnd64' was not found. For a client installation this is expected. [3=mqjbnd64] ... Caused by: java.lang.UnsatisfiedLinkError: mqjbnd64 (Not found in java.library.path) 	at java.lang.ClassLoader.loadLibraryWithPath(ClassLoader.java:1459) 	at java.lang.ClassLoader.loadLibraryWithClassLoader(ClassLoader.jav a:1410) 	at java.lang.System.loadLibrary(System.java:589) 	at com.ibm.mq.jmqi.local.LocalMQ.loadLib(LocalMQ.java:1112) 	... 31 more which the MQ resource adapter stored in an internal list. - Next, the MQ resource adapter tried to connect to the queue manager using the CLIENT transport. This attempt was successful. - The MQ resource adapter then flowed an xa_commit() call to the queue manager to complete the transaction. - Because the queue manager had already committed the transaction, it returned XA error code XAER_NOTA (-4) back to the MQ resource adapter. - When the MQ resource adapter received this XA error code from the queue manager, it created the following internal JMSExcetion: [com.ibm.msg.client.jms.DetailedJMSException: JMSFMQ6312: An exception occurred in the Java(tm) MQI. The Java(tm) MQI has thrown an exception describing the problem. See the linked exception for further information., javax.transaction.xa.XAException: The method 'xa_commit' has failed with errorCode '-4'.] and added it to the internal list. - The MQ resource adapter now processed the exceptions in the list, to determine which one to return back to the application server. - The correct exception to return was the second one, as it indicated that the queue manager had already completed all of the work for the transaction that was being recovered. However, the MQ resource adapter only ever checked the first exception in the list. Because this exception represented a failure to connect to the queue manager, the MQ resource adapter constructed an XAException containing: - The XA error code -7 (XAER_RMFAIL). - And the first exception as a linked exception. This exception was then returned back to the application server. - When the application server received the exception, it determined that the transaction recovery had failed. This caused it to write the message: WTRN0141I: Recovered transaction (tid=<identifier>) commit of xid <transaction identifier>with XAResource:<resource identiifer> resulted in XAER_RMFAIL to it's SystemOut.log file, and then wait for a period of time before trying to recover the transaction again. Whenever the application server tried to recover the transaction again, the same sequence of events occurred. As a result, the transaction could never be recovered, and the entry for it was never removed from the application server's transaction logs.
Problem conclusion
In order to resolve this issue, the MQ resource adapter has been updated to check both exceptions in the list when determining what to return back to the application server when recovering a transaction associated with an activation specification that has been configured to use the BINDINGS_THEN_CLIENT transport, rather than just the first one. This ensures that the correct exception is passed back the application server for the current transaction recovery scenario, which in turn allows the recovery to continue as expected. --------------------------------------------------------------- The fix is targeted for delivery in the following PTFs: Version Maintenance Level v9.1 LTS 9.1.0.11 v9.2 LTS 9.2.0.5 v9.x CD 9.2.5 The latest available maintenance can be obtained from 'WebSphere MQ Recommended Fixes' http://www-1.ibm.com/support/docview.wss?rs=171&uid=swg27006037 If the maintenance level is not yet available information on its planned availability can be found in 'WebSphere MQ Planned Maintenance Release Dates' http://www-1.ibm.com/support/docview.wss?rs=171&uid=swg27006309 ---------------------------------------------------------------
Temporary fix
Comments
APAR Information
APAR number
IT37502
Reported component name
IBM MQ BASE MP
Reported component ID
5724H7271
Reported release
910
Status
CLOSED PER
PE
NoPE
HIPER
NoHIPER
Special Attention
NoSpecatt / Xsystem
Submitted date
2021-07-04
Closed date
2021-11-08
Last modified date
2021-11-08
APAR is sysrouted FROM one or more of the following:
APAR is sysrouted TO one or more of the following:
Fix information
Fixed component name
IBM MQ BASE MP
Fixed component ID
5724H7271
Applicable component levels
[{"Line of Business":{"code":"LOB45","label":"Automation"},"Business Unit":{"code":"BU059","label":"IBM Software w\/o TPS"},"Product":{"code":"SSYHRD","label":"IBM MQ"},"Platform":[{"code":"PF025","label":"Platform Independent"}],"Version":"910"}]
Document Information
Modified date:
09 November 2021