APAR status
Closed as program error.
Error description
The queue manager data filesystem on an AIX queue manager became unavailable to the operating system. During this time it was observed that an MQ classes for Java application threads running within the WebSphere Application Server v8.5 environment hung when attempting to connect to the queue manager. A Javacore revealed a small number of threads with the following stack: at java/net/SocketInputStream.socketRead0(Native Method) at java/net/SocketInputStream.read(SocketInputStream.java:165(Compi led Code)) at java/net/SocketInputStream.read(SocketInputStream.java:134(Compi led Code)) at com/ibm/mq/jmqi/remote/impl/RemoteTCPConnection.receive(RemoteTC PConnection.java:1673(Compiled Code)) at com/ibm/mq/jmqi/remote/impl/RemoteConnection.receiveTSH(RemoteCo nnection.java:2725(Compiled Code)) at com/ibm/mq/jmqi/remote/impl/RemoteConnection.initSess(RemoteConn ection.java:1049(Compiled Code)) at com/ibm/mq/jmqi/remote/impl/RemoteConnection.connect(RemoteConne ction.java:742(Compiled Code)) at com/ibm/mq/jmqi/remote/impl/RemoteConnectionSpecification.getSes sionFromNewConnection(RemoteConnectionSpecification.java:358(Com piled Code)) at com/ibm/mq/jmqi/remote/impl/RemoteConnectionSpecification.getSes sion(RemoteConnectionSpecification.java:267(Compiled Code)) at com/ibm/mq/jmqi/remote/impl/RemoteConnectionPool.getSession(Remo teConnectionPool.java:162(Compiled Code)) at com/ibm/mq/jmqi/remote/api/RemoteFAP.jmqiConnect(RemoteFAP.java: 1710(Compiled Code)) at com/ibm/mq/jmqi/remote/api/RemoteFAP.jmqiConnect(RemoteFAP.java: 1348(Compiled Code)) at com/ibm/mq/MQSESSION.MQCONNX_j(MQSESSION.java:924(Compiled Code)) at com/ibm/mq/MQManagedConnectionJ11.<init>(MQManagedConnectionJ11. java:221(Compiled Code)) at com/ibm/mq/MQClientManagedConnectionFactoryJ11._createManagedCon nection(MQClientManagedConnectionFactoryJ11.java:553(Compiled Code)) at com/ibm/mq/MQClientManagedConnectionFactoryJ11.createManagedConn ection(MQClientManagedConnectionFactoryJ11.java:593(Compiled Code)) at com/ibm/mq/StoredManagedConnection.<init>(StoredManagedConnectio n.java:96(Compiled Code)) at com/ibm/mq/MQSimpleConnectionManager.allocateConnection(MQSimple ConnectionManager.java:198(Compiled Code)) at com/ibm/mq/MQQueueManagerFactory.obtainBaseMQQueueManager(MQQueu eManagerFactory.java:893(Compiled Code)) at com/ibm/mq/MQQueueManagerFactory.procure(MQQueueManagerFactory.j ava:780(Compiled Code)) at com/ibm/mq/MQQueueManagerFactory.constructQueueManager(MQQueueMa nagerFactory.java:729(Compiled Code)) at com/ibm/mq/MQQueueManagerFactory.createQueueManager(MQQueueManag erFactory.java:177(Compiled Code)) at com/ibm/mq/MQQueueManager.<init>(MQQueueManager.java:753(Compile d Code)) at MyApplication.myMethod(...) These threads were not seen to be progressing over time. Many threads were also observed to be waiting on the above threads, with stacks of the form: at java/lang/Object.wait(Native Method) at java/lang/Object.wait(Object.java:167(Compiled Code)) at com/ibm/mq/jmqi/remote/impl/RemoteConnectionSpecification.getSes sionFromEligibleConnection(RemoteConnectionSpecification.java:41 0(Compiled Code)) (entered lock: com/ibm/mq/jmqi/remote/impl/RemoteConnectionSpecification$Connec tionsLock@0x00000007C59BC9D8, entry count: 1) at com/ibm/mq/jmqi/remote/impl/RemoteConnectionSpecification.getSes sion(RemoteConnectionSpecification.java:258(Compiled Code)) at com/ibm/mq/jmqi/remote/impl/RemoteConnectionPool.getSession(Remo teConnectionPool.java:162(Compiled Code)) at com/ibm/mq/jmqi/remote/api/RemoteFAP.jmqiConnect(RemoteFAP.java: 1710(Compiled Code)) at com/ibm/mq/jmqi/remote/api/RemoteFAP.jmqiConnect(RemoteFAP.java: 1348(Compiled Code)) at com/ibm/mq/MQSESSION.MQCONNX_j(MQSESSION.java:924(Compiled Code)) at com/ibm/mq/MQManagedConnectionJ11.<init>(MQManagedConnectionJ11. java:221(Compiled Code)) at com/ibm/mq/MQClientManagedConnectionFactoryJ11._createManagedCon nection(MQClientManagedConnectionFactoryJ11.java:553(Compiled Code)) at com/ibm/mq/MQClientManagedConnectionFactoryJ11.createManagedConn ection(MQClientManagedConnectionFactoryJ11.java:593(Compiled Code)) at com/ibm/mq/StoredManagedConnection.<init>(StoredManagedConnectio n.java:96(Compiled Code)) at com/ibm/mq/MQSimpleConnectionManager.allocateConnection(MQSimple ConnectionManager.java:198(Compiled Code)) at com/ibm/mq/MQQueueManagerFactory.obtainBaseMQQueueManager(MQQueu eManagerFactory.java:893(Compiled Code)) at com/ibm/mq/MQQueueManagerFactory.procure(MQQueueManagerFactory.j ava:780(Compiled Code)) at com/ibm/mq/MQQueueManagerFactory.constructQueueManager(MQQueueMa nagerFactory.java:729(Compiled Code)) at com/ibm/mq/MQQueueManagerFactory.createQueueManager(MQQueueManag erFactory.java:177(Compiled Code)) at com/ibm/mq/MQQueueManager.<init>(MQQueueManager.java:753(Compile d Code)) at MyApplication.myMethod(...)
Local fix
Problem summary
**************************************************************** USERS AFFECTED: Users of the following Java components: MQ classes for Java MQ classes for JMS MQ Resource Adapter MQ Managed File Transfer MQ Explorer who are connecting to the queue manager using a remote 'CLIENT' transport connection. Platforms affected: MultiPlatform **************************************************************** PROBLEM DESCRIPTION: When an application using the MQ classes for Java/JMS wants to establish a connection to a remote queue manager (using CLIENT transport), the connection request is broken down into several parts: (1) Establish a TCP/IP connection to the queue manager host system (2) Establish a communication protocol between the MQ classes for Java/JMS API with the queue manager. This will take into account a number of aspects, such as what features of the MQ protocol each side supports, including what heart beat interval to use on the TCP/IP socket. (3) Establish the logical connection, including authentication/authorisation of the user connecting. At these various stages there is scope for indefinite waits to occur in the client application when there a problems with the network or the queue manager. For example, in stage (1) the network topology might 'swallow' the connection request instead of responding with a positive or negative result, causing a lengthy delay in the connection request. There is a tuning property to configure a timeout for this, which is: com.ibm.mq.cfg.TCP.Connect_Timeout In stage (2), in the unusual (and rarely seen) event that the TCP/IP socket was established, but the queue manager did not respond to the negotiation request, there is another possibility of a lengthy delay. An existing tuning property allowed a timeout to be set for this stage of the connection request, which was: com.ibm.mq.cfg.MQRCVBLKTO which had its functionality extended under APAR IT03021 such that it was also used for the maximum length of time to wait for the queue manager to respond to the initial negotiation request. However, this "MQRCVBLKTO" property also overrides the timeout for all subsequent TCP/IP socket reads, regardless of what the channel heartbeat is set to. This is not a significant problem when connecting a MQ v7.0 or later client to a MQ v7.0 or later queue manager (where the channel's SHARECNV attribute is 1 or greater), as the client API will subsequently issue a heartbeat request on the client side to determine if the socket is still connected to the queue manager. However, if the application is connected to a version of the queue manager which does not support client side heartbeat requests, for example MQ v6.0 (or earlier), or connecting over a channel which has the SHARECNV(0) configuration, then the side effect of using the MQRCVBLKTO property is that if the application has requested an operation which takes an extended amount of time to complete, the MQRCVBLKTO function would interrupt the MQGET, and the MQ classes for Java/JMS would declare the connection as disconnected and report a MQRC 2009 'MQRC_CONNECTION_BROKEN' exception back to the application. For example, in the scenario where the requesting application issue a request to consume a message, with a wait interval longer than that specified by the MQRCVBLKTO, then if there were no messages on the destination Queue or Topic available within that time frame - the MQRCVBLKTO function would interrupt the request and a MQRC 2009 exception would be reported back to the application.
Problem conclusion
This APAR IT39802 has added a new tuning property to the MQ classes for Java/JMS, named: com.ibm.mq.cfg.TCP.QmgrNegotiationFlowTimeout which takes an integer value representing the number of seconds to wait for the queue manager to respond in the initial protocol communication negotiation phase. Once a heart beat interval has been passed from the queue manager to the client, this is then used as the basis for future socket timeouts in the same way as if neither the MQRCVBLKTO or QmgrNegotiationFlowTimeout had been set, avoiding the problem with the v6.0 queue manager and the long MQGET requests. For example, if it was desired to configure the application to wait a maximum of 30 seconds for the queue manager to respond to these initial negotiation flows, then the following JVM argument would be used: -Dcom.ibm.mq.cfg.TCP.QmgrNegotiationFlowTimeout=30 This property only has an effect when connecting to the queue manager over TCP/IP (that is to say - using CLIENT transport). Note: If you use both the MQRCVBLKTO and QmgrNegotiationFlowTimeout properties concurrently, then the QmgrNegotiationFlowTimeout will be used for the negotiation timeout, and MQRCVBLKTO for further waits for data from the queue manager. --------------------------------------------------------------- The fix is targeted for delivery in the following PTFs: Version Maintenance Level v9.1 LTS 9.1.0.12 v9.2 LTS 9.2.0.7 v9.3 LTS 9.3.0.1 v9.x CD 9.3.1 The latest available maintenance can be obtained from 'WebSphere MQ Recommended Fixes' http://www-1.ibm.com/support/docview.wss?rs=171&uid=swg27006037 If the maintenance level is not yet available information on its planned availability can be found in 'WebSphere MQ Planned Maintenance Release Dates' http://www-1.ibm.com/support/docview.wss?rs=171&uid=swg27006309 ---------------------------------------------------------------
Temporary fix
Comments
APAR Information
APAR number
IT39802
Reported component name
MQ WINDOWS V7
Reported component ID
5724H7220
Reported release
710
Status
CLOSED PER
PE
NoPE
HIPER
NoHIPER
Special Attention
NoSpecatt / Xsystem
Submitted date
2022-01-31
Closed date
2022-07-12
Last modified date
2022-09-08
APAR is sysrouted FROM one or more of the following:
APAR is sysrouted TO one or more of the following:
Fix information
Fixed component name
MQ WINDOWS V7
Fixed component ID
5724H7220
Applicable component levels
[{"Business Unit":{"code":"BU059","label":"IBM Software w\/o TPS"},"Product":{"code":"SSFKSJ","label":"WebSphere MQ"},"Platform":[{"code":"PF025","label":"Platform Independent"}],"Version":"7.1","Line of Business":{"code":"LOB45","label":"Automation"}}]
Document Information
Modified date:
08 September 2022