APAR status
Closed as program error.
Error description
An IBM MQ V9.1.0.0 classes for JMS application is run with trace enabled. After running for a period of time, a deadlock occurs. A Javacore taken when the deadlock occurs contains the following information: **************************************************************** ************** Deadlock detected!! Thread "<Thread 1>" is waiting for: java/util/Collections$SynchronizedMap@<object reference> which is owned by: Thread "<Thread 2>" which is waiting for: java/util/logging/FileHandler@<object reference> which is owned by: Thread "<Thread 1>" **************************************************************** ************** "<Thread1>" : : : : : : : : : : : Blocked on: java/util/Collections$SynchronizedMap@<object reference> Owned by: "<Thread2>" : : : : : : : : : : : Java callstack: at java/util/Collections$SynchronizedMap.entrySet(Collections.java: 2624(Compiled Code)) at com/ibm/msg/client/jms/admin/JmsDestinationImpl.hashCode(JmsDest inationImpl.java:795(Compiled Code)) at com/ibm/msg/client/wmq/common/internal/WMQPropertyContext.hashCo de(WMQPropertyContext.java:184(Compiled Code)) at com/ibm/msg/client/wmq/common/internal/WMQDestination.hashCode(W MQDestination.java:4842) at java/lang/Object.toString(Object.java:161(Compiled Code)) at com/ibm/msg/client/commonservices/j2se/trace/TraceUtils$GenericF ormatAdapter.formatObject(TraceUtils.java:492(Compiled Code)) : : : : : : : : : : : at java/util/logging/StreamHandler.publish(StreamHandler.java:222(C ompiled Code)) (entered lock: java/util/logging/FileHandler@0x101BEE28, entry count: 2) at java/util/logging/FileHandler.publish(FileHandler.java:693(Compi led Code)) (entered lock: java/util/logging/FileHandler@0x101BEE28, entry count: 1) at java/util/logging/Logger.log(Logger.java:749(Compiled Code)) : : : : : : : : : : : at com/ibm/msg/client/commonservices/j2se/trace/DefaultTracer.trace Data(DefaultTracer.java:357(Compiled Code)) : : : : : : : : : : : "<Thread2>" : : : : : : : : : : : Blocked on: java/util/logging/FileHandler@0x101BEE28 Owned by: "Thread-14" (J9VMThread:0x7652DD00, java/lang/Thread:0x105A9600) : : : : : : : : : : : Java callstack: at java/util/logging/FileHandler.publish(FileHandler.java:690(Compi led Code)) (entered lock: java/util/logging/FileHandler@0x101BEE28, entry count: 1) at java/util/logging/Logger.log(Logger.java:749(Compiled Code)) : : : : : : : : : : : at com/ibm/msg/client/commonservices/trace/Trace.exit(Trace.java:84 3(Compiled Code)) at com/ibm/msg/client/wmq/common/internal/WMQDestination.mapFromCan onical(WMQDestination.java:4478(Compiled Code)) at com/ibm/msg/client/jms/internal/JmsPropertyContextImpl.getObject Property(JmsPropertyContextImpl.java:587(Compiled Code)) at com/ibm/msg/client/jms/internal/JmsReadablePropertyContextImpl.g etStringProperty(JmsReadablePropertyContextImpl.java:334(Compile d Code)) at com/ibm/msg/client/jms/admin/JmsDestinationImpl.hashCode(JmsDest inationImpl.java:801(Compiled Code)) (entered lock: java/util/Collections$SynchronizedMap@0x10FB8F98, entry count: 1) at com/ibm/msg/client/wmq/common/internal/WMQPropertyContext.hashCo de(WMQPropertyContext.java:184(Compiled Code)) at com/ibm/msg/client/wmq/common/internal/WMQDestination.hashCode(W MQDestination.java:4842) : : : : : : : : : : : **************************************************************** **************
Local fix
Problem summary
**************************************************************** USERS AFFECTED: This issue affects users of: - The IBM MQ V9.0.0.3 (and later) classes for JMS. - The IBM MQ V9.0.0.3 (and later) resource adapter. - The IBM MQ V9.1.0.0 classes for JMS. - The IBM MQ V9.1.0.0 resource adapter. who are asked to collect trace by IBM Support. Platforms affected: MultiPlatform **************************************************************** PROBLEM DESCRIPTION: APAR IT23101 modified the IBM MQ classes for JMS and resource adapter to add some additional locking to the internal method: JmsDestinationImpl.hashcode() in order to prevent two threads accessing an internal Map of properties on a JMS destination at the same time. As a result of this change, it was possible for a deadlock to occur if IBM MQ messaging client trace was enabled, and two or more threads were trying to trace information about the same JMS destination at the same time. The following sequence of events would lead up to the deadlock: - Thread 1 locked a java/util/logging/FileHandler object, in order to write information about the JMS destination to a trace file. - Thread 2 tried to construct a String representation of the same JMS destination, by calling the method Destination.toString(). The JMS destination was stored internally in a WMQDestination object. As the WMQDestination class did not provide a toString() method, the attempt to construct a String representing the JMS destination resulted in the method: WMQDestination.hashCode() being called. This method, in turn, called the internal method: JmsDestinationImpl.hashCode() which locked an java/util/Collections$SynchronizedMap object (added as part of APAR IT23101) in order to read the properties of the JMS destination. - Next, Thread 1 tried to read the properties of the JMS destination. However, it became blocked as it required access to the java/util/Collections$SynchronizedMap object that was locked by Thread 2. - While processing the properties of the JMS destination, Thread 2 tried to trace out the property values. In order to do this, it required access to the java/util/logging/FileHandler object which was locked by Thread 1.
Problem conclusion
The IBM MQ classes for JMS have been updated so that the internal WMQDestination class now has a toString() method, which returns a String containing a URI representing the JMS destination. When information about a JMS destination needs to be written out to a trace file, the new WMQDestination.toString() method is called, rather than WMQDestination.hashCode(). Because the WMQDestination.toString() method does not take any internal locks, one thread is able to trace out information about the JMS destination at the same time as another thread is reading the destination's properties, which prevents the deadlock from occurring. --------------------------------------------------------------- The fix is targeted for delivery in the following PTFs: Version Maintenance Level v9.0 LTS 9.0.0.6 v9.1 CD 9.1.1 v9.1 LTS 9.1.0.1 The latest available maintenance can be obtained from 'WebSphere MQ Recommended Fixes' http://www-1.ibm.com/support/docview.wss?rs=171&uid=swg27006037 If the maintenance level is not yet available information on its planned availability can be found in 'WebSphere MQ Planned Maintenance Release Dates' http://www-1.ibm.com/support/docview.wss?rs=171&uid=swg27006309 ---------------------------------------------------------------
Temporary fix
Comments
APAR Information
APAR number
IT26587
Reported component name
IBM MQ BASE M/P
Reported component ID
5724H7261
Reported release
900
Status
CLOSED PER
PE
NoPE
HIPER
NoHIPER
Special Attention
NoSpecatt / Xsystem
Submitted date
2018-10-11
Closed date
2018-10-16
Last modified date
2018-10-16
APAR is sysrouted FROM one or more of the following:
APAR is sysrouted TO one or more of the following:
Fix information
Fixed component name
IBM MQ BASE M/P
Fixed component ID
5724H7261
Applicable component levels
[{"Business Unit":{"code":"BU053","label":"Cloud & Data Platform"},"Product":{"code":"SSYHRD","label":"IBM MQ"},"Component":"","ARM Category":[],"Platform":[{"code":"PF025","label":"Platform Independent"}],"Version":"9.0","Edition":"","Line of Business":{"code":"LOB45","label":"Automation"}}]
Document Information
Modified date:
16 October 2018