IT39358: SSL protocol exception reported by IBM MQ Java/JMS clients during TLS handshake when qmgr keystore has many certificates

APAR status

• Closed as program error.

Error description

• MQ Java/JMS clients fail with SSLProtocolException when
  certificate request message from server exceeds 32768
  bytes(32KB) during TLS handshake. The size of the message which
  includes DN list might exceed 32KB if the keystore contains lot
  of certificates on the queue manager side. The following
  exception is reported on the
  client side. The user who reported this problem configured 299
  certificates on the keystore at the queue manager side.
  
    javax.net.ssl.SSLProtocolException: The size of the handshake
  message (32920) exceeds the maximum allowed size (32768)
          at com.ibm.jsse2.aj.e(aj.java:140)
          at com.ibm.jsse2.aj.b(aj.java:3)
          at com.ibm.jsse2.aZ.a(aZ.java:44)
          at com.ibm.jsse2.bi.b(bi.java:219)
          at com.ibm.jsse2.bi.f(bi.java:317)
          at com.ibm.jsse2.bi.a(bi.java:218)
          at com.ibm.jsse2.bi.startHandshake(bi.java:212)
  

Local fix

• Set the jdk.tls.maxHandshakeMessageSize property with a value
  large than 32K on the client side and restart the application.
  
  For example: To specify a value of 512K, the Java command line
  might include:
  
  -Djdk.tls.maxHandshakeMessageSize=524288
  

Problem summary

• ****************************************************************
  USERS AFFECTED:
  Users using MQ Java/JMS clients with java version that supports
  system property jdk.tls.maxHandshakeMessageSize and are using a
  keystore with several certificates on the queue manager side.
  
  The issue was seen with a keystore containing approximately 300
  certificates when reported to IBM however the behaviour is
  highly dependent on the size of certificates and the length of
  certificate chains causing the 32KB limit to be exceeded.
  
  
  Platforms affected:
  MultiPlatform
  
  ****************************************************************
  PROBLEM DESCRIPTION:
  The JDK added a new system property
  jdk.tls.maxHandshakeMessageSize to improve certificate chain
  handling. This property sets the maximum allowed size for
  handshake message in TLS handshaking with default value of 32768
  bytes(32KB). When the keystore on the queue manager contains lot
  of certificates then the handshake message might exceed the
  default 32KB limit causing a failure on the client.
  

Problem conclusion

• The solution for the problem is to set the
  jdk.tls.maxHandshakeMessageSize property with a value large
  than 32K on the client side.
  
  For example: To specify a value of 512K, the Java command line
  might include:
  
  -Djdk.tls.maxHandshakeMessageSize=524288
  
  An workaround for the problem is to reduce the size of the
  certificate request message sent and
  the procedure is as follows
  
  * Apply the fix for this APAR IT39358
  * Set the environment variable "AMQ_NO_CERTREQ_DNLIST_ENABLE" to
  1 or TRUE on the queue manager.
     Setting this environment variable  disables sending of the
  permitted DN List in the CertificateRequest message and
     thereby reducing the size of the message.
  * Restart the queue manager
  
  ---------------------------------------------------------------
  The fix is targeted for delivery in the following PTFs:
  
  Version    Maintenance Level
  v9.2 LTS   9.2.0.7
  v9.3 LTS   9.3.0.5
  v9.x CD    9.3.2
  
  The latest available maintenance can be obtained from
  'WebSphere MQ Recommended Fixes'
  http://www-1.ibm.com/support/docview.wss?rs=171&uid=swg27006037
  
  If the maintenance level is not yet available information on
  its planned availability can be found in 'WebSphere MQ
  Planned Maintenance Release Dates'
  http://www-1.ibm.com/support/docview.wss?rs=171&uid=swg27006309
  ---------------------------------------------------------------
  

Temporary fix

Comments

APAR Information

• APAR is sysrouted FROM one or more of the following:

• APAR is sysrouted TO one or more of the following:

Fix information

• Fixed component name

  MQ BASE V9.2

• Fixed component ID

  5724H7281

Applicable component levels