Diagnostic improvements to MQ SSL/TLS error AMQ9633

Body

The AMQ9633 "bad certificate" error message has been enhanced with some additional information to help you understand which certificate was rejected and why. This is the most common error seen for SSL/TLS handshake errors, and these instructions explain how to use the information in the message to understand the cause of the error.

Example AMQ9633 error

Below is an example of the new style of message with the new section highlighted in bold text, taken from a queue manager error log:

23/10/2014 08:31:41 - Process(1234.2) User(Fred) Program(amqrmppa.exe)
                      Host(hostname.example.com) Installation(Installation1)
                      VRMF(9.3.0.5) QMgr(QMGR1)
                     
AMQ9633: Bad SSL certificate for channel '????'.

EXPLANATION:
A certificate encountered during SSL handshaking is regarded as bad for one of
the following reasons:
(a) it was formatted incorrectly and could not be validated
(b) it was formatted correctly but failed validation against the Certification
  Authority (CA) root and other certificates held on the local system
(c) it was found in a Certification Revocation List (CRL) on an LDAP server
(d) a CRL was specified but the CRL could not be found on the LDAP server
(e) an OCSP responder has indicated that it is revoked

The channel is '????'; in some cases its name cannot be determined and so is
shown as '????'. The remote host is 'host2.example.com (10.0.0.2)'. The channel
did not start.

The details of the certificate which could not be validated are
'[Class=]GSKVALMethod::X509[Issuer=]CN=Example Certificate Authority,O=IBM,C=US[#=]67322456fe4af6de[Subject=]CN=Example MQ Certificate,O=IBM,C=US'.

The certificate validation error was 575010.
ACTION:
Check which of the possible causes applies on your system. Correct the error,
and restart the channel.

This error might indicate that the remote end of the channel is configured to
send the wrong certificate. Check the certificate label configuration at the
remote end of the channel and ensure that the local key repository contains all
of the necessary CA certificates.

For completeness, I should also mention that MQ 8.0 is able to determine the channel name in many cases so you should see fewer errors for channel '????' (although there are still some cases where we can't do that yet). The error messages are improving over time!

Understanding this error

The bold text in the message above shows the details of the problematic certificate. The main fields are:

• [Issuer=] - the Issuer Distinguished Name (DN) of the certificate, in this case CN=Example Certificate Authority,O=IBM,C=US
• [Subject=] - the Subject DN of the certificate, in this case CN=Example MQ Certificate,O=IBM,C=US
• [#=] - the serial number of the certificate in hexadecimal string form, in this case 67322456fe4af6de

So now we know which certificate had a problem. But what went wrong?

For this, there's a numeric certificate validation error, which is usually a 6-digit number beginning with "575". The certificate validation errors are listed in the product documentation in Table 2 of the Transport Layer Security (TLS) return codes topic. For convenience, here's a copy of the current table:

This table shows that the example error 575010 means that no certificate chain was built. A certificate chain is a sequence of related certificates which are linked by digital signatures (each certificate is the signer of the next one, up to the root certificate which is self-signed).

Failure to build a certificate chain usually means that our key repository lacks one of the Certificate Authority (CA) certificates needed to validate the remote certificate. Other errors have different causes, but since a missing CA certificate is a common error then it's worth looking at the next steps.

How to fix this error

In this example, our queue manager which wrote the error message doesn't have one of the CA certificates required. The first thing to do is to check that the issuer of the certificate is present in the queue manager's key repository. In this case, the issuer has a DN of "CN=Example Certificate Authority,O=IBM,C=US" so check to see if the CA certificate is in the key repository. To do this, we must search for a certificate with a Subject DN of "CN=Example Certificate Authority,O=IBM,C=US"; this is because a CA certificate's Subject DN is equal to the Issuer DN of any certificates signed by that CA, so we must find the certificate with a Subject DN that matches the problematic certificate's Issuer DN.

To check for the CA certificate, either:

• Start the iKeyman GUI using the strmqikm command. Open the key repository file and then navigate to the Signer Certificates section using the drop-down list in the middle of the window. Double-click each certificate label in turn to view the certificate.
• Use the command-line to find out which certificates are in the key repository by listing their labels. Then display the details for each certificate in turn until you find the one with the correct subject:

runmqckm -cert -list -db key.kdb -pw password

runmqckm -cert -details -db key.kdb -pw password -label certlabel

Since this is a 575010 error, you will probably find that the CA certificate is missing so this is the CA certificate you need to add. Follow the steps described in the product documentation Adding a CA certificate (or the public part of a self-signed certificate) into a key repository, on AIX, Linux or Windows systems topic to add the missing CA certificate.

However, it's possible that you might find the CA certificate is already present. In that case, check whether it's a root CA: generally a root CA certificate has Subject DN and Issuer DN fields set to the same value. If the CA certificate is not a root CA then take the Issuer DN and look for the certificate which has that as its Subject DN: add that certificate if it's missing. Repeat the process as many times as necessary until you reach a root CA certificate that's in the key repository.

Important final steps

Finally, once all certificates are in place, remember that MQ caches the key repository in memory so that changes don't take effect immediately. To clear the cache:

• For a client application, restart the application.
• For a queue manager, issue the REFRESH SECURITY TYPE(SSL) MQSC command. Note that this command will terminate any active SSL/TLS channels, so take care to run it at a suitable time.

People often forget to clear the cached key repository and then wonder why their changes don't take effect. If you change your key repository in any way, always remember to clear the cached data.

The ability to examine the certificates and clear the cached key repository state are useful for many SSL/TLS connectivity problems. Now you know how to examine the certificates, you can use this technique to fix other AMQ9633 errors using the numeric validation error and the certificate details.