IBM Support

Diagnostic improvements to MQ SSL/TLS error AMQ9633

Technical Blog Post


Abstract

Diagnostic improvements to MQ SSL/TLS error AMQ9633

Body

 

From MQ 7.1.0.2 onwards, the dreaded AMQ9633 "bad certificate" error message was enhanced with some additional information to help you understand which certificate was rejected and why. This is the most common error seen for SSL/TLS handshake errors and so I'd like to explain how to use the new information to understand the cause of the error.

Example AMQ9633 error

Below is an example of the new style of message with the new section highlighted in bold text, taken from a queue manager error log:

23/10/2014 08:31:41 - Process(1234.2) User(Fred) Program(amqrmppa.exe)
                      Host(hostname.example.com) Installation(Installation1)
                      VRMF(7.5.0.3) QMgr(QMGR1)
                     
AMQ9633: Bad SSL certificate for channel '????'.

EXPLANATION:
A certificate encountered during SSL handshaking is regarded as bad for one of
the following reasons:
(a) it was formatted incorrectly and could not be validated
(b) it was formatted correctly but failed validation against the Certification
  Authority (CA) root and other certificates held on the local system
(c) it was found in a Certification Revocation List (CRL) on an LDAP server
(d) a CRL was specified but the CRL could not be found on the LDAP server
(e) an OCSP responder has indicated that it is revoked

The channel is '????'; in some cases its name cannot be determined and so is
shown as '????'. The remote host is 'host2.example.com (10.0.0.2)'. The channel
did not start.

The details of the certificate which could not be validated are
'[Class=]GSKVALMethod::X509[Issuer=]CN=Example Certificate Authority,O=IBM,C=US[#=]67322456fe4af6de[Subject=]CN=Example MQ Certificate,O=IBM,C=US'.

The certificate validation error was 575010.

ACTION:
Check which of the possible causes applies on your system. Correct the error,
and restart the channel.

This error might indicate that the remote end of the channel is configured to
send the wrong certificate. Check the certificate label configuration at the
remote end of the channel and ensure that the local key repository contains all
of the necessary CA certificates.

 

For completeness, I should also mention that MQ 8.0 is able to determine the channel name in many cases so you should see fewer errors for channel '????' (although there are still some cases where we can't do that yet). The error messages are improving over time!

Understanding this error

The bold text in the message above shows the details of the problematic certificate. The main fields are:

  • [Issuer=] - the Issuer Distinguished Name (DN) of the certificate, in this case CN=Example Certificate Authority,O=IBM,C=US
  • [Subject=] - the Subject DN of the certificate, in this case CN=Example MQ Certificate,O=IBM,C=US
  • [#=] - the serial number of the certificate in hexadecimal string form, in this case 67322456fe4af6de

So now we know which certificate had a problem. But what went wrong?

For this, there's a numeric certificate validation error, which is usually a 6-digit number beginning with "575". The certificate validation errors are listed in the product documentation in Table 2 of the Secure Sockets Layer (SSL) and Transport Layer Security (TLS) return codes topic. For convenience, here's a copy of the current table:

 

Return code (decimal)Explanation
575001Internal error
575002ASN error due to a malformed certificate
575003Cryptographic error
575004Key database error
575005Directory error
575006Invalid implementation library
575008No appropriate validator
575009The root CA is not trusted
575010No certificate chain was built
575011Digital signature algorithm mismatch
575012Digital signature mismatch
575013X.509 version does not allow Key IDs
575014X.509 version does not allow extensions
575015Unknown X.509 certificate version
575016The certificate validity range is invalid
575017The certificate is not yet valid
575018The certificate has expired
575019The certificate contains unknown critical extensions
575020The certificate contains duplicate extensions
575021The issuers directory name does not match the issuer's issuer
575022The Authority Key ID serial number value does not match the serial number of the issuer
575023The Authority Key ID and Subject Key ID do not match
575024Unrecognized issuer alternative name
575025The certificate Basic Constraints forbid use as a CA
575026The certificate has a non-zero Basic Constraints path length but is not a CA
575027The certificate Basic Constraints maximum path length was exceeded
575028The certificate is not permitted to sign other certificates
575029The certificate is not signed by a CA
575030Unrecognized Subject Alternative Name
575031The certificate chain is invalid
575032The certificate is revoked
575033Unrecognized CRL distribution point
575034Name chaining failed
575035Certificate is not in a chain
575036The CRL is not yet valid
575037The CRL has expired
575038The certificate version does not allow critical extensions
575039Unknown CRL distribution points
575040No CRLs for CRL distribution points
575041Indirect CRLs are not supported
575042Missing issuing CRL distribution point name
575043Distribution points do not match
575044No available CRL data source
575045CA Subject name is null
575046Distinguished names do not chain
575047Missing Subject Alternative Name
575048Unique ID mismatch
575049Name not permitted
575050Name excluded
575051CA certificate is missing Critical Basic Constraints
575052Name constraints are not critical
575053Name constraints minimum subtree value if set is not zero
575054Name constraints maximum subtree value if set is not allowed
575055Unsupported name constraint
575056Empty policy constraints
575057Bad certificate policies
575058Certificate policies not acceptable
575059Bad acceptable certificate policies
575060Certificate policy mappings are critical
575061Revocation status could not be determined
575062Extended key usage error
575063Unknown OCSP version
575064Unknown OCSP response
575065Bad OCSP key usage extension
575066Bad OCSP nonce
575067Missing OCSP nonce
575068No OCSP client available

So we can see that our example error 575010 means that no certificate chain was built. A certificate chain is a sequence of related certificates which are linked by digital signatures (each certificate is the signer of the next one, up to the root certificate which is self-signed).

Failure to build a certificate chain usually means that our key repository lacks one of the Certificate Authority (CA) certificates needed to validate the remote certificate. Other errors have different causes, but since a missing CA certificate is a common error then it's worth looking at the next steps.

How to fix this error

In this example, our queue manager which wrote the error message doesn't have one of the CA certificates required. The first thing to do is to check that the issuer of the certificate is present in the queue manager's key repository. In this case, the issuer has a DN of "CN=Example Certificate Authority,O=IBM,C=US" so check to see if the CA certificate is in the key repository. To do this, we must search for a certificate with a Subject DN of "CN=Example Certificate Authority,O=IBM,C=US"; this is because a CA certificate's Subject DN is equal to the Issuer DN of any certificates signed by that CA, so we must find the certificate with a Subject DN that matches the problematic certificate's Issuer DN.

To check for the CA certificate, either:

  • Start the iKeyman GUI using the strmqikm command. Open the key repository file and then navigate to the Signer Certificates section using the drop-down list in the middle of the window. Double-click each certificate label in turn to view the certificate.
  • Use the command-line to find out which certificates are in the key repository by listing their labels. Then display the details for each certificate in turn until you find the one with the correct subject:

runmqckm -cert -list -db key.kdb -pw password

runmqckm -cert -details -db key.kdb -pw password -label certlabel

Since this is a 575010 error, you'll probably find that the CA certificate is missing so this is the CA certificate you need to add. Follow the steps described in the product documentation Adding a CA certificate (or the public part of a self-signed certificate) into a key repository, on UNIX®, Linux or Windows systems topic to add the missing CA certificate.

However, it's possible that you might find the CA certificate is already present. In that case, check whether it's a root CA: generally a root CA certificate has Subject DN and Issuer DN fields set to the same value. If the CA certificate is not a root CA then take the Issuer DN and look for the certificate which has that as its Subject DN: add that certificate if it's missing. Repeat the process as many times as necessary until you reach a root CA certificate that's in the key repository.

Important final steps

Finally, once all certificates are in place, remember that MQ caches the key repository in memory so that changes don't take effect immediately. To clear the cache:

  • For a client application, restart the application.
  • For a queue manager, issue the REFRESH SECURITY TYPE(SSL) MQSC command. Note that this command will terminate any active SSL/TLS channels, so take care to run it at a suitable time.

People often forget to clear the cached key repository and then wonder why their changes don't take effect. If you change your key repository in any way, always remember to clear the cached data.

The ability to examine the certificates and clear the cached key repository state are useful for many SSL/TLS connectivity problems. Now you know how to examine the certificates, you can use this technique to fix other AMQ9633 errors using the numeric validation error and the certificate details.

 

 

title image (modified) credit: (cc) Some rights reserved by Nemo

 

[{"Business Unit":{"code":"BU004","label":"Hybrid Cloud"},"Product":{"code":"","label":""},"Component":"","Platform":[{"code":"","label":""}],"Version":"","Edition":""}]

UID

ibm11081227