From MQ 22.214.171.124 onwards, the dreaded AMQ9633 "bad certificate" error message was enhanced with some additional information to help you understand which certificate was rejected and why. This is the most common error seen for SSL/TLS handshake errors and so I'd like to explain how to use the new information to understand the cause of the error.
Example AMQ9633 error
Below is an example of the new style of message with the new section highlighted in bold text, taken from a queue manager error log:
23/10/2014 08:31:41 - Process(1234.2) User(Fred) Program(amqrmppa.exe)
AMQ9633: Bad SSL certificate for channel '????'.
A certificate encountered during SSL handshaking is regarded as bad for one of
the following reasons:
(a) it was formatted incorrectly and could not be validated
(b) it was formatted correctly but failed validation against the Certification
Authority (CA) root and other certificates held on the local system
(c) it was found in a Certification Revocation List (CRL) on an LDAP server
(d) a CRL was specified but the CRL could not be found on the LDAP server
(e) an OCSP responder has indicated that it is revoked
The channel is '????'; in some cases its name cannot be determined and so is
shown as '????'. The remote host is 'host2.example.com (10.0.0.2)'. The channel
did not start.
The details of the certificate which could not be validated are
'[Class=]GSKVALMethod::X509[Issuer=]CN=Example Certificate Authority,O=IBM,C=US[#=]67322456fe4af6de[Subject=]CN=Example MQ Certificate,O=IBM,C=US'.
The certificate validation error was 575010.
Check which of the possible causes applies on your system. Correct the error,
and restart the channel.
This error might indicate that the remote end of the channel is configured to
send the wrong certificate. Check the certificate label configuration at the
remote end of the channel and ensure that the local key repository contains all
of the necessary CA certificates.
For completeness, I should also mention that MQ 8.0 is able to determine the channel name in many cases so you should see fewer errors for channel '????' (although there are still some cases where we can't do that yet). The error messages are improving over time!
Understanding this error
The bold text in the message above shows the details of the problematic certificate. The main fields are:
- [Issuer=] - the Issuer Distinguished Name (DN) of the certificate, in this case CN=Example Certificate Authority,O=IBM,C=US
- [Subject=] - the Subject DN of the certificate, in this case CN=Example MQ Certificate,O=IBM,C=US
- [#=] - the serial number of the certificate in hexadecimal string form, in this case 67322456fe4af6de
So now we know which certificate had a problem. But what went wrong?
For this, there's a numeric certificate validation error, which is usually a 6-digit number beginning with "575". The certificate validation errors are listed in the product documentation in Table 2 of the Secure Sockets Layer (SSL) and Transport Layer Security (TLS) return codes topic. For convenience, here's a copy of the current table:
|Return code (decimal)||Explanation|
|575002||ASN error due to a malformed certificate|
|575004||Key database error|
|575006||Invalid implementation library|
|575008||No appropriate validator|
|575009||The root CA is not trusted|
|575010||No certificate chain was built|
|575011||Digital signature algorithm mismatch|
|575012||Digital signature mismatch|
|575013||X.509 version does not allow Key IDs|
|575014||X.509 version does not allow extensions|
|575015||Unknown X.509 certificate version|
|575016||The certificate validity range is invalid|
|575017||The certificate is not yet valid|
|575018||The certificate has expired|
|575019||The certificate contains unknown critical extensions|
|575020||The certificate contains duplicate extensions|
|575021||The issuers directory name does not match the issuer's issuer|
|575022||The Authority Key ID serial number value does not match the serial number of the issuer|
|575023||The Authority Key ID and Subject Key ID do not match|
|575024||Unrecognized issuer alternative name|
|575025||The certificate Basic Constraints forbid use as a CA|
|575026||The certificate has a non-zero Basic Constraints path length but is not a CA|
|575027||The certificate Basic Constraints maximum path length was exceeded|
|575028||The certificate is not permitted to sign other certificates|
|575029||The certificate is not signed by a CA|
|575030||Unrecognized Subject Alternative Name|
|575031||The certificate chain is invalid|
|575032||The certificate is revoked|
|575033||Unrecognized CRL distribution point|
|575034||Name chaining failed|
|575035||Certificate is not in a chain|
|575036||The CRL is not yet valid|
|575037||The CRL has expired|
|575038||The certificate version does not allow critical extensions|
|575039||Unknown CRL distribution points|
|575040||No CRLs for CRL distribution points|
|575041||Indirect CRLs are not supported|
|575042||Missing issuing CRL distribution point name|
|575043||Distribution points do not match|
|575044||No available CRL data source|
|575045||CA Subject name is null|
|575046||Distinguished names do not chain|
|575047||Missing Subject Alternative Name|
|575048||Unique ID mismatch|
|575049||Name not permitted|
|575051||CA certificate is missing Critical Basic Constraints|
|575052||Name constraints are not critical|
|575053||Name constraints minimum subtree value if set is not zero|
|575054||Name constraints maximum subtree value if set is not allowed|
|575055||Unsupported name constraint|
|575056||Empty policy constraints|
|575057||Bad certificate policies|
|575058||Certificate policies not acceptable|
|575059||Bad acceptable certificate policies|
|575060||Certificate policy mappings are critical|
|575061||Revocation status could not be determined|
|575062||Extended key usage error|
|575063||Unknown OCSP version|
|575064||Unknown OCSP response|
|575065||Bad OCSP key usage extension|
|575066||Bad OCSP nonce|
|575067||Missing OCSP nonce|
|575068||No OCSP client available|
So we can see that our example error 575010 means that no certificate chain was built. A certificate chain is a sequence of related certificates which are linked by digital signatures (each certificate is the signer of the next one, up to the root certificate which is self-signed).
Failure to build a certificate chain usually means that our key repository lacks one of the Certificate Authority (CA) certificates needed to validate the remote certificate. Other errors have different causes, but since a missing CA certificate is a common error then it's worth looking at the next steps.
How to fix this error
In this example, our queue manager which wrote the error message doesn't have one of the CA certificates required. The first thing to do is to check that the issuer of the certificate is present in the queue manager's key repository. In this case, the issuer has a DN of "CN=Example Certificate Authority,O=IBM,C=US" so check to see if the CA certificate is in the key repository. To do this, we must search for a certificate with a Subject DN of "CN=Example Certificate Authority,O=IBM,C=US"; this is because a CA certificate's Subject DN is equal to the Issuer DN of any certificates signed by that CA, so we must find the certificate with a Subject DN that matches the problematic certificate's Issuer DN.
To check for the CA certificate, either:
- Start the iKeyman GUI using the strmqikm command. Open the key repository file and then navigate to the Signer Certificates section using the drop-down list in the middle of the window. Double-click each certificate label in turn to view the certificate.
- Use the command-line to find out which certificates are in the key repository by listing their labels. Then display the details for each certificate in turn until you find the one with the correct subject:
runmqckm -cert -list -db key.kdb -pw password
runmqckm -cert -details -db key.kdb -pw password -label certlabel
Since this is a 575010 error, you'll probably find that the CA certificate is missing so this is the CA certificate you need to add. Follow the steps described in the product documentation Adding a CA certificate (or the public part of a self-signed certificate) into a key repository, on UNIX®, Linux or Windows systems topic to add the missing CA certificate.
However, it's possible that you might find the CA certificate is already present. In that case, check whether it's a root CA: generally a root CA certificate has Subject DN and Issuer DN fields set to the same value. If the CA certificate is not a root CA then take the Issuer DN and look for the certificate which has that as its Subject DN: add that certificate if it's missing. Repeat the process as many times as necessary until you reach a root CA certificate that's in the key repository.
Important final steps
Finally, once all certificates are in place, remember that MQ caches the key repository in memory so that changes don't take effect immediately. To clear the cache:
- For a client application, restart the application.
- For a queue manager, issue the REFRESH SECURITY TYPE(SSL) MQSC command. Note that this command will terminate any active SSL/TLS channels, so take care to run it at a suitable time.
People often forget to clear the cached key repository and then wonder why their changes don't take effect. If you change your key repository in any way, always remember to clear the cached data.
The ability to examine the certificates and clear the cached key repository state are useful for many SSL/TLS connectivity problems. Now you know how to examine the certificates, you can use this technique to fix other AMQ9633 errors using the numeric validation error and the certificate details.
title image (modified) credit: (cc) Some rights reserved by Nemo