IBM Support

SSL3_GET_SERVER_CERTIFICATE certificate verify failed after removing a single CA ROOT Certificate from the trusted root file

Technical Blog Post


Abstract

SSL3_GET_SERVER_CERTIFICATE certificate verify failed after removing a single CA ROOT Certificate from the trusted root file

Body

Problem(Abstract)

SSL3_GET_SERVER_CERTIFICATE certificate verify failed after removing a single CA ROOT Certificate from the trusted root file. After removing a certain CA ROOT Certificate from the trusted root file, which had no certificate chain, it produced an error CSPA309E.

Symptom

In this case the following error definition means the certificate chain is broken, SSL3_READ_BYTES tlsv1 alert unknown ca - SSL alert number 48:

/usr/bin $ ./openssl version

OpenSSL 0.9.7d 17 Mar 2004 (+ security patches to 2006-09-29)

Error captured in the SMGR/COMM trace

PID=670 06/04 09:04:34:877662 -> ndm_error_get
=670 errhandle: 0x003CEE78
=672 cdspssl: SSL Library Version: OpenSSL 0.9.7m-fips 23 Feb 2007
=670 errkey: 4
=670 <- ndm_error_get:GET_FIRST_ERROR Failed Entry

PID=672 06/04 09:04:34:935543 COMM TRACE RECEIVE
=672 RH VALUE 0x38000000
=672 RH DECODE RESPONSE: NEGATIVE CHG_DIR
=672
=672 OFFSET BUFFER LENGTH = 7
=672 00000000 15030100 020230 * 0 *
=672 SSL_ReadHandshake (position: 0, reading: 5, available: 7)
=672 SSL_ReadHandshake (position: 5, reading: 2, available: 7)
=672 cdspssl: SSL_do_handshake failed
=672 cdspssl: Error queue first 14094418:lib(20):func(148):reason(1048)
PID=670 06/04 09:04:34:942374 -> ndm_error_set
=670 msgid: CSPA309E
=670 feedback: 134
=670 rc: 8
=670 subst: :&VAR1=SSL3_GET_SERVER_CERTIFICATE certificate verify failed:
=670 status: 2
=670 num_entries: 2
PID=670 06/04 09:04:34:942555 ndmmsg_selstext entered.
=670 ARG error : 0x003CEE78
=670 ARG msg_id : CSPA309E
=670 ARG msg_text: 0xFFBF42F8
=670 ARG subst : :&VAR1=SSL3_GET_SERVER_CERTIFICATE certificate verify failed:
PID=670 06/04 09:04:34:942684 -> sdcf_read
PID=670 06/04 09:04:34:942715 -> sdcf_rewind
=670 <- sdcf_rewind: ok
PID=670 06/04 09:04:34:942782 -> sdcf_fileread
=670 cfh: 0x003D5968
=670 buffer: 0xFFBF2E5C
=670 key: CSPA309E
=670 <- sdcf_fileread: found CSPA309E
=670 <- sdcf_read: record ok
PID=670 06/04 09:04:34:943024 substitution entered.
=670 ARG work : The SSL library failed, reason=&VAR1
=670 ARG msg_text:
=670 ARG subst : :&VAR1=SSL3_GET_SERVER_CERTIFICATE certificate verify failed:
PID=670 06/04 09:04:34:943107 get_symbol entered.
=670 ARG work: The SSL library failed, reason=&VAR1
=670 ARG segment: 0xFFBEFD54
=670 ARG symbol : 0xFFBF0D54
=670 get_symbol exited.
PID=670 06/04 09:04:34:943229 get_symbol entered.
=670 ARG work:
=670 ARG segment: 0xFFBEFD54
=670 ARG symbol : 0xFFBF0D54
=670 get_symbol exited.
=670 substitution exited.
=670 ndmmsg_selstext exited.
=670 ndm_error_set: proc_len is 0
=670 process info from Cntl structure
PID=670 06/04 09:04:34:943407 stat_log entered.
=670 ARG error: 003CEE78
=670 ARG pname: sample
=670 ARG pnumber: 4
=670 ARG startt:
=670 ARG stopt:
=670 ARG submitter:
=670 ARG snode: cesol10-cd41
=670 ARG ccode: 8
=670 ARG recids: CSPA
=670 ARG reccat: CAPR
=670 ARG event_buffer:
MSGI=CSPA309E|LNOD=P|PNOD=cesol10-cd41|MSGT=The SSL library failed, reason=SSL3_GET_SERVER_CERTIFICATE certificate verify failed Message
ID CSPA309E, rc=8, fdbk=134.|MSST=The SSL library failed,
reason=SSL3_GET_SERVER_CERTIFICATE certificate verify failed
=670 stat_log exited.
=670 <- ndm_error_set: ok
=670 cdspssl: 670:error:14090086:SSL
routines:SSL3_GET_SERVER_CERTIFICATE:certificate verify failed:s3_clnt.c:844:

 

Cause

 

A bug was found in the version of OpenSSL we are currently using with Connect:Direct Secure+ in which the sort routine that sorts the certificates in a trust store (the trusted.txt file, e.g.) The certificates are sorted by each certificate's subject distinguished name (DN). There are several keys to the sort, one of which is the type of encoding the DN is written in. Some DNs are encoded with ASCII, some are encoded with UTF8. The bug is exposed when comparing two certificates that are encoded differently.

 

Diagnosing the problem

 

Run the following OpenSSL command on each certificate located in the trusted root file. The parameter cert.pem is an example name for each certificate.

openssl x509 -in cert.pem -noout -subject -nameopt sep_comma_plus,show_type

Example Output - Customer data is obscured

UTF-8 Encoding
$ openssl x509 -in cert1.txt -noout -subject -nameopt sep_comma_plus,show_type
subject= C=PRINTABLESTRING:US,O=UTF8STRING:Customer Name,OU=UTF8STRING:Customer Name,CN=UTF8STRING:Customer Name Public Root Certificate Authority

ASCII Encoding
$ openssl x509 -in cert2.txt -noout -subject -nameopt sep_comma_plus,show_type
subject= C=PRINTABLESTRING:US,O=PRINTABLESTRING:Customer Name,OU=PRINTABLESTRING:Customer Name CN=PRINTABLESTRING:Customer Name Global Root

Resolving the problem

Engineering has analyzed this issue and found the bug in OpenSSL that causes it.  The bug is in the sort routine that sorts the certificates in a trust store (the trusted.txt file, e.g.)  The certificates are sorted by each certificate's subject distinguished name (DN).  There are several keys to the sort, one of which is the type of encoding the DN is written in.  Some DNs are encoded with ASCII, some are encoded with UTF8. The bug is exposed when comparing two certificates that are encoded differently.  
Due to this sort bug, the binary search for a specific certificate can create errors when the certificates have a mix of encoding types.  There is no structured way to work around the bug by rearranging the certificates.

The bug was introduced in OpenSSL 0.9.7f, and not resolved until OpenSSL 0.9.8k

The certificates in the customer's trust store are encoded both with ASCII and UTF8, which is why they encountered the bug.  Unfortunately we won't be able to provide a code fix for current versions of CDU.  The bug exists behind the FIPS certified crypto boundary, and any changes there would invalidate our FIPS certification.  The issue is resolved by applying the latest maintenance dated 28 Aug 2014 or later.

Until the maintenance is applied, another option for the customer would be to reissue their trusted certificates using the same encoding for the DNs.  If that's not possible, then the another option is to use the smallest number of certificates in the trusted file for a client node, ideally only the intermediate and root certificates needed to validate the server.  The bug may still be encountered even then, but it would be much easier to rearrange the certificates to avoid it.  

Here is an OpenSSL command you can use to query the subject DN of a certificate to see how it's encoded:
openssl x509 -in cert.txt -noout -subject -nameopt sep_comma_plus,show_type

[{"Business Unit":{"code":"BU012","label":"WCE"}, "Product":{"code":"SS4PJT","label":"IBM Sterling Connect:Direct"},"Component":"","Platform":[{"code":"PF025","label":"Platform Independent"}],"Version":"","Edition":""}]

UID

ibm11123875