Debugging node identities issues

Node identities consist of X.509 Transport Layer Security (TLS) certificates and private key pairs.

To support secure communication between cluster nodes, the administration daemon of the native REST API uses node identities for TLS connections. Each node in the cluster must have a node identity configured. Although you can reuse the same certificate and private key pair across multiple nodes, it is more secure to generate unique certificates per node. For more information, see Node identities.

The Go crypto x509 library is used to establish mutual TLS connections between cluster nodes.

To debug potential X.509 certificate issues, use the OpenSSL server and client programs to verify that node identities can establish connections with each other. The following commands initiate a server and a client that attempts to establish a connection:

openssl s_server -accept 7569 -cert <certfile> -CAfile <cafile> -key <keyfile> -Verify 1 -tls1_3 -debug -state

openssl s_client -connect 0.0.0.0:7569 -CAfile <cachain> -cert <cert> -key <key> -verify 1 -tls1_3 -debug -showcerts -status

Known issues with node identities

The following are common errors related to node identities:

Certificate signed by unknown authority

tls: failed to verify certificate: x509: certificate signed by unknown authority: unable to validate Admin Daemon's node identity. 
Use the --cacert flag to point to a valid CA chain to validate the server certificate, add the CA chain to your system trust store, or disable client side validation using --insecure-skip-tls-verify flag

Certificate does not contain subject alternative names (SANs)

tls: failed to verify certificate: x509: cannot validate certificate for 192.168.1.10 because it doesn't contain any IP SANs: 
make sure --url flag target matches the SAN addresses of the Admin Daemon's node identity

SANs do not match with the details in the certificate

tls: failed to verify certificate: x509: certificate is valid for example.com, www.example.com, api.example.com, not example: 
make sure --url flag target matches the SAN addresses of the Admin Daemon's node identity

To resolve these issues, configure the CA trust chain and ensure that the certificate includes the correct SAN details. For more information, see Node identities.