Problem

Overview

This article provides instructions on how to rotate your nodes' certificates to prevent them from expiring for both self-signed and CA-signed certificates. The focus is on node-to-node communication, but client-to-node certificate rotation would be handled in a similar manner. 

To implement this, nodes should be restarted in a rolling fashion, so single node downtime cannot be avoided. The clients will experience connection errors to the downed node, and any in-flight queries will fail if sent to the downed node. These errors can be handled by implementing a retry mechanism. If the replication factor of keyspaces is greater than one, then one node going down at a time should not affect the client's ability to read and write. However, if the replication factor of keyspace equals one, then one node being down means that a chunk of data will not be available for reads and writes, and there will be downtime.

Applies to

• Cassandra 2.x
• Cassandra 3.x

Prerequisites

This article assumes that valid new certificates/keystores have already been generated. This article is also not meant to be comprehensive as there are a number of TLS configurations that can be in place. Knowledge of TLS certificates is assumed.

CA-Signed Certificates (Chain Length > 1) - Leaf Certificate

Important: The certificate chain must be the same between the old and new certificates. Refer to "Intermediate or Root Certificate" section if it is changing.

On each node in the cluster:

• Update server_encryption_options.keystore and server_encryption_options.keystore_password to the new keystore values
• Restart the node
• Only proceed to the next node if communication is successfully established between all nodes

CA-Signed Certificates (Chain Length > 1) - Intermediate or Root Certificate

• Add the new public certificate chain (root and any intermediates) to an existing node's truststore such that both the old certificates and new certificates exist in the same truststore

 keytool -importcert -file {cert} -alias {new_alias} -keystore {existing_truststore}

• Distribute the truststore to all nodes in the cluster replacing the existing truststore in use by the node
• Perform a rolling restart of the cluster to load the new truststore and establish trust for the new keys prior to them being in place

On each node in the cluster:

• Update server_encryption_options.keystore and server_encryption_options.keystore_password to the new keystore values
• Restart the node
• Only proceed to the next node if communication is successfully established between all nodes

Optional cleanup once the keystore has been updated on all nodes:

• Backup the truststore, then remove the old certificate chain from the truststore and distribute the truststore with only the new certificate chain to all nodes, replacing the existing truststore as specified in the cassandra.yaml

keytool -delete -alias {old_alias} -keystore {truststore}

• Perform a rolling restart of the cluster to load the pruned truststore

Self-Signed Certificates (Chain Length 1)

• Add the certificate for each unique generated key pair to an existing node's truststore such that both the old certificates and new certificates exist in the same truststore

 keytool -importcert -file {cert} -alias {new_alias} -keystore {existing_truststore}

• Distribute the truststore to all nodes in the cluster replacing the existing truststore in use by the node
• Perform a rolling restart of the cluster to load the new truststore and establish trust for the new keys prior to them being in place

On each node in the cluster:

• Update server_encryption_options.keystore and server_encryption_options.keystore_password to the new keystore values
• Restart the node
• Only proceed to the next node if communication is successfully established between all nodes

Optional cleanup once the keystore has been updated on all nodes:

• Backup the truststore, then remove the old certificate chain from the truststore and distribute the truststore with only the new certificate chain to all nodes, replacing the existing truststore as specified in the cassandra.yaml

keytool -delete -alias {old_alias} -keystore {truststore}

• Perform a rolling restart of the cluster to load the pruned truststore