Managing custom trusted certificates

You can now import custom trusted certificates to Concert for TLS verification. Integrating trusted certificates across all deployment types involves adding them to a known location. In virtual machine (VM) deployments, there is a dedicated folder, while in Kubernetes, it is a specific secret. This method ensures consistent handling of certificates across different environments.

Before you begin

Concert uses an internal certificate authority (CA) bundle with all trusted certificates and during operations such as updates, Concert might regenerate its CA bundle. Therefore, any certificates that are added to the well-known locations should persist there. Later, if these certificates are removed, Concert may interpret this as you no longer require those certificates to be part of the CA bundle.

Adding invalid or corrupted certificates to Concert's CA bundle can have adverse effects. It may render the CA bundle unusable, leading to unexpected behavior, including constant restarts. Always exercise caution while updating the CA bundle to avoid such issues.

Before the CA bundle gets corrupted, you can backup Concert's private key, self-signed certificate, and CA bundle. The data that is saved on these backup 's critical to ensure that both Concert's internal communications and the connections toward external services are safe, and therefore they must be securely stored.

It is recommended to have a backup that is taken before triggering any update to Concert's CA bundle.

Backup and restore on VM

Export some variables for other commands to use. Adjust them to match your environment:

$ export CONCERT_TLS_DIR="<path_to_concert_std>/localstorage/volumes/infra/tls"
$ export BACKUP_FILE="${HOME}/concert-tls-backup.tar.gz"

To perform a backup of exising key, certificate, and bundle, use the following command:

$ tar cvzf "${HOME}/concert-tls-backup.tar.gz" -C "${CONCERT_TLS_DIR}" tls.key tls.crt tls-ca-bundle.pem
tls.key
tls.crt
tls-ca-bundle.pem

To restore the content of the backup file, run the following command:

$ tar xzvf "${HOME}/concert-tls-backup.tar.gz" -C "${CONCERT_TLS_DIR}"
tls.key
tls.crt
tls-ca-bundle.pem

Backup and restore on Kubernetes

Export the namespace where Concert is deployed, and the name of the local file that will be used for backup for other commands to use.

$ export CONCERT_NS=<your_concert_namespace>
$ export CONCERT_TLS_SECRET="${HOME}/app-cfg-internal-tls.yaml"

Note: You must be logged in to your cluster, and your user account must have the required permissions to read Kubernetes secrets.

Use the following command to save Concert's TLS secret to a file:

$ kubectl -n "${CONCERT_NS}" get secret app-cfg-internal-tls -oyaml > "${CONCERT_TLS_SECRET}"

To restore the previously saved secret, use this command:

$ kubectl -n "${CONCERT_NS}" apply -f "${CONCERT_TLS_SECRET}"

VM deployment and triggering CA bundle update

Copy any certificate that you want to add to Concert's CA bundle to localstorage/infra/custom-ca-certs folder. If you want to remove previously added certificates from the CA bundle, delete them from localstorage/infra/custom-ca-certs.

After modifying the contents of localstorage/infra/custom-ca-certs folder, run the cert-mgmt script. The cert-mgmt script is part of ibm-concert-std.tgz package that is published on Concert public GitHub.

Note: Refer to installation section for steps to download the script ibm-concert-std.tgz.

Kubernetes deployments

Export the name of the namespace where Concert is deployed, so it can be referenced by using the following command:

$ export CONCERT_NS=<concert's namespace>

Ensure that the secret app-cfg-custom-ca-certs exists in the namespace where Concert is deployed. If it does not exist, you can create it by using the following command:

$ kubectl --namespace "${CONCERT_NS}" create secret generic app-cfg-custom-ca-certs

Edit the secret to add any certificate that you want to add to the secret's data section. The key of each entry in this section is the name of the certificate file, and the value is the content of the certificate, encoded in base64 format.

To modify the secret, you can use the following command:

$ kubectl --namespace "${CONCERT_NS}" edit secret app-cfg-custom-ca-certs

Triggering CA bundle on Kubernetes

After the update is complete, log in to the management pod:

$ MGMT_POD=$(kubectl --namespace "${CONCERT_NS}" get pods -o=name | grep 'roja-mgmt-' | head -n1)
$ kubectl --namespace "${CONCERT_NS}" exec -it "${MGMT_POD}" -- bash

Now, you can access the pod and run the following command to verify that the content of the secret has been properly propagated to the pod

bash-5.1$ ls /mnt/infra/custom-ca-certs

Use the following command to update CA bundle:

bash-5.1$ /app/bin/process_custom_ca_certs.sh

Adding more than one certificate to the CA bundle

To ensure that related certificates (such as a root CA and an intermediate CA) are added in the correct sequence within the CA bundle, they must be entered in the proper order. By doing so, Concert can correctly establish and maintain the desired certificate hierarchy.

Custom certificates are always appended at the end of Concert CA bundle, following the alphabetical ordering or their corresponding file name. The recommended approach would be to add a numbered prefix so that the order is clearly indicated:

$ ls /localstorage/volumes/infra/custom-ca-certs
00-server.crt
01-intermediate.crt
02-root.crt

Similarly, when dealing with Kubernetes secrets, it is essential to maintain consistent naming conventions for the keys within the data section.