Managing custom certificates (for VM and Kubernetes deployments)

Edit online

You can use your own trusted custom certificates so Concert can connect to your external connections (integrations). You can import custom trusted certificates for SSL/TLS verification by adding them to a known location. This helps ensure consistent handling of certificates across different environments. Currently this is supported for VM and Kubernetes deployments only.

In virtual machine (VM) deployments, there is a dedicated folder, while in Kubernetes, it is a specific secret.

Draft comment: erin.pelkey@ibm.com
This is insufficient. What dedicated folder are we referring to for VM deployments? Are we trying to say that the user has to put the certificates in a specific folder? For Kubernetes, what do we mean by "it is a specific secret"?

Concert uses an internal certificate authority (CA) bundle with all trusted certificates.

Draft comment: erin.pelkey@ibm.com
It seems like we're missing the second half of this sentence. 'Concert uses an internal certificate authority (CA) bundle with all trusted certificates...' in order to do what, exactly? Briefly, we need to clarify what these certificates are for or how they're used.

During certain operations, such as updates, Concert might regenerate its CA bundle.

Draft comment: erin.pelkey@ibm.com
What 'updates' are we referring to here? Updates to the software version? Something else?

Therefore, any certificates that are added to the common locations should persist there. Later, if these certificates are removed, Concert may interpret this as you no longer require those certificates to be part of the CA bundle.

Draft comment: erin.pelkey@ibm.com
We haven't explained where or how these certificates are used. What happens if you remove the certificates? Which transactions would be unverified?

Attention: Adding invalid or corrupted certificates to Concert's CA bundle can have adverse effects. It may render the CA bundle unusable, leading to unexpected behavior, such as constant restarts. Always exercise caution while updating the CA bundle to avoid such issues.

As a best practice, it is highly recommended that you create a backup of the private key, self-signed certificate, and CA bundle before updating to ensure that both Concert's internal communications and the external connections (integrations) to which they correspond are secure.

Backing up and restoring certificates for VM deployments

Export some variables for other commands to use. Adjust them to match your environment:

$ export CONCERT_TLS_DIR="<path_to_concert_std>/localstorage/volumes/infra/tls"
$ export BACKUP_FILE="${HOME}/concert-tls-backup.tar.gz"

To perform a backup of an existing key, certificate, and bundle, use the following command:

$ tar cvzf "${HOME}/concert-tls-backup.tar.gz" -C "${CONCERT_TLS_DIR}" tls.key tls.crt tls-ca-bundle.pem
tls.key
tls.crt
tls-ca-bundle.pem

To restore the content of the backup file, run the following command:

$ tar xzvf "${HOME}/concert-tls-backup.tar.gz" -C "${CONCERT_TLS_DIR}"
tls.key
tls.crt
tls-ca-bundle.pem

Triggering a CA bundle update on a VM deployment

Draft comment: erin.pelkey@ibm.com
Updated the title as the previous title "VM deployment and triggering CA bundle update" was unclear.

Copy any certificate that you want to add to Concert's CA bundle to localstorage/volumes/infra/custom-ca-certs folder. If you want to remove previously added certificates from the CA bundle, delete them from localstorage/volumes/infra/custom-ca-certs.

After modifying the contents of localstorage/volumes/infra/custom-ca-certs folder, run the cert-mgmt script. The cert-mgmt script is part of ibm-concert-std.tgz package that is published on Concert public GitHub.

Note: Refer to the VM installation section for steps to download the script ibm-concert-std.tgz.

After you run the script, restart Concert services:

<concert-install-directory>/bin/stop
 <concert-install-directory>/bin/start

Attention: Restart the Concert services to apply the changes.

Backup and restore of certificates for Kubernetes deployment

Export the namespace where Concert is deployed, and the name of the local file that will be used for backup for other commands to use.

$ export CONCERT_NS=<your_concert_namespace>
$ export CONCERT_TLS_SECRET="${HOME}/app-cfg-internal-tls.yaml"

Note: You must be logged in to your cluster, and your user account must have the required permissions to read Kubernetes secrets.

Use the following command to save Concert's TLS secret to a file:

$ kubectl -n "${CONCERT_NS}" get secret app-cfg-internal-tls -oyaml > "${CONCERT_TLS_SECRET}"

To restore the previously saved secret, use this command:

$ kubectl -n "${CONCERT_NS}" apply -f "${CONCERT_TLS_SECRET}"

Attention: During certificate bundle loading or update if you get any error follow the backup steps listed and take backup of your certificates. Comparing historical record of existing certificate may help diagnose the error.

Updating the certificate secret on Kubernetes deployments

Draft comment: erin.pelkey@ibm.com
Updated the title as the previous title "Kubernetes deployment" was unclear.

Use the following command to export the name of the namespace where Concert is deployed:

$ export CONCERT_NS=<concert's namespace>

Ensure that the secret app-cfg-custom-ca-certs exists in the namespace where Concert is deployed. If it does not exist, create it by using the following command:

$ kubectl --namespace "${CONCERT_NS}" create secret generic app-cfg-custom-ca-certs

Edit the secret to add any certificate that you want to add to the secret's data section. The key of each entry in this section is the name of the certificate file, and the value is the content of the certificate, encoded in base64 format.

To modify the secret, you can use the following command:

$ kubectl --namespace "${CONCERT_NS}" edit secret app-cfg-custom-ca-certs

Triggering a CA bundle update on Kubernetes

Draft comment: erin.pelkey@ibm.com
What is the difference between this step and the previous one? The order of operations is unclear.

After the update is complete, log in to the management pod:

$ MGMT_POD=$(kubectl --namespace "${CONCERT_NS}" get pods -o=name | grep 'roja-mgmt-' | head -n1)
$ kubectl --namespace "${CONCERT_NS}" exec -it "${MGMT_POD}" -- bash

Now, you can access the pod and run the following command to verify that the content of the secret has been properly propagated to the pod

bash-5.1$ ls /mnt/infra/custom-ca-certs

Use the following command to update CA bundle:

bash-5.1$ /app/bin/process_custom_ca_certs.sh

Once CA bundle is updated, you need to restart Concert services:

kubectl rollout restart deployment/concert-solis-gw

Adding more than one certificate to the CA bundle

Draft comment: erin.pelkey@ibm.com
Is this section related to VM or Kubernetes deployments? Is this a required step or just additional information? I think the organization of this topic could be improved, or we should break sections out into separate topics.

To ensure that related certificates (such as a root CA and an intermediate CA) are added in the correct sequence within the CA bundle, they must be entered in the proper order. By doing so, Concert can correctly establish and maintain the desired certificate hierarchy.

Custom certificates are always appended at the end of Concert CA bundle, following the alphabetical ordering or their corresponding file name. The recommended approach would be to add a numbered prefix so that the order is clearly indicated:

$ ls /localstorage/volumes/infra/custom-ca-certs
00-server.crt
01-intermediate.crt
02-root.crt

Similarly, when dealing with Kubernetes secrets, it is essential to maintain consistent naming conventions for the keys within the data section.