Refreshing certificates

You can refresh certificates that are created by the installer after you install IBM Cloud Private.

Required user type or access level: Cluster administrator or team administrator

You can refresh certificates for the following components:

Component	Directory
Root CA	`cluster/cfc-certs/root-ca`
Etcd	`cluster/cfc-certs/etcd`
Front-Proxy	`cluster/cfc-certs/front-proxy`
Kubernetes	`cluster/cfc-certs/kubernetes`
IPSec	`cluster/cfc-certs/ipsec`
Helm	`cluster/cfc-certs/helm`

Before you begin

Verify that your IBM Cloud Private cluster is running. You will see failures if the nodes are not up and running.

Only refresh the certificate that is expired. Do not refresh the root-ca certificate if it is not expired. Check which certificates are expired by running the following command:

cd <installation_directory>/cluster/ cfc-certs
find . -type f -name "*.crt" -o -name "*.pem" | grep -v 'key.pem$' | xargs -L 1 -t -i bash -c 'openssl x509 -noout -text -in {}| egrep "Before|After"'

After the installation, if you updated the SSH key for the cluster nodes that is used to communicate with the other cluster nodes, replace the <installation_directory>/cluster/ssh_key file with the private key file. For more information, see Sharing SSH keys among cluster nodes.

You can compare the <installation_directory>/cluster/ssh_key with ~/.ssh/id_rsa and copy the file if they are different:
```
sudo cp ~/.ssh/id_rsa <installation_directory>/cluster/ssh_key
```
Complete the following steps to back up the existing certificates:
1. Log in to the boot node.
2. Change to the cluster directory and create a backup of all the existing certificates by running the following commands:
```
cd <installation_directory>/cluster/
cp -r cfc-certs cfc-certs.bak
```
  Note: When you run the command to refresh a certificate, the related management services might be unavailable for a short duration of time. The command has no impact on the applications in your cluster.

Refreshing root CA certificates

Update the old Certificate Authority (CA): Delete the old CA cert ca.crt by running the following command:
```
   rm -rf <cluster_dir>/cfc-certs/root-ca
```

Delete the certificates and keys that are related with the old CA. For example:

   rm -rf <cluster_dir>/cfc-certs/kubernetes
   rm -rf <cluster_dir>/cfc-certs/helm
   rm -rf <cluster_dir>/cfc-certs/ipsec

Run the following command to refresh the root-ca certificates:

For IBM Cloud Private:

For Linux, run the following command:

sudo docker run --net=host -t -e LICENSE=accept \
-v "$(pwd)":/installer/cluster ibmcom/icp-inception-amd64:3.2.1-ee \
replace-certificates --tags "root-ca-certs"

For Linux® on Power® (ppc64le), run the following command:

sudo docker run --net=host -t -e LICENSE=accept \
-v "$(pwd)":/installer/cluster ibmcom/icp-inception-ppc64le:3.2.1-ee \
replace-certificates --tags "root-ca-certs"

For Linux® on IBM® Z and LinuxONE, run the following command:
```
sudo docker run --net=host -t -e LICENSE=accept \
-v "$(pwd)":/installer/cluster ibmcom/icp-inception-s390x:3.2.1-ee \
replace-certificates --tags "root-ca-certs"
```
Note: When you refresh the Root CA, all certificates that are signed by the Root CA are automatically refreshed. These certificates include the Kubernetes, IPSec, and Helm.

Manually restart the services by reloading the them. All default tokens and related services must reload. For more information, see Reloading services.
Manually recreate the image pull secret. For more information, see Recreate image pull secret
Refresh all cert-manager certificates that use the CA. For more information, see Refreshing Cert-Manager Certificates.
Restart Docker
```
 sudo systemctl restart docker
```
Run the following command to restart Kubelet on your master node.
```
 sudo systemctl restart kubelet
```

Refreshing etcd certificates

Delete the old certificates by running the following command:
```
   rm -rf <cluster_dir>/cfc-certs/etcd
```

Run the following command to replace the etcd certificates:

For Linux, run the following command:

sudo docker run --net=host -t -e LICENSE=accept \
-v "$(pwd)":/installer/cluster ibmcom/icp-inception-amd64:3.2.1-ee \
replace-certificates --tags "etcd-certs"

For Linux® on Power® (ppc64le), run the following command:

sudo docker run --net=host -t -e LICENSE=accept \
-v "$(pwd)":/installer/cluster ibmcom/icp-inception-ppc64le:3.2.1-ee \
replace-certificates --tags "etcd-certs"

For Linux® on IBM® Z and LinuxONE, run the following command:

sudo docker run --net=host -t -e LICENSE=accept \
-v "$(pwd)":/installer/cluster ibmcom/icp-inception-s390x:3.2.1-ee \
replace-certificates --tags "etcd-certs"

Refreshing front proxy certificates

Delete the old certificates by running the following command:
```
 rm -rf <cluster_dir>/cfc-certs/front
```

Run the following command to replace the front proxy certificates:

For Linux, run the following command:

sudo docker run --net=host -t -e LICENSE=accept \
-v "$(pwd)":/installer/cluster ibmcom/icp-inception-amd64:3.2.1-ee \
replace-certificates --tags "front-proxy-certs"

For Linux® on Power® (ppc64le), run the following command:

sudo docker run --net=host -t -e LICENSE=accept \
-v "$(pwd)":/installer/cluster ibmcom/icp-inception-ppc64le:3.2.1-ee \
replace-certificates --tags "front-proxy-certs"

For Linux® on IBM® Z and LinuxONE, run the following command:

sudo docker run --net=host -t -e LICENSE=accept \
-v "$(pwd)":/installer/cluster ibmcom/icp-inception-s390x:3.2.1-ee \
replace-certificates --tags "front-proxy-certs"

Refreshing kubernetes certificates

Delete the old certificates by running the following command:
```
 rm -rf <cluster_dir>/cfc-certs/kubernetes
```

Run the following command to replace the kubernetes certificates:

For Linux, run the following command:

sudo docker run --net=host -t -e LICENSE=accept \
-v "$(pwd)":/installer/cluster ibmcom/icp-inception-amd64:3.2.1-ee \
replace-certificates --tags "kubernetes-certs"

For Linux® on Power® (ppc64le), run the following command:

sudo docker run --net=host -t -e LICENSE=accept \
-v "$(pwd)":/installer/cluster ibmcom/icp-inception-ppc64le:3.2.1-ee \
replace-certificates --tags "kubernetes-certs"

For Linux® on IBM® Z and LinuxONE, run the following command:

sudo docker run --net=host -t -e LICENSE=accept \
-v "$(pwd)":/installer/cluster ibmcom/icp-inception-s390x:3.2.1-ee \
replace-certificates --tags "kubernetes-certs"

Refreshing ipsec certificates

Delete the old certificates by running the following command:
```
 rm -rf <cluster_dir>/cfc-certs/ipsec
```

Run the following command to replace the ipsec certificates:

For Linux, run the following command:

sudo docker run --net=host -t -e LICENSE=accept \
-v "$(pwd)":/installer/cluster ibmcom/icp-inception-amd64:3.2.1-ee \
replace-certificates --tags "ipsec-certs"

For Linux® on Power® (ppc64le), run the following command:

sudo docker run --net=host -t -e LICENSE=accept \
-v "$(pwd)":/installer/cluster ibmcom/icp-inception-ppc64le:3.2.1-ee \
replace-certificates --tags "ipsec-certs"

For Linux® on IBM® Z and LinuxONE, run the following command:

sudo docker run --net=host -t -e LICENSE=accept \
-v "$(pwd)":/installer/cluster ibmcom/icp-inception-s390x:3.2.1-ee \
replace-certificates --tags "ipsec-certs"

Refreshing helm certificates

Delete the old certificates by running the following command:
```
 rm -rf <cluster_dir>/cfc-certs/helm
```

Run the following command to replace the helm certificates:

For Linux, run the following command:

sudo docker run --net=host -t -e LICENSE=accept \
-v "$(pwd)":/installer/cluster ibmcom/icp-inception-amd64:3.2.1-ee \
replace-certificates --tags "helm-certs"

For Linux® on Power® (ppc64le), run the following command:

sudo docker run --net=host -t -e LICENSE=accept \
-v "$(pwd)":/installer/cluster ibmcom/icp-inception-ppc64le:3.2.1-ee \
replace-certificates --tags "helm-certs"

For Linux® on IBM® Z and LinuxONE, run the following command:

sudo docker run --net=host -t -e LICENSE=accept \
-v "$(pwd)":/installer/cluster ibmcom/icp-inception-s390x:3.2.1-ee \
replace-certificates --tags "helm-certs"

Reloading services

Reloading default token

After Root CA is replaced, you must delete the default token from all namespaces, and restart related services. You can use the following code to delete the default token from all namespaces and restart the related services.

kubectl get secret --no-headers --all-namespaces -o=custom-columns=NAME:.metadata.name,NAMESPACE:.metadata.namespace | grep default-token | while read token; do
    secret_name=$(echo $token | awk '{print $1}')
    secret_namespace=$(echo $token | awk '{print $2}')
    echo "-----------------------------------------------------------------------"
    echo "|                      Token: ${secret_name}"
    echo "|                  Namespace: ${secret_namespace}"
    echo "-----------------------------------------------------------------------"
    echo "Deleteing default token ..."
    kubectl -n ${secret_namespace} delete secret ${secret_name} &>/dev/null
    echo "Reloading services ..."
    kubectl -n ${secret_namespace} get po --field-selector=status.phase!=Completed,status.phase!=Succeeded,status.phase!=Unknown --no-headers -o=custom-columns=NAME:.metadata.name | while read pod; do
    secret_used=$(kubectl -n ${secret_namespace} get po ${pod} -oyaml | egrep 'secretName: default-token|secretName: calico-node-token' &>/dev/null || echo no && echo yes)
    if [[ "$secret_used" == "yes" ]]; then
        echo "    - Restarting pod ${pod} ..."
        kubectl -n ${secret_namespace} delete po ${pod} --grace-period=0 --force &>/dev/null
    fi
    done
    echo
done

Recreate image pull secret

For Linux, run the following command:

sudo docker run --net=host -t -e LICENSE=accept \
-v "$(pwd)":/installer/cluster ibmcom/icp-inception-amd64:3.2.1-ee \
image-pull-secret

For Linux® on Power® (ppc64le), run the following command:

sudo docker run --net=host -t -e LICENSE=accept \
-v "$(pwd)":/installer/cluster ibmcom/icp-inception-ppc64le:3.2.1-ee \
image-pull-secret

For Linux® on IBM® Z and LinuxONE, run the following command:

sudo docker run --net=host -t -e LICENSE=accept \
-v "$(pwd)":/installer/cluster ibmcom/icp-inception-s390x:3.2.1-ee \
image-pull-secret

Refreshing cert-manager certificates

After the Root CA is replaced, you must refresh all certificates that are signed by the Root CA, and restart the services that use those certificates. Cert-manager creates the default ClusterIssuer from the Root CA, so all of the certificates issued by cert-manager and signed by the default ClusterIssuer must also be refreshed.

The following code deletes the Kubernetes Secrets associated with each cert-manager certificate to refresh the certificate and restart the services that use the certificate.

kubectl get cert --all-namespaces -o custom-columns=:spec.secretName,:metadata.namespace --no-headers -l certmanager.k8s.io/issuer-name=icp-ca-issuer,certmanager.k8s.io/issuer-kind=ClusterIssuer | while read secret ; do
    name=$(echo $secret | awk '{print $1}')
    namespace=$(echo $secret | awk '{print $2}')
    kubectl delete secret $name -n $namespace
    echo "Secret $name was deleted."
done