Changing the secret for portal data and system tools

Replace the encryption key that generates the secret used for securing the database and system tools that are used by the portal subsystem.

Before you begin

If you are using a stand-alone, single data center deployment, you can update the encryption key with the PortalSecretRotation CR, as explained in Renewing the portal-ca with the PortalSecretRotation CR. The PortalSecretRotation CR is not supported for a two data center configuration or a Cloud Pak for Integration deployment, so you must update the key manually as explained in the following steps.

About this task

The portal subsystem uses a database plus several system tools to manage storage, hot backups, and cluster synchronization. The following services are all secured with the secret that is generated by the portal CR during installation:

MySQL root account for each portal site
Csync2 cluster synchronization service

Updating the portal-encryption-secret will make the following changes to your deployment:

Change the root MySQL password
Generate a new, site-specific MySQL password for each site
Generate a new csync2 cluster key
Re-encrypt all client_ids and client_secrets in the portal database

If you need to change the secret, the update is applied to all of the listed services. You can change the secret by generating a new value and restarting pods to pick up the new value. In a two data center deployment, you must manually copy the new secret to the other site.

Procedure

Generate a new secret and apply it to the deployment using the appropriate steps for your environment:

Single data center
Run the following command to generate a new secret and apply it to the deployment:
```
kubectl -n <namespace> patch secret portal-enc-key -p "{\"data\": {\"encryption_secret\":\"$(head -c74 /dev/urandom | base64 -w0 | tr '+/=' 'abc' | base64 -w0)\"}}"
```
where namespace is the namespace where you installed the portal subsystem.

The portal-enc-key was generated during installation and added to the CR as the encryptionSecret setting. When you generate a new secret, the value is updated automatically. If you saved the secret in a file after the Portal installation as recommended, be sure to update that file now.
Two data centers
1. Generate the new secret on DC1 by running the following command:
```
head -c74 /dev/urandom | base64 -w0 | tr '+/=' 'abc' | base64 -w0 > secret.txt
```
  The portal-enc-key was generated during installation and added to the CR as the encryptionSecret setting. When you generate a new secret, the value is updated automatically. If you saved the secret in a file after the Portal installation as recommended, be sure to update that file now.
2. Open a terminal to each DC in the same working directory where you saved secret.txt (or, copy the file to somewhere you can open a terminal to each DC).
3. In each terminal, verity that you can access secret.txt by typing the following command:
```
echo $(<secret.txt)
```
4. In each terminal, type the following command -- but do not press Enter yet:
```
kubectl -n <namespace> patch secret portal-enc-key -p "{\"data\": {\"encryption_secret\":\"$(<secret.txt)\"}}"
```
  where namespace is the namespace where you installed the portal subsystem.
5. Press Enter in both terminals at the same time (or as close in time as possible).
  There is a 5-minute time-out that starts as soon as any of the pods has the new secret mounted by Kubernetes. All pods will get the secret at slightly different times, but within about a minute of each other. If the 5-minute time-out expires before all pods across both DCs have the new secret, then the pods will rotate the new secret in at different times, resulting in errors in the logs and some down time for various pods, until they manage to rotate the new secret into use and recover.

Results

When pods restart, the new secret is used to secure the services. The portal-www admin logs display messages as the new secret is used to re-encrypt the database. The following example shows sample log messages when the updated secret is applied to the database:

[   upgrade stdout]  2872 888ec0:8cb2cc:73add2 2020-08-06 13:16:21: NEW k8s encryption key detected on this system! Reencrypting portal data
[   upgrade stderr]  2954 888ec0:73add2:7fce1b 2020-08-06 13:16:21: refresh_encryption_key: Command Started: /opt/ibm/bin/refresh_encryption_key FEgX8F5MXtwvoJ8Wpxw5DB/6MKK/qPKfRt5FaId3jy5IjbpXL8wi3/33q3KOySh8/zhl1rVNFZTlzRnVfkGheLr9Le/Pb9jq
[   upgrade stdout]  2872 888ec0:8cb2cc:73add2 2020-08-06 13:16:21: No portal service exists! Simply overriding the key
[   upgrade stdout]  2872 888ec0:8cb2cc:73add2 2020-08-06 13:16:21: New key installed (FEgX8F5MXtwvoJ8Wpxw5DB/6MKK/qPKfRt5FaId3jy5IjbpXL8wi3/33q3KOySh8/zhl1rVNFZTlzRnVfkGheLr9Le/Pb9jq)
[   upgrade stderr]  2954 888ec0:73add2:7fce1b 2020-08-06 13:16:21: refresh_encryption_key: Command Finished: /opt/ibm/bin/refresh_encryption_key FEgX8F5MXtwvoJ8Wpxw5DB/6MKK/qPKfRt5FaId3jy5IjbpXL8wi3/33q3KOySh8/zhl1rVNFZTlzRnVfkGheLr9Le/Pb9jq with RC 0
[   upgrade stderr]  2872 888ec0:8cb2cc:73add2 2020-08-06 13:16:21: Command Finished: /opt/ibm/bin/sysupgrade with RC 0

If there is a portal service registered and developer portal sites installed, then this task re-encrypts that data as well. When the site encryption is complete, the log displays a "Command Finished" message for the refresh_encryption_key command, as shown in the following example:

Command Finished: /opt/ibm/bin/refresh_encryption_key <KEY> with RC 0