Kubernetes secret synchronization

Procedure

• Standard procedure (During initial installation)

  If Netcool® Operations Insight® is not installed in the backup cluster, then do the following steps to ensure that a single set of encryption keys is used by both the clusters. If both the clusters are installed, then you must follow the process in the alternative procedure section.

  1. Install Netcool Operations Insight on the primary cluster.
  2. Run the following command to read the data part of the secret from the primary cluster and store it in a JSON file:

     oc get secret $RELEASENAME-cem-model-secret -o=jsonpath='{.data}' > cem-model-secret-data.json 

     Example output of the cat cem-model-secret-data.json command:

     {"hkeyname":"AAA","hkeyvalue":"BBB","keyname":"XXX","keyvalue":"YYY"}

  3. Copy the cem-model-secret-data.json file to the backup cluster. Run the following command to create the secret on the backup cluster:

     oc create secret generic $RELEASENAME-cem-model-secret 
     oc patch secret $RELEASENAME-cem-model-secret -p '{"data": '$(cat cem-model-secret-data.json)'}'

  4. Install Netcool Operations Insight in the backup cluster.

• Alternative procedure (After initial installation)
  Important: Ensure that you have jq 1.6 installed. You can install jq through operating system package managers, or download from https://stedolan.github.io/jq/.
  If both the primary and backup clusters are installed, then do the following steps to synchronize the encryption keys that are used by both the clusters.

  Note: You do not need to follow this procedure if you completed the steps in the standard procedure section.
  1. Save the model secrets from each cluster into separate files.
     1. On the primary cluster, run the following command:

        oc get secret $RELEASENAME-cem-model-secret -o json > primary-model-secret.json

     2. On the backup cluster, run the following command:

        oc get secret $RELEASENAME-cem-model-secret -o json > backup-model-secret.json

     3. Copy both the model-secret.json files into the same location onto the same cluster.
  2. Save the following script into merge-model-secrets.sh into the same location where you have saved the model-secret.json files.

     jq -s '[.[].data|[to_entries[]|.value|=@base64d]|from_entries]|
         {data:{
             hkeyname:  (.[0].hkeyname|@base64),
             hkeyvalue: (.[0].hkeyvalue|@base64),
             oldhkeys:  (
                 {keys:(
                     [
                         (.[1:][]|{key:.hkeyname,value:.hkeyvalue}),
                         (.[].oldhkeys|select(.)|fromjson|to_entries[].value[]|{key:.name,value})
                     ] | from_entries | to_entries | [.[]|{name:.key,value}]
                 )} | @base64),
             keyname:   (.[0].keyname|@base64),
             keyvalue:  (.[0].keyvalue|@base64),
             oldkeys:  (
                 {keys:(
                     [
                         (.[1:][]|{key:.keyname,value:.keyvalue}),
                         (.[].oldkeys|select(.)|fromjson|to_entries[].value[]|{key:.name,value})
                     ] | from_entries | to_entries | [.[]|{name:.key,value}]
                 )} | @base64)
         }}' "$@"

  3. Copy primary-model-secret.json and backup-model-secret.json to the location where you have saved the script. Then, run the following commands:

     chmod u+x merge-model-secrets.sh
     sh merge-model-secrets.sh primary-model-secret.json backup-model-secret.json > model-secret-patch.json

  4. Copy model-secret-patch.json to both the clusters, and then run the following command on each cluster:

     oc patch secret $RELEASENAME-cem-model-secret --patch-file model-secret-patch.json

  5. Delete the RBA pods on the backup cluster and allow Kubernetes to redeploy the pods. Deleting the RBA pods on the backup cluster ensures that the new secrets are used by the running processes.
     CAUTION:
     This action causes a temporary outage until Kubernetes starts the new pods again.

     oc delete pod -l component=rba-as
     oc delete pod -l component=rba-rbs
     oc delete pod -l component=scheduling-ui
     oc delete pod -l component=brokers
     oc delete pod -l component=cem-users
     oc delete pod -l component=channelservices
     oc delete pod -l component=eventpreprocessor
     oc delete pod -l component=incidentprocessor
     oc delete pod -l component=integration-controller
     oc delete pod -l component=normalizer
     oc delete pod -l component=notificationprocessor

  6. Optional: Delete the couchdb pod on the backup cluster by running the following command:

     oc debug --image=ubi8 $(oc get pods -l component=rba-rbs | awk '{ print $1 }' | tail -n 1) -- /bin/bash -c 'IFS=","; BASEURL="${RBA_DATABASE_CLUSTER_PROTOCOL}://${RBA_DATABASE_CLUSTER_USER}:${RBA_DATABASE_CLUSTER_PASSWORD}@${RBA_DATABASE_CLUSTER_HOST}:${RBA_DATABASE_CLUSTER_PORT}"; for i in $(curl $BASEURL/_all_dbs -s); do DATABASE=$(echo $i | sed "s/\[//g" | sed "s/\"//g" | sed "s/\]//g"); if [[ $DATABASE != _* ]]; then echo -n "Deleting database ${DATABASE}..."; curl -X DELETE $BASEURL/$DATABASE -s; echo "done"; fi; done'

     Then, restart the pods:

     oc delete pod -l component=rba-as
     oc delete pod -l component=rba-rbs
     oc delete pod -l component=scheduling-ui
     oc delete pod -l component=brokers
     oc delete pod -l component=cem-users
     oc delete pod -l component=channelservices
     oc delete pod -l component=eventpreprocessor
     oc delete pod -l component=incidentprocessor
     oc delete pod -l component=integration-controller
     oc delete pod -l component=normalizer
     oc delete pod -l component=notificationprocessor