Issue Resolution Core operator secret usage

Key rotation

Warning: The secret aiops-ir-core-model-secret contains an encryption key and a HMAC key that is used to protect sensitive data that is stored in CouchDB. Care must be taken when rotating these keys since data that is protected with previous keys cannot be recovered if the keys are lost.

Replacement values for keyvalue must be 16 bytes and in hexadecimal format (32 digits). Replacement values for hkeyvalue must be 32 bytes and in hexadecimal format (64 digits).

Each key has an associated identifier. Encrypted data is stored along with the identifier of the key that was used to protect it. The identifier must be changed when and only when the key is changed. When the operator creates a new key, it uses the current time as the identifier (for example 2022-06-13T10:58:10Z). This is a convention but other string formats may be used. If another format is used, there might be some restrictions on the characters that can be used. ASCII alphanumeric characters are recommended.

When rotating keys, all old (key, identifier) pairs must be stored in the oldhkeys and oldkeys properties so that data that is protected with the old keys can be used. The current key is always used when data is stored or modified. The following examples show the use of the old keys' properties.

1. Secret created by the operator.

   {
     "hkeyname": "2022-06-13T10:58:10Z",
     "hkeyvalue": "2905c99e41fd6594de9e8632382da185eec94cea545cbd96082552274a454d2b",
     "keyname": "2022-06-13T10:58:10Z",
     "keyvalue": "5cc068f45904c0ff998213941684790e"
   }
   

   
   

2. Secret after the first rotation.

   The identifiers myfirsthkey and myfirstkey were used. Time based could have been used.

   {
     "hkeyname": "myfirsthkey",
     "hkeyvalue": "7539eb64abdae09a500659ca0593140babd00046b175c3a83e3ed28f8d79daca",
     "oldhkeys": "{\"keys\":[{\"name\":\"2022-06-13T10:58:10Z\",\"value\":\"2905c99e41fd6594de9e8632382da185eec94cea545cbd96082552274a454d2b\"}]}",
     "keyname": "myfirstkey",
     "keyvalue": "211812c9750778c0175bca4a11cc1e3b",
     "oldkeys": "{\"keys\":[{\"name\":\"2022-06-13T10:58:10Z\",\"value\":\"5cc068f45904c0ff998213941684790e\"}]}"
   }
   

   
   

3. Secret after the second rotation.

   The identifiers mysecondhkey and mysecondkey were used. Time based could have been used.

   {
     "hkeyname": "mysecondhkey",
     "hkeyvalue": "07a290a68d6ff7eb45b7febbfd8c5079da5ee40c068f2e1bb87293c22c39b712",
     "oldhkeys": "{\"keys\":[{\"name\":\"2022-06-13T10:58:10Z\",\"value\":\"2905c99e41fd6594de9e8632382da185eec94cea545cbd96082552274a454d2b\"},{\"name\":\"myfirsthkey\",\"value\":\"7539eb64abdae09a500659ca0593140babd00046b175c3a83e3ed28f8d79daca\"}]}",
     "keyname": "mysecondkey",
     "keyvalue": "f01b36adba59c50841a1444cee8eb09e",
     "oldkeys": "{\"keys\":[{\"name\":\"2022-06-13T10:58:10Z\",\"value\":\"5cc068f45904c0ff998213941684790e\"},{\"name\":\"myfirstkey\",\"value\":\"211812c9750778c0175bca4a11cc1e3b\"}]}"
   }
   

   
   

   If the oldhkeys and oldkeys properties exist, they must hold the JSON representation of an object with a single property (the name does not matter, this example used keys) that has an array of objects. Each of the inner objects has a name and a value property holding the identifier and value of a previous key.

   oc delete pod -l app.kubernetes.io/component=rba-as && oc delete pod -l app.kubernetes.io/component=rba-rbs
   

ObjectServer passwords

Use the following steps to change the ObjectServer password. Ensure that you retain a copy of the original password until these steps are complete.

1. Run the following command to obtain the current passwords from the aiops-ir-core-omni-secret secret.

   For example:

   oc get secret aiops-ir-core-omni-secret -o json | jq -r '.data.OMNIBUS_PROBE_PASSWORD|@base64d'`
   oc get secret aiops-ir-core-omni-secret -o json | jq -r '.data.OMNIBUS_ROOT_PASSWORD|@base64d'`
   

2. Open a session on the primary ObjectServer pod.

   oc rsh aiops-ir-core-ncoprimary-0
   

3. Log in to the OMNIbus query tool nco_sql, and supply the root password when prompted.

   $NCHOME/omnibus/bin/nco_sql -u root -s AGG_P_C
   

4. If you want to change the root password, enter the following command.

   alter user 'root' set password '<NEWPASSWORD>';
   go
   

   Where <NEWPASSWORD> is the password that you want to set.

5. If you also want to change the probe password, enter the following command.

   alter user 'probe' set password '<PROBE_PASSWORD>';
   go
   

   Where <PROBE_PASSWORD> is the password that you want to set.

6. Test the new root password by entering quit to leave the SQL prompt, and then log in again and supply the new root password when prompted.

   $NCHOME/omnibus/bin/nco_sql -u root -s AGG_P_C
   

7. Exit the oc rsh session.

8. Wait for 60 seconds to ensure that the password updates have been transferred to the backup ObjectServer.

9. Remote shell into the backup ObjectServer, and test the new root password.

   oc rsh -c objserv aiops-ir-core-ncobackup-0`
   

10. Log in to the OMNIbus query tool nco_sql, and supply the root password when prompted.

    $NCHOME/omnibus/bin/nco_sql -u root -s AGG_B_C
    

11. Exit the oc rsh session.

12. Update the aiops-ir-core-omni-secret with one or more new passwords.

    For example,

    oc set data secret aiops-ir-core-omni-secret --from-literal=OMNIBUS_PROBE_PASSWORD=<PROBE_PASSWORD>
    oc set data secret aiops-ir-core-omni-secret --from-literal=OMNIBUS_ROOT_PASSWORD=<NEWPASSWORD>
    

    Where

    • <NEWPASSWORD> is the new root password
    • <PROBE_PASSWORD> is the new probe password

13. If you changed the root password, run the following command to restart aiops-ir-core-ncodl-if

    oc delete pod -l app.kubernetes.io/component=ncodl-if