How to enable at rest file encryption with opsc_system_key

Problem

Summary
This article will describe the process by which to apply encryption to opscenter and agent configuration files in order to hash the passwords at rest.

Applies To
All OpsCenter versions

Solution
OpsCenter

1) First from the opscenter machine, find the opscenter_system_key_tool and create an encryption key:

bin/opscenter_system_key_tool create ECB 128

2) That key will get created wherever opscenterd.conf is located, typically in /etc/opscenterd/

3) Set the following in opscenterd.conf:

[security]
config_encryption_active = True
config_encryption_key_name = opsc_system_key
config_encryption_key_path = /etc/opscenterd

4) Use the key to encrypt the opscenterd.conf and clustername.conf values:

/usr/share/opscenter/bin/opscenter_system_key_tool value

You will then be prompted to enter the value to encrypt. Thus, the above is as-is, you don't need to specify a value where it states value.

5) Replace those values into the opscenterd.conf and clustername.conf files with the new encrypted value.

6) Restart opscenterd and the service should come without any errors.

Agents
1) Copy the opsc_system_key to all of the agent nodes

2) On the agents, put the opsc_system_key in the same location as address.yaml - required

3) Make sure the owner and permissions are correct for the opsc_system_key

4) Set the following in address.yaml:

config_encryption_active: true
config_encryption_key_name: opsc_system_key
config_encryption_key_path: /var/lib/datastax-agent/conf

8) Encrypt the values manually from the opscenter node using:

/usr/share/opscenter/bin/opscenter_system_key_tool value

9) Now, amend the values in address.yaml with the encrypted values.

10) Restart the datastax-agent.


Possible Issues
I have at times seen issues with SSL configuration errors like keystore has been tampered with:

ERROR [async-dispatch-4] 2021-09-14 11:37:12,827Z Unexpected error connecting to Cassandra, retrying soon.
 java.lang.Exception: Error loading keystore /var/lib/cassandra/ssl/my-keystore.jks
at opsagent.cassandra_client$load_keystore.invokeStatic(cassandra_client.clj:47)
at opsagent.cassandra_client$load_keystore.invoke(cassandra_client.clj:36)
at opsagent.cassandra_client$get_key_managers.invokeStatic(cassandra_client.clj:65)
at opsagent.cassandra_client$get_key_managers.invoke(cassandra_client.clj:62)
at opsagent.cassandra_client$build_ssl_options.invokeStatic(cassandra_client.clj:86)
at opsagent.cassandra_client$build_ssl_options.invoke(cassandra_client.clj:82)
at opsagent.cassandra_client$build_cluster.invokeStatic(cassandra_client.clj:153)
at opsagent.cassandra_client$build_cluster.invoke(cassandra_client.clj:128)
at opsagent.cassandra_client$build_cassandra_connection.invokeStatic(cassandra_client.clj:173)
at opsagent.cassandra_client$build_cassandra_connection.doInvoke(cassandra_client.clj:167)
at clojure.lang.RestFn.invoke(RestFn.java:2742)
at opsagent.cassandra.MonitoredDatabase.start(cassandra.clj:576)
at com.stuartsierra.component$fn__1777$G__1771__1779.invoke(component.clj:4)
at com.stuartsierra.component$fn__1777$G__1770__1782.invoke(component.clj:4)
at clojure.lang.AFn.applyToHelper(AFn.java:154)
at clojure.lang.AFn.applyTo(AFn.java:144)
at clojure.lang.Var.applyTo(Var.java:705)
at clojure.core$apply.invokeStatic(core.clj:667)
at clojure.core$apply.invoke(core.clj:660)
at com.stuartsierra.component$try_action.invokeStatic(component.clj:116)
at com.stuartsierra.component$try_action.invoke(component.clj:115)
at clojure.lang.Var.invoke(Var.java:406)
at opsagent.config_service$update_system$fn__27020.invoke(config_service.clj:223)
at clojure.lang.ArraySeq.reduce(ArraySeq.java:116)
at clojure.core$reduce.invokeStatic(core.clj:6827)
at clojure.core$reduce.invoke(core.clj:6810)
at opsagent.config_service$update_system.invokeStatic(config_service.clj:217)
at opsagent.config_service$update_system.doInvoke(config_service.clj:213)
at clojure.lang.RestFn.invoke(RestFn.java:425)
at opsagent.config_service$start_system_BANG_.invokeStatic(config_service.clj:243)
at opsagent.config_service$start_system_BANG_.invoke(config_service.clj:236)
at opsagent.config_service$fn__27134$fn__27135$state_machine__6717__auto____27136$fn__27138.invoke(config_service.clj:266)
at opsagent.config_service$fn__27134$fn__27135$state_machine__6717__auto____27136.invoke(config_service.clj:266)
at clojure.core.async.impl.ioc_macros$run_state_machine.invokeStatic(ioc_macros.clj:973)
at clojure.core.async.impl.ioc_macros$run_state_machine.invoke(ioc_macros.clj:972)
at clojure.core.async.impl.ioc_macros$run_state_machine_wrapped.invokeStatic(ioc_macros.clj:977)
at clojure.core.async.impl.ioc_macros$run_state_machine_wrapped.invoke(ioc_macros.clj:975)
at clojure.core.async$ioc_alts_BANG_$fn__6932.invoke(async.clj:384)
at clojure.core.async$do_alts$fn__6872$fn__6875.invoke(async.clj:253)
at clojure.core.async.impl.channels.ManyToManyChannel$fn__1855.invoke(channels.clj:265)
at clojure.lang.AFn.run(AFn.java:22)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.io.IOException: Keystore was tampered with, or password was incorrect
at sun.security.provider.JavaKeyStore.engineLoad(JavaKeyStore.java:792)
at sun.security.provider.JavaKeyStore$JKS.engineLoad(JavaKeyStore.java:57)
at sun.security.provider.KeyStoreDelegator.engineLoad(KeyStoreDelegator.java:224)
at sun.security.provider.JavaKeyStore$DualFormatJKS.engineLoad(JavaKeyStore.java:71)
at java.security.KeyStore.load(KeyStore.java:1445)
at opsagent.cassandra_client$load_keystore.invokeStatic(cassandra_client.clj:43)
... 43 more
Caused by: java.security.UnrecoverableKeyException: Password verification failed
at sun.security.provider.JavaKeyStore.engineLoad(JavaKeyStore.java:790)
... 48 more

The issue wasn't actually a problem with the keystore. The jmx_password wasn't encrypted. Thus, make sure all of the passwords are encrypted, otherwise, you may see odd unrelated errors.