Issues with the Cassandra pods on a cluster

Learn to resolve the issues with the Cassandra pods on a cluster.

Problem

Issues with the Cassandra pods on a cluster. Error message for Error fetching policies is displayed while viewing the list of user-defined scope policies on the WebGUI.

Seasonal events and related events training fails. cloud native analytics training fails on loading the local data with the following error:
Caused by: com.ibm.itsm.analytics.runtime.api.Node$NodeExecutionException: Event dataset is empty

Resolution

Following are the steps to resolve the issues with the Cassandra on a cluster:

  1. Run the following command to check the status of the Cassandra stateful sets.
    oc get sts | grep cassandra 
  2. If the pods are not ready (for example, 1/1, 3/3, and so on), then try scaling down all the stateful sets to 0 and scale them back to the original value.

    Following is the example to do the scaling of the sets:

    • Make a note of the current value of the Cassandra pods.
    • Scale down the pods in a similar way to the following example:
      oc scale sts <release_name>-cassandra --replicas=0 
    • Verify that all the Cassandra pods are stopped.
      oc get pods|grep cassandra
    • Scale the pods back to their original value in a similar way to the following example:
      oc scale sts <release_name>-cassandra --replicas=3