Manually restarting Guardium Insights

Restart the Guardium Insights pods after a shut down.

Before you begin

Log in to your Guardium Insights cluster. From there, switch to the Guardium Insights OpenShift® namespace by using the oc project command. For example:
oc project <guardium_insights_namespace>

This task requires the use of Guardium Insights support scripts. To learn how to access these scripts, see this topic.

Procedure

  1. When you shut down Guardium Insights, you scaled down the Kafka pod. When you complete this step, you set replicas: 1 to replicas: 0. Set the value back to replicas: 1:
    1. Switch to the ibm-common-services namespace:
      oc project ibm-common-services
    2. To search for the events operator, issue this command:
      oc get pods
      In the returned output, locate the operator, for example:
      ibm-events-operator-v3.7.1               1/1     1            1           112d
    3. Edit the replicas and set the operator to running by issuing this command:
      oc edit deployment ibm-events-operator-v3.7.1
    4. Scroll to spec.replicas and set it to 1 (one).
      Locate the following part of the file:
      uid: 0481226f-52ea-4d39-aee0-bea37ab3fab6
      spec:
        progressDeadlineSeconds: 600
        replicas: 0
        revisionHistoryLimit: 1
        selector:

      And set replicas: 0 to replicas: 1.

    5. Switch back to the Guardium Insights OpenShift namespace:
      oc project <guardium_insights_namespace>
    6. To verify that the Kafka pods are running, run the oc get sts command and verify that the Zookeeper and Kafka stateful sets are running.
  2. When you shut down Guardium Insights, you scaled down most of the replicas in your Guardium Insights OpenShift namespace. When you completed this step, you set the number of replicas for each stateful set and noted the results. For each stateful set, return the number of replicas to the previous number by running the following command:
    oc scale sts <statefulset> --replicas=<previous_replica_number>

    where <statefulset> is each of the preceding stateful sets and <previous_replica_number> is the previous number of replicas that you determined in this step. Using the previous example, you can run the following commands:

    oc scale sts bitnami-zookeeper --replicas=3
    oc scale sts staging-ibm-redis-sentinel --replicas=3
    oc scale sts staging-ibm-redis-server --replicas=2
    oc scale sts staging-kafka --replicas=3
    oc scale sts staging-mongodb-arbiter --replicas=1
    oc scale sts staging-mongodb-primary --replicas=1
    oc scale sts staging-mongodb-secondary --replicas=1
  3. When you shut down Guardium Insights, you renamed the database-db2wh node in this step. Undo this change:
    1. Run the following command:
      oc edit node <database-db2wh_node>

      where <database-db2wh_node> is the ICP4Data node (for example, worker2.myenv.ibm.com).

    2. Scroll to labels and then change the temporary value that you set back to icp4data.
  4. After you change the database-db2wh node, the new pod starts with Guardium Insights services running and the db2u statefulset moves to a Running state. Verify that the Db2® Warehouse pod is running:
    1. Exec into the Db2 Warehouse pod by running the following command:
      oc exec -ti staging-ibm-db2u-db2u-0 /bin/bash
    2. Run the following command to verify that the built-in High Availability (HA) is enabled. The returned output is similar to HA Management is RUNNING.
      sudo wvcli system status
  5. To start the Guardium Insights services again, run the following command:
    ./scripts/startorstopGuardiumInsights.sh start
  6. To verify that all Guardium Insights pods are running, run the following command:
    oc get pods -lapp.kubernetes.io/instance=<guardium_insights_namespace>

    A successful output shows all pods in a Running state.

  7. Resume all cron jobs by running the following command:
    oc patch `oc get cronjob -lapp.kubernetes.io/instance=<guardium_insights_namespace> -oname -n=$<guardium_insights_namespace>` -p '{"spec" : {"suspend" : false }}' -n=$<guardium_insights_namespace>

    where <guardium_insights_namespace> is the Guardium Insights OpenShift namespace that you created when preparing your environment.

  8. Resume any data streams in Guardium Insights that you stopped.