Graceful shutdown

Provides detailed instructions to put Data Cataloging in idle state on an OpenShift® environment.

About this task

The procedure might be helpful in situations where Data Cataloging is behaving unexpectedly or there is a maintenance window of not more than one hour.

Procedure

  1. Stop or wait for all running scans and policies to stop. Do not skip this step until everything looks stopped.
  2. Do oc login to your OpenShift Container Platform cluster.
  3. Do oc project <project> to select namespace where Data Cataloging service is installed.
  4. Stop all pods and set replicas to zero for the following deployments. and respect the following order:
    Note: Follow the suggested sequence.
    1. Consumers
      1. Run the following command to get the list of the deployments and filter by consumer.
        $oc get deployments |grep consumer
        Example output:
        isd-consumer-ceph-le                    10/10   10           10          6d20h
        isd-consumer-cos-le                     10/10   10           10          6d20h
        isd-consumer-cos-scan                   10/10   10           10          6d20h
        isd-consumer-file-scan                  10/10   10           10          6d20h
        isd-consumer-protect-scan               10/10   10           10          6d20h
        isd-consumer-scale-le                   10/10   10           10          6d20h
        isd-consumer-scale-scan                 10/10   10           10          6d20h
        
      2. Run the following to command to scale down replicas to 0 for every consumer deployment.
        $oc scale --replicas=0 deployment <deployment_name>
        Example output:
        $oc scale --replicas=0 deployment isd-consumer-file-scan
        deployment.apps/ isd-consumer-file-scan scaled
      3. Ensure you scale down replicas to 0 for every consumer deployment.
        Example output:
        $oc get deployments|grep consumer-cos-scan
        isd-consumer-file-scan                   0/0     0            0           6d20h
    2. Producers
      1. Run the following command to get the list of the deployments and filter by producer.
        $oc get deployments |grep producer
        Example output:
        isd-producer-ceph-le                    1/1     1            1           6d20h
        isd-producer-cos-le                     1/1     1            1           6d20h
        isd-producer-cos-scan                   1/1     1            1           6d20h
        isd-producer-file-scan                  1/1     1            1           6d20h
        isd-producer-protect-scan               1/1     1            1           6d20h
        isd-producer-scale-le                   1/1     1            1           6d20h
        isd-producer-scale-scan                 1/1     1            1           6d20h
        
      2. Run the following to command to scale down replicas to 0 for every producer deployment.
        Important: Ensure that you scale down replicas to 0 for every producer deployment.
        $oc scale --replicas=0 deployment <deployment_name>
        Example output:
        $oc scale --replicas=0 deployment isd-producer-file-scan
        deployment.apps/ isd-producer-file-scan scaled
      3. Ensure that you have scale down replicas to 0 for every producer deployment.
        Example output:
        
        $oc get deployments|grep producer-cos-scan
        isd-producer-cos-scan                   0/0     0            0           6d20h
    3. Connection manager
      1. Run the following command to get the list of the statefulsets and filter by connection manager.
        $oc get statefulsets |grep connmgr
        Example output:
        $oc get statefulsets |grep connmgr
        connmgr                    0/0     0            0           6d20h
        
      2. Run the following to command to scale down replicas to 0 for every connection manager statefulset.
        Important: Ensure that you have scale down replicas to 0 for every connection manager statefulset.
        $oc scale --replicas=0 statefulset
        Example output:
        $oc scale --replicas=0 statefulset connmgr
        statefulset.apps/ connmgr scaled
      3. Ensure that you have scale down replicas to 0 for every connection manager statefulset.
        Example output:
        $oc get statefulsets|grep connmgr
        connmgr                   0/0     0            0           6d20h
    4. Check whether db2whrest stopped inserting data properly
      Ensure that no database activity is present before stopping db2whrest. It is necessary to check that specific csv files created are empty, which means there are no pending jobs being processed.
      1. Run the following command to go to db2whrest.
        oc rsh deployment/isd-db2whrest

        Now, list all CSV files related to merging that are empty.

        For example:
        $for file in $(find /gpfs/gpfs0/db2wh/home/bluadmin -name *LOAD*.csv); do file $file; done |grep data
        /gpfs/gpfs0/db2wh/home/bluadmin/ACOG_LOAD4.csv: data
        /gpfs/gpfs0/db2wh/home/bluadmin/ACOG_LOAD1.csv: data
        $
        
      2. Wait for a few seconds and try again until you get an empty list.
    5. DB2WHRest
      1. Run the following command to get the list of the deployments and filter by db2whrest.
        $oc get deployments |grep db2whrest
        Example output:
        $oc get deployments |grep db2whrest
        isd-db2whrest                    1/1     1            1           6d20h
      2. Run the following command to scale down replicas to 0 for every db2whrest deployment.
        $oc scale --replicas=0 deployment <deployment_name>
        Example output:
        $oc scale --replicas=0 deployment db2whrest
        deployment.apps/ db2whrest scaled
      3. Ensure that all scale down replicas are 0 for every db2whrest deployment.
        Example output:
        $oc get deployments|grep db2whrest
        isd-db2whrest                   0/0     0            0           6d20h
  5. Stop Db2 instance. For procedure, see Stopping and starting a Db2 instance.
  6. After checking that everything is down, Data Cataloging is successfully in an idle state. It is recommended to be in this state for a short period, not more than one hour. Warning messages on other Data Cataloging pods are expected.