Graceful shutdown
Provides detailed instructions to put Data Cataloging in idle state on an OpenShift® environment.
About this task
The procedure might be helpful in situations where Data Cataloging is behaving unexpectedly or there is a maintenance window of not more than one hour.
Procedure
- Stop or wait for all running scans and policies to stop. Do not skip this step until everything looks stopped.
- Do
oc
login to your OpenShift Container Platform cluster. - Do
oc
project <project>
to select namespace where Data Cataloging service is installed. - Stop all pods and set replicas to zero for the following deployments. and respect the
following order: Note: Follow the suggested sequence.
-
- Consumers
-
- Run the following command to get the list of the deployments and filter by
consumer.
$oc get deployments |grep consumer
Example output:isd-consumer-ceph-le 10/10 10 10 6d20h isd-consumer-cos-le 10/10 10 10 6d20h isd-consumer-cos-scan 10/10 10 10 6d20h isd-consumer-file-scan 10/10 10 10 6d20h isd-consumer-protect-scan 10/10 10 10 6d20h isd-consumer-scale-le 10/10 10 10 6d20h isd-consumer-scale-scan 10/10 10 10 6d20h
- Run the following to command to scale down replicas to 0 for every consumer
deployment.
$oc scale --replicas=0 deployment <deployment_name>
Example output:$oc scale --replicas=0 deployment isd-consumer-file-scan deployment.apps/ isd-consumer-file-scan scaled
- Ensure you scale down replicas to 0 for every consumer deployment.Example output:
$oc get deployments|grep consumer-cos-scan isd-consumer-file-scan 0/0 0 0 6d20h
- Run the following command to get the list of the deployments and filter by
consumer.
-
- Producers
-
- Run the following command to get the list of the deployments and filter by
producer.
$oc get deployments |grep producer
Example output:isd-producer-ceph-le 1/1 1 1 6d20h isd-producer-cos-le 1/1 1 1 6d20h isd-producer-cos-scan 1/1 1 1 6d20h isd-producer-file-scan 1/1 1 1 6d20h isd-producer-protect-scan 1/1 1 1 6d20h isd-producer-scale-le 1/1 1 1 6d20h isd-producer-scale-scan 1/1 1 1 6d20h
- Run the following to command to scale down replicas to 0 for every producer deployment.Important: Ensure that you scale down replicas to 0 for every producer deployment.
$oc scale --replicas=0 deployment <deployment_name>
Example output:$oc scale --replicas=0 deployment isd-producer-file-scan deployment.apps/ isd-producer-file-scan scaled
- Ensure that you have scale down replicas to 0 for every producer deployment.Example output:
$oc get deployments|grep producer-cos-scan isd-producer-cos-scan 0/0 0 0 6d20h
- Run the following command to get the list of the deployments and filter by
producer.
-
- Connection manager
-
- Run the following command to get the list of the statefulsets and filter by connection
manager.
$oc get statefulsets |grep connmgr
Example output:$oc get statefulsets |grep connmgr connmgr 0/0 0 0 6d20h
- Run the following to command to scale down replicas to 0 for every connection manager
statefulset.Important: Ensure that you have scale down replicas to 0 for every connection manager statefulset.
$oc scale --replicas=0 statefulset
Example output:$oc scale --replicas=0 statefulset connmgr statefulset.apps/ connmgr scaled
- Ensure that you have scale down replicas to 0 for every connection manager
statefulset.Example output:
$oc get statefulsets|grep connmgr connmgr 0/0 0 0 6d20h
- Run the following command to get the list of the statefulsets and filter by connection
manager.
-
- Check whether db2whrest stopped inserting data properly
- Ensure that no database activity is present before stopping db2whrest. It is necessary to check that specific csv files created are empty, which means there are no pending jobs being processed.
- Run the following command to go to
db2whrest.
oc rsh deployment/isd-db2whrest
Now, list all CSV files related to merging that are empty.
For example:$for file in $(find /gpfs/gpfs0/db2wh/home/bluadmin -name *LOAD*.csv); do file $file; done |grep data /gpfs/gpfs0/db2wh/home/bluadmin/ACOG_LOAD4.csv: data /gpfs/gpfs0/db2wh/home/bluadmin/ACOG_LOAD1.csv: data $
- Wait for a few seconds and try again until you get an empty list.
-
- DB2WHRest
-
- Run the following command to get the list of the deployments and filter by
db2whrest.
$oc get deployments |grep db2whrest
Example output:$oc get deployments |grep db2whrest isd-db2whrest 1/1 1 1 6d20h
- Run the following command to scale down replicas to 0 for every db2whrest
deployment.
$oc scale --replicas=0 deployment <deployment_name>
Example output:$oc scale --replicas=0 deployment db2whrest deployment.apps/ db2whrest scaled
- Ensure that all scale down replicas are 0 for every db2whrest deployment.Example output:
$oc get deployments|grep db2whrest isd-db2whrest 0/0 0 0 6d20h
- Run the following command to get the list of the deployments and filter by
db2whrest.
-
- Stop Db2 instance. For procedure, see Stopping and starting a Db2 instance.
- After checking that everything is down, Data Cataloging is successfully in an idle state. It is recommended to be in this state for a short period, not more than one hour. Warning messages on other Data Cataloging pods are expected.