Clearing out topology data without reinstalling

You can clear all topology data without having to reinstall your system.

Before you begin

Ensure you have all the required administrator or management permissions to perform administration tasks.

Also ensure that you have the information specific to your deployment to hand, such as your specific release name prefix and namespace; (you will be setting these variables in step three, after stopping the observer jobs and backing up any required configuration settings).

About this task

To clear all topology data without having to reinstall your system, you perform the following sequence of steps:

  1. Stop all observer jobs from running to ensure no new data is received until the old data has been removed.
  2. Back up any configurations if required.
  3. Set the environment variables manually. This will help with the pre- and post-validation of the topology and inventory databases.
  4. Pre-validate the topology and inventory databases to confirm that the data to be removed is present.
  5. Run the script to clean the topology and inventory databases.
  6. Post-validate to confirm that the data has been removed from the topology and inventory databases.
  7. Scale up the topology pods to restart the system and any configured observer jobs.

To clear topology data

  1. Stop all observer jobs.

    a. Expand Define, click Integrations, click the Manage observer jobs tab, then click Configure, schedule, and manage observer jobs. The Observer jobs page is displayed.

    b. Ensure that no observer jobs are running or are scheduled to run.

  2. Back up IBM Cloud Pak for AIOps configurations if required. For more information, see Backing up and restoring UI topology configuration data in the IBM Netcool Operations Insight documentation.

  3. Set the required environment variables. These will be specific to your deployment.

    a. Set the release and namespace as required (as in the following examples):

    export release=aiops
export NAMESPACE=cp4aiops

    b. Set the following variables:

    export POSTGRES_PASS=$(oc get secret $release-topology-postgres-user -o jsonpath='{.data.password}' | base64 -d)
export POSTGRES_USER=$(oc get secret $release-topology-postgres-user -o jsonpath='{.data.username}' | base64 -d)
export INVENTORY_DB=$(oc get asmformation $release-topology -o jsonpath='{.spec.helmValues.global.postgres.dbname}')

export INVENTORY_SCHEMA=$(oc get asmformation $release-topology -o jsonpath='{.spec.helmValues.global.postgres.schema}')
export CASSANDRA_PASS=$(oc get secret $release-topology-cassandra-auth-secret -o jsonpath='{.data.password}' | base64 -d)
export CASSANDRA_USER=$(oc get secret $release-topology-cassandra-auth-secret -o jsonpath='{.data.username}' | base64 -d)
export installation=$(oc get installation.orchestrator.aiops.ibm.com -o jsonpath="{.items[0].metadata.name}")
export POSTGRES_POD=$(oc get clusters.postgresql.k8s.enterprisedb.io $installation-edb-postgres --namespace $NAMESPACE -o jsonpath="{.status['currentPrimary']}")

  4. Pre-validate the topology and inventory databases to confirm the presence of the data to be cleared.

    a. Use the following command to verify that the topology database has the janusgraph keyspace:

    oc exec $release-topology-cassandra-0 -- bash -c "echo 'SELECT * FROM system_schema.keyspaces;exit' |cqlsh --ssl -u '$CASSANDRA_USER' -p '$CASSANDRA_PASS'"

    The following example output indicates that the janusgraph keyspace is present in the topology (Cassandra) database:

    | keyspace_name | durable_writes | replication                                                                           |
| ------------- | -------------- | ------------------------------------------------------------------------------------- |
|         aiops |           True | {'class': 'org.apache.cassandra.locator.NetworkTopologyStrategy', 'datacenter1': '1'} |
|        x_in_y |           True | {'class': 'org.apache.cassandra.locator.NetworkTopologyStrategy', 'datacenter1': '1'} |
|    janusgraph |           True |   {'class': 'org.apache.cassandra.locator.SimpleStrategy', 'replication_factor': '1'} |
|   system_auth |           True |   {'class': 'org.apache.cassandra.locator.SimpleStrategy', 'replication_factor': '1'} |

    b. Use the following command to verify that the inventory database contains data (under types, edges and other tables).

    oc exec -it $POSTGRES_POD -- /bin/bash -c 'export PGPASSWORD='$POSTGRES_PASS'; psql --host localhost --username '$POSTGRES_USER' --dbname '$INVENTORY_DB' --command "select * from types"'

    The following example output shows that data is present in the inventory (postgres) database:

    |  id |               type | description |
| --- | ------------------ | ----------- |
|   1 | undergroundStation |             | 
|   2 |        railStation |             |
|   5 | interchangeStation |             |
|  31 |      completeGroup |             |
|  32 |                TFL |             |
| 550 |  waiopsApplication |             |
| 552 |             server |             |
| 553 |                 vm |             |
| 554 |           database |             |

  5. Clear out the topology and inventory databases.

    Important: Make a note of the current scaling of the following core services that access the topology and search databases before you configure and run the script:

    $release-topology-inventory
$release-topology-topology
$release-topology-layout
$release-topology-merge
$release-topology-status

    The following command provides the current scaling of core services:

    oc get pods --field-selector=status.phase=Running --no-headers=true --output=custom-columns="CNAME:.metadata.ownerReferences[0].name" | grep $release-topology | uniq --count

    Remember that the $release prefix for the topology services will be specific to your scenario, for example aiops-.

    a. Scale down the core services which access the storage.

    oc scale deploy $release-topology-inventory --replicas=0
oc scale deploy $release-topology-topology --replicas=0
oc scale deploy $release-topology-layout --replicas=0
oc scale deploy $release-topology-merge --replicas=0
oc scale deploy $release-topology-status --replicas=0

    b. Drop the topology janusgraph keyspace.

    oc exec $release-topology-cassandra-0 -- bash -c "echo 'DROP KEYSPACE janusgraph;exit' |cqlsh --ssl -u '$CASSANDRA_USER' -p '$CASSANDRA_PASS'"

    c. Finally, clear out the inventory data

    oc exec -it $POSTGRES_POD -- /bin/bash -c 'export PGPASSWORD='$POSTGRES_PASS'; psql --host localhost --username '$POSTGRES_USER' --dbname '$INVENTORY_DB' --command "CREATE FUNCTION pg_temp.clear_inventory(inventory_schema text) RETURNS void AS \
\$\$ \
    DECLARE \
        statements CURSOR FOR \
        SELECT schemaname, tablename \
        FROM pg_tables \
        WHERE schemaname = FORMAT('\''%s'\'',inventory_schema) \
        AND NOT tablename IN ('\''flyway_schema_history'\'','\''spatial_ref_sys'\'' ); \
    BEGIN \
        FOR stmt IN statements LOOP \
            EXECUTE '\''TRUNCATE TABLE '\'' || string_agg(format('\''%I.%I'\'', stmt.schemaname, stmt.tablename), '\'', '\'') || '\'' CASCADE;'\''; \
        END LOOP; \
    END; \
\$\$ \
LANGUAGE plpgsql;" --command "SELECT pg_temp.clear_inventory('\'''$INVENTORY_SCHEMA''\'');"'

  6. Post-validate to confirm that the data has been removed from the topology and inventory databases.

    a. Use the following command to verify that the topology database no longer has the janusgraph keyspace:

    oc exec $release-topology-cassandra-0 -- bash -c "echo 'SELECT * FROM system_schema.keyspaces;exit' |cqlsh --ssl -u '$CASSANDRA_USER' -p '$CASSANDRA_PASS'"

    The following example output indicates that the janusgraph keyspace is no longer present in the topology (Cassandra) database:

    | keyspace_name | durable_writes | replication                                                                           |
| ------------- | -------------- | ------------------------------------------------------------------------------------- |
|         aiops |           True | {'class': 'org.apache.cassandra.locator.NetworkTopologyStrategy', 'datacenter1': '1'} |
|        x_in_y |           True | {'class': 'org.apache.cassandra.locator.NetworkTopologyStrategy', 'datacenter1': '1'} |
|   system_auth |           True |   {'class': 'org.apache.cassandra.locator.SimpleStrategy', 'replication_factor': '1'} |

    b. Use the following command to verify that the inventory database contains no data.

    oc exec -it $POSTGRES_POD -- /bin/bash -c 'export PGPASSWORD='$POSTGRES_PASS'; psql --host localhost --username '$POSTGRES_USER' --dbname aiops_topology --command "select * from types"'

    The following example output shows that no data is present in the inventory (postgres) database:

    | id | type | description |
| -- | ---- | ----------- |
    
(0 rows)

  7. Scale up the pods to the same state as before you cleared out the data.

    Note: The number of replicas can be found using the command mentioned in the Important note in Step 5.

    oc scale deploy $release-topology-inventory --replicas=<number_of_replicas>
oc scale deploy $release-topology-topology --replicas=<number_of_replicas>
oc scale deploy $release-topology-layout --replicas=<number_of_replicas>
oc scale deploy $release-topology-merge --replicas=<number_of_replicas>
oc scale deploy $release-topology-status --replicas=<number_of_replicas>