Backing up and restoring database data (on-prem)

You can back up and restore existing topology data using scripts included in the Agile Service Manager on-prem installation. This can be helpful when updating your system, or for maintenance reasons.

About this task

Agile Service Manager stores its information in the Cassandra database, using in turn PostgreSQL for providing inventory-like search and query capabilities of the same data. The backup and restore procedures are therefore based on backing up the Cassandra database (where all the information is stored) and, upon its restoration, triggering a topology re-broadcast to re-populate PostgreSQL to keep the consistency of the information in both stores.

The backup_casssandra.sh and restore_cassandra.sh scripts are stored in the ASM_HOME/bin directory.

Tip: The backup utility runs from inside the Cassandra Docker Container, and thus filesystem paths are relative from inside that container. Typically these file paths can be accessed from the host system via

${ASM_HOME}/logs/cassandra

and

${ASM_HOME}/data/cassandra/backup

To restore topology data on a new system, you start with a clean instance of Cassandra, then start the Topology Service to create keys and indexes in Cassandra, and when the topology is complete, you run the restore script.

Note: To restore Cassandra to a previous state, for example in scenarios where the Docker container has not been re-created, you run the Restore script without cleaning Cassandra first.

Procedure

Backing up data

From the ASM_HOME directory, run the backup script:

Example

# ./bin/backup_casssandra.sh 

# Description : The backup script will complete the backup in multiple phases - 
#  1. Take statistics of the keyspace(s) before backup 
#  2. Clear existed snapshots 
#  3. Take backup of keyspace(s) SCHEMA in temporary BACKUP_TEMP_DIR
#  4. Take snapshot of keyspace(s) 
#  5. Copy snapshot to temporary BACKUP_TEMP_DIR 
#  6. Compact the temporary BACKUP_TEMP_DIR in one tar file and send it to BACKUP_DIR

USAGE: backup_cassandra.sh 
   [ -k keyspace to backup ] # default is ALL keyspaces 
   [ -b  temporary backup dir ] # default is /opt/ibm/cassandra/data/backup/../backup_temp 
   [ -d  datadir ] # default is /opt/ibm/cassandra/data/data
   [ -s storagedir ] # default is /opt/ibm/cassandra/data/backup
   [ -u Cassandra username ]
   [ -p Cassandra password ]
   [ -log logdir ] # default is /opt/ibm/cassandra/logs
   [ -speed tardiskspeed ] # default is 17M
   [ -f ] # for non interactive mode

********** START CONFIGURATION ***************** 
KEYSPACE_TO_BACKUP=ALL 
BACKUP_TEMP_DIR=/opt/ibm/cassandra/data/backup/../backup_temp 
BACKUP_DIR=/opt/ibm/cassandra/data/backup 
CASSANDRA_DATA=/opt/ibm/cassandra/data/data 
LOG_PATH=/opt/ibm/cassandra/logs 
TAR_SPEED_LIMIT=17M 
FORCE=N 
USER=cassandra 
PASS=XXXX 
********** END CONFIGURATION ***************** 

Tue Mar  5 13:15:26 UTC 2019 ************************************ 
Tue Mar  5 13:15:26 UTC 2019  Do you want to continue (y/n) ?

Restore topology data (on an existing system)

From the ASM_HOME directory, run the restore script:

Example

# ./bin/restore_cassandra.sh 

# Description : The restore script will complete the restore in multiple phases - 
1. Take statistics of the cassandra node before restore 
2. Check if the keyspace exists and if it does not exist, create it using the schema cql file saved in the backup file 
3. Truncate all tables in keyspace  
4. Clear all files in commitlog directory 
5. Copy contents of desired snapshot to active keyspace. 
6. Refresh all tables in that keyspace 
7. Take statistics of the cassandra node after restore and compare with statistics taken before backup, making sure number of keys per table is the same

USAGE: restore_cassandra.sh 
    -k keyspaceName  # compulsory parameter 
   [ -h backup hostname] # if backup was done on a different hostname than 96c6953586a3 
   [ -b temporary backup dir  ] # default is /opt/ibm/cassandra/data/backup/../backup_temp  
   [ -d  dataDir ] # default is /opt/ibm/cassandra/data/data
   [ -t  snapshotTimestamp ] # timestamp of type date YYYY-MM-DD-HHMM-SS - default is latest
   [ -s  storageDir ] # default is /opt/ibm/cassandra/data/backup
   [ -u Cassandra username ]
   [ -p Cassandra password ]
   [ -log logDir ] # default is /opt/ibm/cassandra/logs
   [ -f ]  # for non interactive mode


********** START CONFIGURATION ***************** 
BACKUP_TEMP_DIR=/opt/ibm/cassandra/data/backup/../backup_temp 
BACKUP_DIR=/opt/ibm/cassandra/data/backup 
DATA_DIR=/opt/ibm/cassandra/data/data 
LOG_PATH=/opt/ibm/cassandra/logs 
LOCAL HOSTNAME=96c6953586a3 
BACKUP_HOSTNAME=96c6953586a3 
SNAPSHOT_DATE_TO_RESTORE=latest 
KEYSPACE_TO_RESTORE=janusgraph 
FORCE=N 
USER=cassandra 
PASS=XXXX 
********** END CONFIGURATION ***************** 

Tue Mar  5 13:17:22 UTC 2019 ************************************ 
Tue Mar  5 13:17:22 UTC 2019  Do you want to continue (y/n) ?

Rebroadcast data.
Run the Rebroadcast administration routine from the Topology configuration page, as described in the Running data administration routines topic. You can also manually rebroadcast data using a script.

Restore topology data (on a new system)

Clean the Cassandra database.
1. Ensure the topology service and Cassandra database are not running.
  Stop Agile Service Manager using the asm_stop.sh command:
```
$ASM_HOME/bin/asm_stop.sh
```
2. Rebroadcast data.
  Run the Rebroadcast administration routine from the Topology configuration page, as described in the Running data administration routines topic. You can also manually rebroadcast data using a script.
3. Remove Cassandra database content.
```
rm -rf /opt/ibm/netcool/asm/data/cassandra/data/*
```
4. Remove inventory database content.
```
rm -rf /opt/ibm/netcool/asm/data/postgres/data/*
```
5. Start Cassandra.
```
docker-compose up -d cassandra
```
Start the topology service.
Start Agile Service Manager using the asm_start.sh command:
```
$ASM_HOME/bin/asm_start.sh
```
This will create keys and indexes in Cassandra.
Run the restore script from the Cassandra container.
Note: For a new host, you must use the -h switch to identify the backup hostname, as it will be different to the hostname into which you are restoring.
```
./restore_cassandra.sh -h <backup_hostname>
```
For example:
```
./restore_cassandra.sh -h 96c6953586a3
```
Rebroadcast data.
Run the Rebroadcast administration routine from the Topology configuration page, as described in the Running data administration routines topic. You can also manually rebroadcast data using a script.

Tip: To resynchronize data manually, call the rebroadcast API of the Topology service, specifying a tenantId:

https://master_fqdn/1.0/topology/swagger#!/Crawlers/rebroadcastTopology