Back up and restore Watson Studio Local

This document walks through a full data backup and restore for Watson Studio Local

The main scenarios that are encountered are:

  • Backup for future recovery
  • Backup for move to a new cluster
  • Restore from backup

Back up and restore images, settings, and user-home volume

Back up and restore custom images, dashboard settings from MongoDB, CloudantDB and user-home volume for the same version of Watson Studio Local 1.2.3 on clusters.

The following data is not included in this backup and won't be available in the restored cluster:

  • Job runs and their associated job logs.

Locate and download patch 10. It includes the latest release of the backup and restore tools.

Performing backups using the master_backup.sh script

The master_backup.sh script does all the back ups, and it can be used to run in interactive offline mode backups or in automated online mode backups:
Offline mode
Offline mode restarts the cluster services. Stop all user environments and lock down the cluster before performing the backup. Offline backup is the recommended method to back up Watson Studio Local.
Note: Do not interrupt the master-backup.sh script while you're doing an offline mode backup.
Online mode
Online mode performs backups while the cluster is up; however, busy data objects might not get backed up.
Requirements for using the master_backup.sh script
  • A sec_file that contains a password for encrypt/decrypt influxdb data is required.
  • Install the bc rpm package on the master node yum install bc.
Note: Backup tar files contain data and configuration information of the cluster. Users are responsible for encrypting and protecting the backup files on local or through network transfer.

Using the master_backup.sh script

Use the following commands for backing up a cluster:
./master_backup.sh [-h] [-v] <-b /backup_dir> [-m offline] [-u excluded] [-c excluded] <-s /path_to_sec_file>

-h              - help page (optional)
    
-v              - display patch version
    
-b backup_dir   - the path to where to save the backup tar files
    
-m offline      - to run backup in "offline" mode. The default option is "online"
    
-u excluded     - to exclude user-home backup. The default option is "included"
    
-c excluded     - to exclude custom images backup. The default option is "included"
    
-s /sec_file    - the path to the password file to encrypt/decrypt influxdb during backup/restore
Examples of using the master_backup.sh script
./master_backup.sh -b /backup_dir -s /sec_file						(run all backups online)

./master_backup.sh -b /backup_dir -u excluded -s /sec_file				(run all backups online exclude user-home)

./master_backup.sh -b /backup_dir -c excluded -s /sec_file				(run all backups online exclude custom images)

./master_backup.sh -b /backup_dir -u excluded -c excluded -s /sec_file			(run all backups online exclude user-home and custom images)

./master_backup.sh -b /backup_dir -m offline -s /sec_file				(run all backups offline)

./master_backup.sh -b /backup_dir -m offline -u excluded -s /sec_file			(run all backups offline exclude user-home)

./master_backup.sh -b /backup_dir -m offline -c excluded -s /sec_file			(run all backups offline exclude custom images)

./master_backup.sh -b /backup_dir -m offline -u excluded -c excluded -s /sec_file	(run all backups offline exclude user-home and custom images)
The master-backup.sh script backs up custom images, dashboard settings from MongoDB, CloudantDB, InfluxDB, and user-home volume in the following order:
  1. Custom images
  2. MongoDB
  3. CloudantDB
  4. InfluxDB
  5. User-home volume
The master-backup.sh script restores custom images, dashboard settings from MongoDB, CloudantDB, InfluxDB, and user-home volume in the following order:
  1. User-home volume
  2. InfluxDB
  3. CloudantDB
  4. MongoDB
  5. Custom images

Running the backup and restore scripts manually

Back up and restore custom images

The backup-restore-customimages.sh backs up and restores custom images. The processing time for backing up and restoring custom images depends on the number of images.
  1. To save the custom images inside user-home storage volume, specify the path of the backup directory. For example: 
    ./backup-restore-customimages.sh [ -b | --backup ] /backupdir
  2. During the restore process, the saved custom image are uploaded from the user-home to docker registry. For example:
    ./backup-restore-customimages.sh [ -r | --restore ] /backupdir

Back up and restore MongoDB

The backup-restore-mongo.sh script backs up and restores the admin dashboard SMTP settings from MongoDB. However, the process won't restore metrics and alerts.
  1. To download the admin settings from MongoDB and tar it with the following format: mongo-${timestamp}.tar.gz. You must specify the path of the backup directory. For example: 
    ./backup-restore-mongo.sh  [ -b | --backup ] /backupdir/
  2. During the restore process, the saved custom images are uploaded back to the server. Specify the full path of the tar file. For example:
    ./backup-restore-mongo.sh [ -r | --restore ] /backupdir/mongo-20190508075844.tar.gz

Back up and restore CloudantDB

The backup-restore-cloudant.sh script backs up and restores the Cloudant database. Cloudant contains user login information and other metadata.
  1. During the backup process, the CloudantDB is backed up into a tar file cloudant-backup.tar. You must specify the path to the backup directory. For example:
    /backup-restore-cloudant.sh [ -b | --backup ] /backupdir

    The archive file is an archived directory of json files that correspond to each exported database.

  2. During the restore process, the CloudantDB tar files are restored. Specify the full path of the tar file. For example: 
    ./backup-restore-cloudant.sh [ -r | --restore ]  /backupdir/cloudant-20190508075844.tar.gz

Back up and restore InfluxDB

The backup-restore-influxdb.sh script backs up and restores InfluxDB. InfluxDB includes machine-learning model evaluation history and all deployment metrics and history.
Note: Restoring InfluxDB overrides the original random created passwords. The password must be backed up to a file. For security reasons, the backup tar file must be protected with a password. Therefore, when you perform the backup, you must provide a random password in a file. The same file is also required for restore.
  1. During the backup process, the InfluxDB is backed up into a tar file /backup_dir/'influxdb-${timestamp}.tar.gz. You must specify the path to the backup directory and the full path to the secret/password file. For example:
    ./backup-restore-influxdb.sh -b /backup_dir /path_to_sec_file
  2. During the restore process, the InfluxDB tar file is restored. Specify the full path of the tar file and the path to the password file. For example:
     Example: ./backup-restore-influxdb.sh -r /backup_dir/influxdb-20190829114646.tar.gz /path_to_sec_file

Back up and restore user-home volume

The backup-restore-user-home.sh script backs up and restores user-home.
  1. During the backup process, back up the user-home volume to a tar file user-home-${timestamp}.tar.gz. You must specify the path of backup directory. For example:
    ./backup-restore-user-home.sh [ -b | --backup ] /backupdir/
  2. During the restore process, the user-home tar is restored over a user-home volume. Specify the full path of the tar file. For example: 
    ./backup-restore-user-home.sh [ -r | --restore ] /backupdir/user-home-20190508075844.tar.gz
Restoring missing files or directories
To restore missing files or directories from the user-home tar file:
  1. Open the user-home.tar file.
  2. Mount user-home NFS/glusterFS volume over a mount-point directory.
  3. Copy the missing the files from the untarred user-home directory to the right location inside the mount-point directory.

Procedures for after performing a restore

All of the pods under the dsx namespace must be restarted after the restore process is complete. To do so, run the following command:

kubectl get po -l 'heritage=Tiller,!job-name' -n dsx --no-headers | awk '{system("kubectl delete po -n dsx " $1)}'