IBM Spectrum Protect Plus: Migrating IBM Spectrum Protect Plus server to 10.1.11

Steps

Follow the procedure carefully. If not followed, this procedure can result in a loss of data.

To update the IBM Spectrum Protect Plus server, complete the following steps:

1. Collect information about the IBM Spectrum Protect Plus server virtual hardware

1. Note the following properties of the IBM Spectrum Protect Plus virtual machine by using the VMware vSphere UI or Hyper-V Management Console.
   • CPU configuration
   • Memory configuration
   • Network adapter configuration
   • Hard disk configuration
   Alternatively, you can use VMware PowerCLI to collect this information. For example:

   PS /Users/spike> get-vm -Name "SPP-VM" | fl  
   PS /Users/spike> get-vm -Name "SPP-VM" | get-harddisk | fl
   PS /Users/spike> get-vm -Name "SPP-VM" | Select-Object -Property Name,@{Name=’Cluster’;Expression={$_.VMHost.Parent}}
   PS /Users/spike> get-vm -name "SPP-VM" | Get-NetworkAdapter
   PS /Users/spike> (get-VM "SPP-VM").Guest.Nics | fl

2. By using Secure Shell (SSH), log in to the IBM Spectrum Protect Plus and console server as the serveradmin user.
3. Note the server network hostname by using the following commands:

   # Short hostname
   hostname -s
   
   # Fully qualified domain name
   hostname --fqdn

4. To check the IBM Spectrum Protect Plus server network configuration, run the following command:

   nmcli device show
   

   Alternatively, examine the contents of the following configuration files:

   /etc/sysconfig/network-scripts/ifcfg-*

5. To collect the information about disk partitioning, LVM information, file system types, and sizes, run the following command:

   lsblk -o name,size,type,mountpoint,fstype
   df

2. Pause all jobs in IBM Spectrum Protect Plus

1. Log on to the IBM Spectrum Protect Plus server.
2. Jobs must not be actively running or scheduled to run during the migration procedure. Pause the schedule for all jobs to ensure that they do not attempt to run while the migration is occurring. Click Jobs and Operations > Schedule and then click Pause All Jobs.
3. Verify that no jobs are running by clicking Jobs and Operations > Running Jobs. If one or more jobs are running, wait for them to complete.

3. Back up the IBM Spectrum Protect Plus catalog

Create a separate SLA policy for backing up the IBM Spectrum Protect Plus catalog before migration. Even if there are any default SLA policies are assigned for catalog backup, it is advised that you create a new SLA policy and capture a fresh backup to minimize the risk of errors.

1. In the IBM Spectrum Protect Plus, go to Policy Overview, and then click Add Policy. The New SLA Policy pane is displayed.
2. Select the SLA policy type, click VMware, Hyper-V, to set the backup options. 
   1. Specify the retention period for the backup snapshots (for example 5 days).
   2. Select Disable Schedule, policies that are created without a schedule can be run on demand.
      Note: Do not enable replication or extra copies in the SLA.
   3. Save the SLA.
3. To create a job definition, in the navigation panel, click Manage Protection > IBM Spectrum Protect Plus >  Backup and select the SLA policy, and then click Save.
4. To run a job, click Jobs and Operations > Schedule, select the catalog job created in the previous step, and click Actions > Start.
5. To view completed jobs, in the navigation panel, click Jobs and Operations > Job History.
6. In the job log, select the message ID CTGGA0833. The message indicates the name of the vSnap volume and the vSnap server where the backup was created: By using volume <volName> on server <vSnapName> for protection. Make a note of both the names.
7. In the job log, select the message ID CTGGA0835. The message indicates the name of the vSnap snapshot: Created snapshot <snapName> on volume <volName>. Make a note of the snapshot name.
8. To verify the catalog restore point is visible, in the navigation panel, click Manage Protection > IBM Spectrum Protect Plus > Restore. Select a vSnap name noted earlier in step (f) and search for the restore points.
   Note: Check the latest restore point and verify its timestamp. Ensure that its name matches the snapshot name noted in the previous step (g), and the SLA policy name matches the one created earlier in step (b).
9. To shut down the IBM Spectrum Protect Plus server, by using Secure Shell (SSH), log in to the IBM Spectrum Protect Plus and console server as the serveradmin user, run the following command:

   shutdown

   Note: By using the VMware vSphere UI or Hyper-V Manager, ensure that you power off the IBM Spectrum Protect Plus virtual machine (VM).

4. Deploy and configure new IBM Spectrum Protect Plus server running 10.1.11

1. Deploy the new IBM Spectrum Protect Plus server running version 10.1.11 by using the VMware OVA or Hyper-V installer. Power off the old IBM Spectrum Protect Plus server and rename it to avoid confusion (for example, by adding the suffix "old" to the name of the older server).
2. Customize the CPU and memory configuration of the virtual machine to match the values previously noted on the old server in step (1)(a).
3. Log in to the new IBM Spectrum Protect Plus server by using SSH. The default username is serveradmin and the password is sppDP758-SysXyz. Change the password when prompted. Also, log in to the IBM Spectrum Protect Plus UI by using default username admin and password password. Changes the values when prompted.
4. Customize the network configuration to specify hostname and IP address. Ensure that the hostname and IP address matches with old IBM Spectrum Protect Plus server as specified earlier in step (1)(d).
   • For IBM Spectrum Protect Plus servers deployed in VMware, you can customize the network configuration as part of the deployment wizard, or through the operating system.
   • For IBM Spectrum Protect Plus servers deployed in Hyper-V, you can customize the network configuration through the operating system.
5. Customize the configuration of the disk partitions like /data, /data2, and /data3 to match the values previously noted on the old server in step (1)(e). For example, the partition sizes on the old server need to match the partition sizes on the new server.
   Note: Do not manually copy any data into these partitions. You can restore the catalog data in a later step of the migration process.

5. Understand vSnap certificate management and optionally disable trust on first use (TOFU) for vSnap server certificates

Beginning with IBM Spectrum Protect Plus version 10.1.11, each vSnap generates a unique self-signed certificate during the initial registration or deployment of the vSnap server.

For new vSnap servers that are registered after IBM Spectrum Protect Plus is migrated to 10.1.11, you must paste or upload the vSnap certificate. Also, you must verify the SSH host key fingerprint of the vSnap server. The hostname or IP address of the vSnap server must match one of the Subject Alternative Names (SAN) in the certificate. For more information about certificate management, see https://www.ibm.com/docs/en/spp/10.1.11?topic=reference-certificate-management).

For existing vSnap servers that are already registered, the IBM Spectrum Protect Plus server will follow a trust on first use (TOFU) mechanism after migration to 10.1.11. On the first connection, the IBM Spectrum Protect Plus appliance retrieves the TLS certificate information from each vSnap server and updates the registration in the IBM Spectrum Protect Plus catalog. This information is used for subsequent connections to the host.

To adopt a more strict security posture, you can disable the TOFU feature. This disabling must be done before you restore the IBM Spectrum Protect Plus catalog. Disabling TOFU is optional and has the following effects:

• Existing vSnap servers are not trusted by default.
• After migration, you must clearly edit the registration of each vSnap server to obtain the server key and upload or paste the TLS certificate.
• If you do not do this action, jobs fail on the next scheduled run and continue to fail until the vSnap is verified by editing the registration.

If you opt to disable, trust on first use:

1. Log in to the IBM Spectrum Protect Plus server by using SSH.
2. Create a file named disable-trust-on-first-use in the /data directory. This file can be an empty file created by using the following command:

   sudo touch /data/disable-trust-on-first-use

6. Restore IBM Spectrum Protect Plus catalog to the new server

1. To identify the vSnap server backup location, follow the instructions specified earlier in step (3)(f).
2. To view the TLS certificate, log in to the vSnap server by using SSH and run the following command:

   vsnap system cert show

3. Log in to the IBM Spectrum Protect Plus UI on the new server and go to System Configuration > Storage > vSnap Servers.
4. Click 'Add vSnap server' and register the vSnap noted in step (3)(f) which contains the last catalog backup. Specify the Primary site and enter the credentials for the vSnap serveradmin user, or alternatively another user created by using command vsnap user create. When prompted to enter the certificate, paste the certificate value (including the BEGIN and END lines) noted in the previous step (b).
5. When the vSnap is registered, go to Manage Protection > IBM Spectrum Protect Plus > Restore. Select the vSnap name to browse the available restore points.
6. Locate the catalog restore point that was previously examined in step (3)(h) which matches the most recent catalog backup created before migration.
7. Click Restore next to the restore point, then select the option Restore the catalog and suspend all scheduled jobs.
8. You are automatically logged out of the IBM Spectrum Protect Plus UI while the restore is in progress. During this phase, log information is written to the following files, which can be monitored by using an SSH session to the IBM Spectrum Protect Plus server.

   /data/log/catalogprotection/managedb-catalogrestore.log
   /data/log/catalogprotection/managedb-catalogrestore-joblog.log
   /data/log/catalogprotection/catalog_restore_wrapper.log

   Note: After the process is complete, the virgo service will start and the data is written to the virgo log /opt/virgo/serviceability/logs/log.log. The IBM Spectrum Protect Plus UI is again accessible, but the job logs take some time to fully load, thus they are not immediately visible. Loading of job logs continues in the background. An alert is generated in the user interface after the job log recovery is completed.

7. Perform health checks on the new server

1. Log in to the IBM Spectrum Protect Plus UI by using the credentials that are used on the old server. 
   • Note: To successfully log in to the IBM Spectrum Protect Plus UI, you must provide the IBM Spectrum Protect Plus license file and previous SYSADMIN level user credentials.
2. Go to Jobs and Operations > Schedule and verify that all expected jobs are listed. All jobs must be in Held state after the catalog restore is complete.
3. Go to System Configuration > Storage > vSnap Servers and verify that all expected vSnap servers are listed.
4. If you optionally disabled TOFU in step (6) earlier, then edit the registration of each vSnap server. Click Get server key to obtain the SSH fingerprint of the vSnap server. Upload or paste the vSnap certificate as obtained by running command vsnap system cert show on the vSnap server.
5. Click Refresh for each vSnap server and ensure that the information is accurate. Regardless of whether you disabled TOFU, the connection to the vSnap server fails if the hostname or IP address of the vSnap does not match one of the SANs in the certificate.
   Note: Refer to the step (11) of the vSnap migration process to verify the accuracy of the SANs, troubleshoot, and resolve the vSnap certificate. For more information, see IBM Spectrum Protect Plus: Migrating vSnap server to 10.1.11.
6. Go to System Configuration > VADP Proxy. If you have one or more proxy servers, verify their health and verify the version is reflected as 10.1.11. Existing proxy servers running version 10.1.9 or 10.1.10 are auto-upgraded to 10.1.11 when the IBM Spectrum Protect Plus server starts up after the catalog restore. Depending on the number of proxy servers, it might take some time to update all proxy servers.

8. Final steps to start by using the migrated IBM Spectrum Protect Plus server

Resume job schedules to start by using the migrated IBM Spectrum Protect Plus server. First, release the schedule for a smaller backup job and ensure that it is operating normally. You can also release schedules in small batches, or all at once.