Restoring the system configuration

You can restore the configuration data for the system after preliminary tasks are completed.

About this task

This procedure only restores the system configuration and does not restore data on the volumes. Use this procedure if the data is not required, or if you can restore the data from an external source.

This procedure is also known as Tier 4 (T4) recovery.
Important: Before running a T4 procedure, contact IBM® support for assistance.

Both automated and manual steps are involved. Do not attempt to make any changes to the environment or configuration until all steps are completed.

The user performing the restore must have a Security Administrator role on the system.

If you do not understand the instructions to run the CLI commands, see the Command-line interface reference information.

Preparing the environment

Before starting a restore, ensure that the environment is ready. For example:
  • Network hardware must be online and operational.
  • Network configuration, such as fibre-channel zoning and firewall settings, must match prior configuration.
  • System hardware, including all nodes and enclosures (as applicable) must be available and powered on.

Preparing the system hardware

Verify that all nodes are available as candidate nodes before you run this recovery procedure. You must remove errors 550 or 578 to put the node in the candidate state. For all nodes that display these errors, follow this procedure.
  1. Point your browser to the service IP address of one of the nodes (for example, https://node_service_ip_address/service/).
  2. Log on to the service assistant.
  3. From the Home page, put the node canister into service state if it is not already in that state.
  4. Select Manage System.
  5. Click Remove System Data.
  6. Confirm that you want to remove the system data when prompted.
  7. Exit service state from the Home page. The 550 or 578 errors are removed, and the node appears as a candidate node.
  8. Remove the system data for the other nodes that display a 550 or a 578 error.
    All nodes previously in this system must have a node status of Candidate and have no errors listed against them.
    Note: A node that is powered off might not show up in this list of nodes for the system. Diagnose hardware problems directly on the node by using the service assistant IP address and by physically verifying the LEDs for the hardware components.
On IBM FlashSystem, verify that all nodes are available as candidate nodes with blank system fields. Complete the following steps on one node in each control enclosure:
  1. Connect to the service assistant on either of the nodes in the control enclosure.
  2. Select Configure Enclosure.
  3. Select the Reset the system ID option. Do not make any other changes on the panel.
  4. Click Modify.
If the system uses encryption, the recovery procedure generates new encryption keys. Make sure that key servers are online and 3 USB flash drives are installed in the node that will be used to restore the configuration. Any existing encryption key files on USB flash drives correspond to the previous system and are no longer required unless using transparent cloud tiering.
Note: On systems with fewer than 3 USB ports, encryption must be enabled manually later on in the recovery. On these systems, follow the instructions that are displayed on screen to manually enable encryption during step 17 when the configuration restore is prepared.

If the system uses encryption with transparent cloud tiering, at least one USB flash drive that contains the encryption key file for the previous system must be installed in the system. This allows the current encrypted data to be unlocked and re-encrypted with the new keys.

Preparing the cluster configuration

One of the nodes previously in I/O group zero must perform the restoring the system configuration. For example, property name="IO_group_id" value="0" .

On the identified node, connect to the technician port and use the setup wizard to create a system.

On the newly created basic system, verify the hardware:
  • On FlashSystem, only the control enclosure on which the clustered system was created and directly attached expansion enclosures are displayed.
  • On SAN Volume Controller systems, only the single node is displayed.

It is recommended to configure an SSH key for the superuser to enable, remaining actions to be performed via the command-line interface. For more information, see Setting up SSH access.

The GUI setup wizard may create some configuration, which should now be deleted for the automated recovery process to be successful:
  • If a default call home email user was created as part of the setup wizard, delete the call home email user now.
  • If you set up email notification in the setup wizard, you must now remove that email user and server so that the original configuration can be restored.
    1. Issue the following CLI command to remove the new email user:
      rmemailuser 0
    2. Issue the following CLI command to remove the new email server:
      rmemailserver 0
    Important: Do not attempt to make any other configuration changes at this stage.

Identifying a configuration backup file

The file can be either a local copy of the configuration backup JSON file that you saved when you backed-up the configuration or an up-to-date file on one of the nodes.

Configuration data is automatically backed up daily at 01:00 system time on the configuration node.

Download and check the configuration backup files on all nodes that were previously in the system to identify the one containing the most recent complete backup.
  1. From the management GUI, click Settings > Support > Support Package.
  2. Expand Manual Upload Instructions and select Download Support Package.
  3. On the Download New Support Package or Log File page, select Download Existing Package.
  4. For each node (canister) in the system, complete the following steps:
    1. Select the node to operate on from the selection box at the top of the table.
    2. Find all the files with names that match the pattern svc.config.*.json*.
    3. Select the files and click Download to download them to your computer.

The JSON files contain a date and time that can be used to identify the most recent backup. After you identify the backup JSON file that is to be used when you restore the system, rename the file to svc.config.backup.json.

Copy onto the system the JSON backup file from which you want to restore.
pscp full_path_to_identified_svc.config.file 
superuser@cluster_ip:/dumps/svcconfig/svc.config.backup.json

Preparing the system for the automated restore process

Some features must be configured before the automated restore:

Replication layer

By default, the newly initialized system is created in the replication layer. The layer of the system is not restored automatically from the configuration backup JSON file. If the system you are restoring was previously configured in the storage layer, you must change the layer manually now. For more information about the replication layer and storage layer, see System layers.

Key server encryption

If the system has key server encryption, the new certificate must be exported by using the chsystemcert -export command, and then installed on all key servers in the correct device group. The device group that is used is the one in which the previous system was defined. It might also be necessary to get the new system's certificate signed.

USB encryption

If the system has fewer than 3 USB ports, encryption must be enabled manually now. For more information, see Encryption with USB flash drives.

Non-default MTU
If a system management IP is configured on a port with MTU other than 1500, then follow this procedure:
  1. Create a system using the other IP address and port than the actual system management IP address and port.
  2. Change the MTU of the Ethernet port to the intended MTU value on the actual port of the system management IP address.
  3. Configure the system management IP address on the actual port of the system management IP address.
  4. Delete the first system management IP address using rmip CLI command.
iSCSI storage controllers
If the system contains any iSCSI storage controllers, these controllers must be detected manually now. The nodes that are connected to these controllers, MTU for the Ethernet ports, the iSCSI port IP addresses, and the iSCSI storage ports must be added to the system before you restore your data.
Note: If the system contains only Fibre Channel storage controllers, proceed to the Running the automated restore process section.
Note: For a stretched topology, after you run the addnode command, change the sites of all of the nodes added in the system. For example, 
chnodecanister -site site_id node_id/node_name

  1. To restore iSCSI initiator port configuration, use the chportethernet command. All the ethernet ports capable of iSCSI backend i.e. storage flag set to yes needs to be configured. Find out the MTU of ethernet port for which "storage" is set to yes in the node_ethernet_port section from the backup configuration file and restore the port MTU.

    1. To restore port MTU, determine iogrp_id from the node_id, port_id, and mtu of the port for which "storage" is set to yes from the configuration backup file, and run the following command: 
      chportethernet -iogrp iogrp_name_or_id -mtu mtu port_id
      where mtu is the MTU of the port, iogrp_name_or_id is the name or ID of the I/O group, port_id is the ID of the port.

      Complete step i for all (earlier configured) Ethernet ports that belong to storage in the node_ethernet_port section from the backup configuration file.

  2. To restore iSCSI initiator port IP addresses, use the mkip command. All the IP addresses belonging to the storage portset (i.e. portset3) needs to be configured. Find out the IP addresses whose "portset_name" property matches with portset3 in the node_ethernet_ip section from the backup configuration file and restore the IP addresses.

    1. To restore IP address, determine port_ID, node_name, portset_name, IP address, prefix, and vlan of the IP address that belongs to portset3 from the configuration backup file, and run the following command: 
      mkip -node node_id_or_name -port port_id -portset portset_id_or_name -ip ip_address -prefix prefix -vlan vlan

      Complete this step for all (earlier configured) IP ports that belong to portset3 in the node_ethernet_ip section from the backup configuration file.

  3. Next, detect, and add the iSCSI storage port candidates by using the detectiscsistorageportcandidate and addiscsistorageport commands. Make sure that you detect the iSCSI storage ports and add these ports in the same order as you see them in the configuration backup file. If you do not follow the correct order, it might result in a T4 failure. You must repeat these steps for all the iSCSI sessions that are listed in the backup configuration file exactly in the same order.

    1. To detect iSCSI storage ports, determine src_port_id, IO_group_id (optional, not required if the value is 255), target_ipv4/target_ipv6 (the target IP that is not blank is required), iscsi_user_name (not required if blank), iscsi_chap_secret (not required if blank), and site (not required if blank) from the configuration backup file, run the following command: 
      svctask detectiscsistorageportcandidate -srcportid src_port_id -iogrp IO_group_id 
-targetip/targetip6 target_ipv4/target_ipv6 -username iscsi_user_name -chapsecret iscsi_chap_secret -site site_id_or_name

      Where src_port_id is the source Ethernet port ID of the configured port, IO_group_id is the I/O group ID or name being detected, target_ipv4/target_ipv6 is the IPv4/IPv6 target iSCSI controller IPv4/IPv6 address, iscsi_user_name is the target controller username being detected, iscsi_chap_secret is the target controller chap secret being detected, and site_id_or_name is the specified ID or name of the site being detected.

    2. Match the discovered target_iscsiname with the target_iscsiname for this particular session in the backup configuration file by running the lsiscsistorageportcandidate command, and use the matching index to add iSCSI storage ports.

      Run the svcinfo lsiscsistorageportcandidate command and determine the ID field of the row whose target_iscsiname matches with the target_iscsiname from the configuration backup file. This is your candidate_id to be used.

    3. To add the iSCSI storage port, determine IO_group_id (optional, not required if the value is 255), site (not required if blank), iscsi_user_name (not required if blank in backup file), and iscsi_chap_secret (not required if blank) from the configuration backup file, provide the target_iscsiname_index matched and then run the following command: 
      addiscsistorageport -iogrp iogrp_id -username iscsi_user_name -chapsecret iscsi_chap_secret -site site_id_or_name candidate_id

      Where iogrp_id is the I/O group ID or name that is added, iscsi_user_name is the target controller username that is being added, iscsi_chap_secret is the target controller chap secret being added, and site_id_or_name specified the ID or name of the site that is being added.

    4. If the configuration is a stretched system, the controller name and site needs to be restored. To restore the controller name and site, determine controller_name and controller site_id/name from the backup json file by matching the inter_WWPN field with the newly added iSCSI controller, and then run the following command: 
      chcontroller -name controller_name -site site_id/name controller_id/name

      Where controller_name is the name of the controller from the backup json file, site_id/name is the ID or name of the site of iSCSI controller from the backup json file, and controller_id/name is the ID or current name of the controller.

Running the automated restore process

Issue the following CLI command to compare the current configuration with the backup configuration data file: 
svcconfig restore -prepare
This CLI command creates a log file in the /dumps/svcconfig directory of the configuration node. The name of the log file is svc.config.restore.prepare.log.
Note: It can take up to a minute for each 256-MDisk batch to be discovered. If you receive error message CMMVC6200W for a MDisk after you enter this command, all the managed disks (MDisks) might not be discovered yet. Allow a suitable time to elapse and try the svcconfig restore -prepare command again.
Issue the following command to copy the log file to another server that is accessible to the system:
pscp superuser@cluster_ip:/dumps/svcconfig/svc.config.restore.prepare.log 
full_path_for_where_to_copy_log_files

Check the log file for errors. If you find errors, correct the condition that caused the errors and reissue the command. You must correct all errors before you can proceed.

Issue the following CLI command to restore the configuration:
svcconfig restore -execute
This CLI command creates a log file in the /dumps/svcconfig directory of the configuration node. The name of the log file is svc.config.restore.execute.log.
Issue the following command to copy the log file to another server that is accessible to the system:
pscp superuser@cluster_ip:/dumps/svcconfig/svc.config.restore.execute.log 
full_path_for_where_to_copy_log_files
Check the log file to ensure that no errors or warnings occurred.
Note: You might receive a warning that states that a licensed feature is not enabled. This message means that after the recovery process, the current license settings do not match the previous license settings. The recovery process continues normally and you can enter the correct license settings in the management GUI later.
When you log in to the CLI again over SSH, you see this output:
IBM_MTM:your_cluster_name:superuser>

Manual actions to complete the restore

The following actions may be required, depending on system configuration:

System Certificates

During recovery, the system creates a new internal root CA and system certificates. If the certificate was stored in the truststore on other systems, the new certificate must be exported and the other truststores update.

If you have an SNMP server that is configured with a certificate in the truststore, then the truststore must be manually set up again.

If the system uses Thales key servers to manage encryption keys, the new root certificate must be exported and installed on the key servers before the configuration restore operation prepares successfully. If the system uses IBM key servers, then the new system certificate must be exported and installed on the key servers. If the previous system was using a certificate that is signed by a third-party CA, then it might also be necessary to get the new system's certificate signed.

If the system is using multifactor authentication then the new system certificate must be exported and installed as a signer certificate in IBM Security Verify, by using the system's id_alias as the friendly name. To export the new system certificate, enter the CLI command:
chsystemcert -export
To find the id_alias of the system, enter the CLI command:
lssystem | grep id_alias
Multi-factor authentication

Follow the steps for exporting the system certificate as described in Configuring multifactor authentication with IBM Security Verify.

If the system is using multifactor authentication with IBM Security Verify, the OpenID Connect and API credentials must be restored manually. For more information, see chauthmultifactorverify.

If the system is using multifactor authentication with Duo Security, the OpenID Connect and API credentials must be restored manually. For more information, see chauthmultifactorduo.

Single sign-on

If the system is using single sign-on, the OpenID Connect credentials must be restored manually. For more information, see chauthsinglesignon.

IP Quorum

If IP Quorum is used, re-enable by downloading a Java application from the Settings > System > IP Quorum tab in the GUI, and then installing the application on the host server.

Cloud Service Providers

After a T4 recovery, cloud accounts are in an offline state. It is necessary to re-enter the authentication information to bring the accounts back online

If you use USB flash drives to manage encryption keys, the T4 recovery causes the connection to a cloud service provider to go offline if the USB flash drive is not inserted into the system. To fix this issue, insert the USB flash drive with the current keys into the system.

If you use key servers to manage encryption keys, the T4 recovery causes the connection to a cloud service provider to go offline if the key server is offline. To fix this issue, ensure that the key server is online and available during T4 recovery.

If you use both key servers and USB flash drives to manage encryption keys, the T4 recovery causes the connection to a cloud service provider to go offline if the key server is offline. To fix this issue, ensure that both the key server is online and a USB flash drive is inserted into the system during T4 recover.

Other features
The following features must be configured manually. See guidance in the linked pages. The configuration backup files may provide further details to assist configuration.

Completing the restore procedure

Verify that the quorum disks are restored to the MDisks that you want by using the lsquorum command. To restore the quorum disks to the correct MDisks, issue the appropriate chquorum CLI commands.

If the superuser account was locked on the previous configuration, you must manually lock it now. Use the CLI command:
chuser -lock superuser

The system configuration has now been restored and data can be copied back to the system if required.