Preparing the system for recovery
To perform a system recovery, start with preparing the system for recovery.
Before you attempt this system recovery procedure, investigate the cause of the failure and attempt to resolve those issues by using other fix procedures. Read and understand all of the instructions before you complete any action.
- Ensure that all backend storage is administered by the system is present.
- If the system was using IP quorum for T3 metadata, verify that all the IP quorum applications are running.
- Ensure that the web browser is not blocking pop-up windows. If it does, progress windows cannot open.
- Ensure that all enclosures are powered up and can communicate with each other.
- Check that all nodes in the system are shown in the service assistant tool or using the service command: sainfo lsservicenodes. Investigate any missing nodes.
- Resolve all hardware errors in nodes so that only node errors 578 or 550 are present. For more information, see Fix hardware errors.
- Remove the system information for node canisters with error code 550 or error code 578 by using the service assistant, but only if the recommended user responses for these node errors are followed. For more information, see Removing system information for nodes with error code 550 or error code 578 using the service assistant.
- If any nodes have been replaced, ensure that the WWNN of the replacement node matches that of the replaced node, and that no prior system data remains on this node.
- If any nodes are at a different hardware level due to previous steps, reinstall the code on those nodes so that it matches the system code level at the time of failure.
Systems using encryption
If USB flash drives encryption was used, ensure that USB flash drive keys are plugged into at least one node. Use this node to perform the recovery.
If the system contains an encrypted cloud account that uses USB flash drives encryption, a USB flash drive with the system master key must be present in the configuration node before the cloud account can move to the online state. This requirement is necessary when the system is powered down, and then restarted.
- Run the recovery on a node that is attached to the key server. The keys are fetched remotely from the key server.
- Run the recovery procedure on a node that is not hardware that is replaced or node that is rescued. All of the information that is required for a node to successfully fetch the key from the key server resides on the node's file system. If the contents of the node's original file system are damaged or no longer exist (rescue node, hardware replacement, file system that is corrupted), then the recovery fails from this node.
If the system uses both USB flash drives and key server encryption, providing either a USB flash drive or a connection to the key server (only one is needed, but both will also work) will unlock the system.
If you use USB flash drives to manage encryption keys, the T3 recovery causes the connection to a cloud service provider to go offline if the USB flash drive is not inserted into the system. To fix this issue, insert the USB flash drive with the current keys into the system.
If you use key servers to manage encryption keys, the T3 recovery causes the connection to a cloud service provider to go offline if the key server is offline. To fix this issue, ensure that the key server is online and available during T3 recovery.
If you use both key servers and USB flash drives to manage encryption keys, the T3 recovery causes the connection to a cloud service provider to go offline if none of the key providers are available. To fix this issue, ensure that either the key server is online or a USB flash drive is inserted into the system (only one is needed, but both will also work) during T3 recovery.
Next step
All nodes, including hot spare nodes, should now be in candidate status, and you can proceed with Running the system recovery.