How to Recover From 564 Node Errors After Installing the SAS Host Interface Card

Resolving The Problem

Recovery if there are no offline volumes in the IO group:
It is important to check for offline volumes before removing a node, as doing so may result in loss of hardened cache data.

1. Confirm there are no offline volumes in the IO group
2. Remove the node canister from the cluster using the management GUI, or by running the following CLI command:
   
   rmnodecanister <nodeid>
   
3. Use the service assistant GUI on the node to force the node to leave cluster
4. Use the service assistant GUI to reboot the node
5. The node should return in the Candidate state and automatically re-join the cluster. If the node does not re-join and instead reports node error 690, run the following CLI command to force the node to re-join:
   
   satask stopservice <nodepanelname>
   
6. Confirm via the management GUI that the new SAS HIC has been accepted

1. Confirm that offline volumes exist in the IO group
2. Power down the node canister by running the following CLI command:
   
   satask stopnode -poweroff <nodepanelname>
   
3. Remove the newly added SAS HIC
4. Allow the node to restart and join the cluster
5. Confirm that there are no longer any offline volumes
6. Remove the node canister from the cluster using the management GUI, or by running the following CLI command:
   
   rmnodecanister <nodeid>
   
7. Use the service assistant GUI on the node to force the node to leave cluster
8. Use the service assistant GUI to power off the node
9. Remove the node from its enclosure and install the SAS HIC
10. Restore the node to the enclosure
11. The node should return in the Candidate state and automatically re-join the cluster. If the node does not re-join and instead reports node error 690, run the following CLI command to force the node to re-join:
    
    satask stopservice <nodepanelname>
    
12. Confirm via the management GUI that the new SAS HIC has been accepted