Changing IP addresses or host names of cluster nodes

IBM Storage Scale assumes that the IP addresses and host names of cluster nodes remain constant. In the rare event that a change is necessary, follow the steps in this topic.

Changing the IP addresses or host names of cluster nodes might be necessary either for external reasons or because of an inadvertent action, such as reinstalling a node with a disk image from a different node.

First scenario: All the nodes in the cluster are affected
Second scenario: Some of the nodes in the cluster are affected
Updating IP addresses or host names in other configurations

Important:

For both of the scenarios that are described in this topic, most of the quorum nodes in the cluster must be up and must be reachable through either the old IP addresses or the new IP addresses.
For any IP addresses or host names that are changed, be sure to update the IP addresses or host names as needed in other IBM Storage Scale configurations, such as performance monitoring and call home groups. For more information, see the Updating IP addresses or host names in other configurations procedure.

First scenario: All the nodes in the cluster are affected

Follow the steps in this subtopic if both of the following conditions are true:

The IP address or host name of all of the nodes in the cluster has changed.
One of the following conditions is true:
- The minimum release level of the cluster is 5.1.0 or later.
- The minimum release level of the cluster is earlier than 5.1.0, but you can make all of the old host names or IP addresses active on the network.

Otherwise, follow the steps in the next subtopic.

Issue the following command to stop IBM Storage Scale on all the nodes in the cluster:
```
mmshutdown -a
```
Using the documented procedures for the operating system, add the new host names or IP addresses to the network but do not remove the old ones yet. Make sure that the new host names are returned first by your DNS service when there are multiple names for the same IP address. Do not restart the GPFS daemon on the nodes yet.
Follow the documented operating system procedures to make the old host names or IP addresses active on the network without causing network conflicts with the new host names and IP addresses. For example, you might be able to create temporary network aliases for the old IP addresses or host names.

In some cases you might not be able to make the old host names or IP addresses active on the network. For example, if a node has only a single adapter, then it cannot have both the old host name or IP address and the new one active on the network. In such cases, specify the --force option when you run the mmchnode command in Step 4.
Issue the mmchnode command to update the administration interface and daemon interface of the nodes whose IP addresses or host names have changed:

Note: If the minimum release level of the cluster is earlier than 5.1.0, the daemon interface of a quorum node cannot be changed while CCR is enabled. To resolve this problem, temporarily change the quorum node to a nonquorum node, change the daemon interface of the nonquorum node, and change the nonquorum node back to a quorum node.
1. Create a specification file for the mmchnode command. Add to the file the interface changes that need to be made for each node. For example, the specification file /tmp/specfile lists nodes node-6, node-7, and node-8 and specifies the new daemon interface and the new administrative interface for each node:
```
node-6 --daemon-interface=node-6-2.new-localnet.com --admin-interface=node-6-2.new-localnet.com
node-7 --daemon-interface=node-7-2.new-localnet.com --admin-interface=node-7-2.new-localnet.com
node-8 --daemon-interface=node-8-2.new-localnet.com --admin-interface=node-8-2.new-localnet.com
```
2. Issue the mmchnode command to apply the changes in the specification file:
```
mmchnode --spec-file /tmp/specfile
```
  If you could not make all the host names or IP addresses active on the network in Step 3, issue the mmchnode command with the --force option:
```
mmchnode --spec-file /tmp/specfile --force
```
If the IP addresses over which the subnets attribute is defined are changed, you must update the subnets attribute to include the new addresses. To do so, issue the mmchconfig command and set the subnets attribute to the new specification.
If SMB is enabled, complete the Step 3 in the Updating IP addresses or host names in other configurations procedure before issuing the following command to start the GPFS daemon on all nodes:
```
mmstartup -a
```
Remove the old host names and IP addresses from the network.
Be sure to update any other IBM Storage Scale configurations that might contain the old IP addresses or host names. For more information, see the Updating IP addresses or host names in other configurations procedure.

Second scenario: Some of the nodes in the cluster are affected

Follow the steps in this subtopic if your situation does not match the conditions that are required for the first scenario.

Before changing any host names or IP addresses, do the following actions:
1. Issue the mmshutdown command to stop GPFS on all affected nodes.
2. If the host names or IP addresses of the primary or secondary GPFS cluster configuration server nodes must change, use the mmchcluster command to specify another node to serve as the primary or secondary GPFS cluster configuration server.
3. If the host names or IP addresses of an NSD server node must change, temporarily remove the node from being a server with the mmchnsd command. Then, after the node has been added back to the cluster, use the mmchnsd command to change the NSDs to their original configuration. Use the mmlsnsd command to obtain the NSD server node names.
4. If the affected node is a Cluster Export Services (CES) node, you must disable CES on the node. To do so, issue the following command:
```
mmchnode -N <node> --ces-disable
```
  Note: If you disable all the CES nodes in the cluster, the CES configuration is lost, including the exports. For more information, see Implementing Cluster Export Services.
5. Unless all nodes in the cluster are being deleted, ensure that the mmdelnode command is run from a node that remains in the cluster.
Change the node names and IP addresses using the documented procedures for the operating system.
If the IP addresses over which the subnets attribute is defined are changed, you must update the subnets attribute to include the new addresses. To do so, issue the mmchconfig command and set the subnets attribute to the new specification.
Issue the mmaddnode command to restore the nodes to the GPFS cluster.
If necessary, use the mmchcluster, mmchlicense, and mmchnsd commands to restore the original configuration and the NSD servers.

Note: You can use the mmchnode command if you need to re-enable CES on the node.
If SMB is enabled and an IP is changed on any of the CES nodes, complete the following steps:
1. Issue the mmshutdown command on all the CES nodes.
2. Complete the Step 3 in the Updating IP addresses or host names in other configurations procedure.
3. Issue the mmstartup command on all the CES nodes.
Issue the mmstartup command to start the GPFS daemon on all nodes that you shut down in the Step 1a.
Be sure to update any other IBM Storage Scale configurations that might contain the old IP addresses or host names. For more information, see the Updating IP addresses or host names in other configurations procedure.

Updating IP addresses or host names in other configurations

After you change the IP address or host name of a node, ensure that you update other configurations that might contain the old IP address or host name, such as the following configurations:

Performance monitoring
Call home groups
The CTDB nodes file for CES

Update the performance monitoring configuration if needed:
1. Issue the following command to save the current performance monitoring configuration file into a temporary file:
```
mmperfmon config show --config-file <temporary-file-name>
```
2. Open the temporary file in a text editor and change any occurrence of an old node name or IP address to the new one.
3. If you changed a node name or IP address in Step 2, issue the following command to update the performance monitoring configuration:
```
mmperfmon config update --config-file <temporary-file-name>
```
4. If the performance monitoring configuration is a federation, and affected node is a collector, and you are running a version of IBM Storage Scale that is earlier than 5.0.2, update the names and IP addresses of the peers in the /opt/IBM/zimon/ZIMonCollector.cfg file on all the collector nodes.
If the long admin node names (FQDN) of any call home group members are changed, you must delete the affected call home groups and create new ones:
1. Issue the mmcallhome group list command to check the status of nodes that are members of call home groups. If the command displays ------ instead of the name of a node, then the node is deleted or its FQDN (including the domain) has changed.
2. If a node is deleted or its FQDN has changed, delete its call home group and create a new one. For more information, see mmcallhome command.
Update the cluster trivial database (CTDB) nodes file in the clustered configuration repository (CCR) to include new host IPs in the file.
1. Store the smb.ctdb.nodes file from the CCR to a temporary location by issuing the following command:
```
mmccr fget smb.ctdb.nodes /tmp/smb.ctdb.nodes 
```
2. Change the IPs accordingly in the /tmp/smb.ctdb.nodes file. The order of IPs must match the previous order of nodes, and must not be changed.
  
  Note: These IPs are host IPs, and must not be CES IPs.
3. Send the updated /tmp/smb.ctdb.nodes file to the CCR by issuing the following command:
```
mmccr fput smb.ctdb.nodes /tmp/smb.ctdb.nodes
```