How to offline change IP address on pureScale ?

Body

(We have technote below to online change IP address , customer followed the steps , it's not progress unfortunately .)

/support/pages/node/475275

Following procedure is got by working with GPFS team and has already validated by customer ,  note that this is a completely offline operation .

1. As an instance user, perform the following steps:

    1). Stop the database manager on all hosts by issuing the following command on a single host:

    su -iname
    db2stop
    exit

    where iname represents the instance owner name.
    2). On each host, stop the DB2 instance by issuing the following command:

    db2stop instance on hostname

    where hostname represents the host name for a given member or CF, and the db2stop instance on hostname command is run for each host in the cluster.

2. As the DB2 cluster services administrator (root account), perform the following steps on a single host :

    1). Put the cluster manager into maintenance mode on all hosts by issuing the following command:

    DB2DIR/bin/db2cluster -cm -enter -maintenance -all

    where DB2DIR represents the installation location of your DB2 copy.

   Use following command to check the CM state       
   DB2DIR/bin/db2cluster -cm -list -host -state

    2). Put the cluster file system service on the hosts into maintenance mode by issuing the following command:

    DB2DIR/bin/db2cluster -cfs -enter -maintenance -all

    where DB2DIR represents the installation location of your DB2 copy.

    Use following command to check state of cluster file system  service      
     DB2DIR/bin/db2cluster -cfs -list -host -state

3. Perform maintenance work .
    
   1)Update the /etc/hosts file on all hosts in the domain to reflect the new IP and ensure the DNS server is updated as well.
 
 
4. On each host , update the GPFS configuration information

/usr/lpp/mmfs/bin/mmchnode --admin-interface <hostname> --daemon-interface <hostname> -N <hostname>

Note: During the process , if we hit the error like following , we will need to set up IP alias by using OS command,

hacmpts02:/opt/IBM/V10.55/bin# /usr/lpp/mmfs/bin/mmlscluster
mmlscluster: 6027-2188 Unable to determine the local node identity.
mmlscluster: 6027-1639 Command failed. Examine previous error messages to
determine cause.  
2:23:33 PM
hacmpts02:/opt/IBM/V10.55/bin# /usr/lpp/mmfs/bin/mmchnode --admin-interface
hacmpts02 --daemon-interface hacmpts02 -N hacmpts02
mmchnode: 6027-2188 Unable to determine the local node identity.
mmchnode: 6027-1639 Command failed. Examine previous error messages to
determine cause

Remove IP alias after see the IP address has been changed successfully .

Use command /usr/lpp/mmfs/bin/mmlscluster to check whether the IP has been changed successfully

5. Exit maintenance mode

   1) Exit cluster manager maintenance mode on the host by issuing the following command:

    DB2DIR/bin/db2cluster -cm -exit -maintenance -all

    where DB2DIR represents the installation location of your DB2 copy.
    Ensure that all members and domain are active by issuing the following command:

    DB2DIR/bin/db2cluster -cm -list -host -state

    where DB2DIR represents the installation location of your DB2 copy.
    2) Exit cluster file system maintenance mode on the hosts by issuing the following command:

    DB2DIR/bin/db2cluster -cfs -exit -maintenance -all

    where DB2DIR represents the installation location of your DB2 copy. In case of timeout, check the status of the cluster file system by issuing the DB2DIR/bin/db2cluster -cfs -list -host -state command.

  DB2DIR/bin/db2cluster -cfs -list -host -state

 
6. Post IP change configuration updates for CFS network resiliency.

   1) As root on the updated host, run the following command:
     db2cluster -cfs -network_resiliency -repair

   2) To ensure there are no “Error” entries in the third column for any of the hosts on which the IP address was changed, issue the following as root:

lscondition -l | grep condresp

Example output:

host19: # lscondition -l | grep condresp
“condrespV98_cond_host16_en0" “host16.torolab.ibm.com” “Monitored”
“condrespV98_cond_host17_en0” “host17.torolab.ibm.com” “Monitored”
“condrespV98_cond_host18_en0" “host18.torolab.ibm.com” “Monitored”
“condrespV98_cond_host19_en0” “host19.torolab.ibm.com” “Monitored”

If the row for the updated host contains “Error” instead of “Monitored”, repeat step 7A.

3) Note: If the IP address change also involved changing the default gateway, the /var/ct/cfg/netmon.cf file needs to updated to reflect this change. Ensure that the IP address of the default gateway for the adapter is correct.

Example: in this cluster, the default gateway for the en0 adapters is 9.23.1.1

host16:/var/ct/cfg >cat netmon.cf
!IBQPORTONLY !ALL
!REQD en0 9.23.1.1

Do not modify any other contents of the file, except the IP address of the affected adapter.

7. Restart the instance.

1). If smoothly, we can directly start the instance

 As an instance user, perform the following steps:

    On each host, start the DB2 instance by issuing the following command:

    db2start instance on hostname

    where hostname represents the host name for a given member or CF, and the db2start instance on hostname command is run for each host in the cluster.
    Start the database manager by issuing the following command:

    su -iname
    db2start
    exit

    where iname represents the instance owner name.

Or 2) At this point, if any members or CFs are online, stop the entire instance.
db2stop force

This step may timeout and produce a SQL1692N error code. If this occurs, please verify that the instance is stopped using the db2instance -list command. If the instance is not stopped, re-issue the db2stop force command.

b) As instance owner, repair the cluster resources.
db2cluster -repair -resources

c) Verify that the instance resources have been properly recreated.
db2cluster -verify -resources

d) Start the instance on the host that was stopped.
db2start instance on <host>

e) Start the instance.
db2start