Reverting a master server of a Multi-Master cluster to a stand-alone server

Run the revertMasterToStandalone utility to revert a master server of a Multi-Master cluster, which is in inconsistent or unreachable state, to a stand-alone server. You must run this utility only for recovery scenarios and when the standard Multi-Master operations do not meet your requirement.

About this task

Sometimes one or more master servers might get into inconsistent state when a Multi-Master operation fails without completing the required configurations or because of a network or host system issue. The Multi-Master cluster then might become non-functional.

You must first try to remove the inconsistent master server from the cluster by using Removing a master server from Multi-Master cluster. If removing the master server fails, you can run this utility to clean up the Multi-Master configuration and revert the inconsistent or unreachable master server to a stand-alone server.

Location of the utility

The revertMasterToStandalone utility is available in the agent folder under the SKLM_INSTALL_HOME directory.

Default file location:
Windows
drive:\Program Files\IBM\GKLMV421\agent
Linux and AIX
opt/IBM/GKLMV421/agent
Utility file name:
  • Windows: revertMasterToStandalone.bat
  • Linux or AIX: revertMasterToStandalone.sh

The directory also contains the revertMasterToStandalone.properties file. This file is needed as an input to the utility. Specify the following properties in the file:

Table 1. Property descriptions
Property Description
WAS_HOME Default directory path:
Windows
drive:\Program Files\IBM\WebSphere\Liberty
Linux and AIX
path/IBM/WebSphere/Liberty
JAVA_HOME Optional.

Default directory path:

Windows systems
C:\Program Files\IBM\WebSphere\Liberty\java\jre
Linux and AIX systems
/opt/IBM/WebSphere/Liberty/java/jre
DB_PASSWORD Db2 administrator user password in plain text.

The utility does not validate the password before the run, and will fail in case it is incorrect.

INITIAL_BACKUP_RESTORE_REQUIRED Indicate the required state of the database after the revert operation. You can provide one of the following values:
  • YES: To revert the database to the initial state, that is when the cluster was created.
    Note: Only the primary master server's database contains data in the initial state. The initial backup of the database of a non-primary master server is always empty.
  • NO: To retain the database as is.

Procedure

Complete these steps on the master server that you want to revert to a stand-alone server:

Note: If you plan to run the utility on a Linux host system, ensure that the Db2 administrator user, who is the owner of the IBM Security Guardium Key Lifecycle Manager processes, runs this utility.

  1. Ensure that the revertMasterToStandalone.properties file is updated according to your configuration.
  2. Open the command line.
  3. Navigate to the SKLM_HOME/agent directory.
  4. Run the utility.
    The master server is reverted to a stand-alone server.
  5. Complete the steps in Removing a master server from Multi-Master cluster to remove the entry of the reverted master server from the cluster.
    This step is not required if you revert all the master servers of the cluster.

Results

The operations of the utility are logged in the SKLM_HOME/agent/revertMasterToStandalone.log file.
Important: After reverting the master server with this utility, clear the passwords in the revertMasterToStandalone.properties file.
Troubleshooting: Complete the given steps if the utility fails on the standby master server with the following error:
ERROR: Corrupt HADR state
Resolution steps:
  1. Check whether the database is in the REMOTE_CATCH_PENDING state. To do so, in the Db2 command window, run the db2pd -db sklmdb41 -hadr command and check the HADR state.

    In the REMOTE_CATCH_PENDING state, the Db2 log files are not synchronized between the primary and standby master server.

  2. Copy the log files from the primary master server to the standby master server being reverted.

    To identify the log location, run the db2 get db cfg for SKLMDB41|grep LOGARCH command.

    From the response, "First log archive method" gives the location of the log files on the disk. Manually copy the log files from this location on the primary master server to the standby master server.

  3. To synchronize the log files, on the Db2 command window, run the Roll Forward Pending command: db2 rollforward db sklmdb41 to end of logs and complete.
For more information, see https://www.ibm.com/support/pages/after-performing-database-restore-sql1273n-or-sql1263n-returns-during-rollforward.