IBM Support

Multi-Master cluster configuration scenarios

General Page

IBM Security Guardium Key Lifecycle Manager Multi-Master cluster configuration scenarios

Note: With V4.1, IBM Security Key Lifecycle Manager is renamed as IBM Security Guardium Key Lifecycle Manager.

 

The following scenarios demonstrate how to apply technology to accomplish business goals and solve problems that are associated with a Multi-Master cluster configuration.

The user IDs, names, and passwords that are used in these scenarios are examples only.

To migrate a Multi-Master cluster in inline mode

Follow these steps to migrate a Multi-Master cluster from an earlier version (source) of IBM Security Guardium Key Lifecycle Manager to a later version (target) when the host system of the target version is the same as the host system of the earlier version. You can choose to not migrate the Multi-Master configuration but only migrate the master servers to the newer version.

Migrating an IBM Security Guardium Key Lifecycle Manager Multi-Master cluster in inline mode:

Planning the Multi-Master cluster migration

  • Review the supported upgrade paths and migration methods. See Upgrading: Installing IBM Security Guardium Key Lifecycle Manager in graphical mode.
  • Review the requirements and considerations for configuring a Multi-Master cluster. See Requirements and considerations for Multi-Master configuration.
  • Ensure that the configuration of all the master servers in the Multi-Master cluster is correct. For example, check for any discrepancy in the actual role of the master server and the role that is configured in the SKLMConfig.properties file (for 3.0.1 and earlier versions) or the MMConfig.properties file (for 4.0.0.0 and newer versions). If a master server is acting primary, ensure its role in the configuration file is Primary. Else, the inline migration fails.
  • Note down the HTTPS ports that you plan to use for the master servers in the migrated cluster.
  • If a master server in the cluster is running the Linux® operating system, ensure that the permissions for the /tmp directory on the server is set to 777 that is full execute, read, and write permissions.
  • Ensure that the JAVA_HOME environment variable is set to point to the directory where WebSphere Application Server Liberty Java is located. For example, C:\Program Files\IBM\WebSphere\Liberty\java\8.0.

  • Create a properties file on the primary master server to store the configuration details of the Multi-Master cluster.

    Notes: 
    Ensure that the host name in the mmsetup.properties file is same as the IP_Hostname column value in the KMT_MM_INSTANCE table.
    If you do not specify the PRIMARY_HADR_PORT value in the file, the default port value 60028 is used.
    • In a temporary directory, create the mmsetup.properties file. Example of temporary directory:

Windows

  • %temp%\gklm

Linux

  • $TMPDIR
  • In the mmsetup.properties file, include the hostnames and HTTPS ports that you plan to use for the master servers in the migrated cluster. For the latest instance, ensure that the port number that is already in use or the previously used port number is not specified.
  • Ensure that you specify the same hostnames that are provided in the cluster configuration of the source IBM Security Guardium Key Lifecycle Manager master server.

  • Sample mmsetup.properties file:

    
PRIMARY_HTTP_PORT=9444
PRIMARY_IP_HOSTNAME=myprimaryhost
PRIMARY_HADR_PORT=60028 (optional)
STANDBY_1_HTTP_PORT=9444
STANDBY_1_IP_HOSTNAME=mystandbyhost1 
STANDBY_2_HTTP_PORT=9444
STANDBY_2_IP_HOSTNAME=mystandbyhost2 
NODE_1_HTTP_PORT=9444
NODE_1_IP_HOSTNAME=mynonhadrhost1 
NODE_2_HTTP_PORT=9444
NODE_2_IP_HOSTNAME=mynonhadrhost2
  • During the inline migration process, only the master servers that are correctly specified in the mmsetup.properties file are added to the migrated Multi-Master cluster.
  • If you are performing inline migration of a Multi-Master cluster from V4.1.0.0 to V4.1.1.0 on a Windows system, complete the following steps to remove the duplicate Db2® service entry:
    1. Go to the C:\Windows\System32\drivers\etc\ directory and open the services file in edit mode.

    2. Remove duplicate entry for the Db2 service:

       DB2_db2_instance_name db2_port/tcp

      For example:

      DB2_sklmdb41 60000/tcp
    3. Save the services file and close it.
  • Ensure that any new or incoming client device communication certificates appear as pending for acceptance to allow secure communication between the device and the server. To do so, navigate to Configuration > Key Serving Parameters page and ensure that Keep pending client device communication certificates check box is selected.

Back to top

 

Migrating the master servers in the Multi-Master cluster in inline mode

  1. Migrate all the standby and non-HADR master servers in the IBM Security Guardium Key Lifecycle Manager Multi-Master cluster.

    For instructions, see Upgrading: Installing IBM Security Guardium Key Lifecycle Manager in graphical mode or Upgrading: Installing IBM Security Guardium Key Lifecycle Manager silently.

  2. Ensure that all the migrated standby and non-HADR master servers are up and running.

  3. Migrate the primary master server.

    For instructions, see Upgrading: Installing IBM Security Guardium Key Lifecycle Manager in graphical mode or Upgrading: Installing IBM Security Guardium Key Lifecycle Manager silently.

Back to top

Completing the post-migration tasks

  • Verify that all the configuration properties are correctly updated in the SKLM_HOME/config/SKLMConfig.properties file on the IBM Security Guardium Key Lifecycle Manager target server.

    If you migrated to 4.1.1.6 or later versions, verify that all the configuration properties are also updated in the WAS_HOME/products/sklm/config/MMConfig.properties file. 

    Important: Update the value of the TransportListener.ssl.protocols property in the SKLMConfig.properties file to TLSv1.2 and ensure that it is the same on all the master servers. Use the Update Config Property REST Service to update the property value.
  • Verify that all the master servers are correctly added in the newly created cluster, and check their Db2 HADR configuration status. For instructions, see Viewing the configuration status of all master servers.
  • If any master servers of the earlier Multi-Master cluster are not automatically configured in the new cluster, add them by using one of the following methods:
    • Run the createCluster.bat or createCluster.sh script file. The file is located in the following path:

Windows

  • SKLM_HOME\migration\bin

Linux

  • SKLM_HOME/migration/bin
    Note: The script file uses the mmsetup.properties file as input. Ensure that the mmsetup.properties file is correctly updated.

    Run the command as follows:

    Windows

    createCluster.bat

    Linux

    ./createCluster.sh
  • Add the master servers manually. For instructions, see Adding a non-HADR master server to a cluster.
  • Review the logs for the cluster migration. The logs are stored in the temporary directory.

Windows

  • %temp%/cluster_migration.log

Linux

  • $TMPDIR/cluster_migration.log

If the CTGKM1016E TCP port not available error message is displayed on the graphical user interface of a master server, complete the following steps on the master server:

  1. Stop the earlier version of IBM Security Guardium Key Lifecycle Manager. For instructions, see Stopping the IBM Security Guardium Key Lifecycle Manager server.
  2. Restart the target version of IBM Security Guardium Key Lifecycle Manager. For instructions, see Restarting the Guardium Key Lifecycle Manager server.

Back to top

(Optional) Removing the source cluster

  1. Verify that the clients are working with the new cluster.
  2. Uninstall the earlier version of IBM Security Guardium Key Lifecycle Manager on the master servers. For instructions, see the uninstallation topic for that version in the IBM Documentation. For example, to uninstall version 3.0, see Uninstallation of IBM Security Guardium Key Lifecycle Manager.

Back to top

To cross-migrate a Multi-Master cluster

Follow these steps to migrate a Multi-Master cluster from an earlier version (source) of IBM Security Guardium Key Lifecycle Manager to a later version (target) when the host system of the target version is different than the host system of the existing version.

Complete these steps:

  1. Take a backup of the primary master server of the source Multi-Master cluster.
  2. Identify servers to create the new Multi-Master cluster.
  3. Install the target version of IBM Security Guardium Key Lifecycle Manager on the servers that you identified in Step 2. See Installing.
  4. Identify one of the servers on which you installed the target version as the primary master server for the new cluster.
  5. On the server that you identified as the new primary master server, perform these steps:
    1. Restore the backup of the source primary master server.
    2. In the SKLMConfig.properties file, update the TransportListener.ssl.protocols property to the value: TLSv1.2. You can use the Update Config Property REST Service to update the property.
    3. Stop the IBM Security Guardium Key Lifecycle Manager Agent.
    4. Restart the IBM Security Guardium Key Lifecycle Manager server.

    5. (Only applicable to cross-migration by using the utility scripts) During the restore process, if you specified the RESTORE_USER_ROLES property as RESTORE_USER_ROLES=y in the restoreVversion utility (for example, restoreV25.bat), refresh the user credentials on the IBM Security Guardium Key Lifecycle Manager server:

      e.1. Log in to the IBM Security Guardium Key Lifecycle Manager graphical user interface. Click Administration > Multi-Master.

      e.2. On the page, click the Multi-Master link and in the Confirm dialog, click OK.

      e.3. In the Masters table, select the master server and click Modify Master.

      e.4. In the Multi-Master Configuration - Modify Master window, specify the values for the IBM Security Guardium Key Lifecycle Manager password.

      e.5. Click Accept host certificate automatically and click Update.

      e.6. In the information message dialog, click Close.

      e.7. Click Cancel.

  6. (Only applicable for Version 3.0.0 and 3.0.1) Remove the following properties from the sklmconfig.properties file:
    • multimasterNodes
    • multimasterRole
    • multimasterSetupState
    • multimasterStandbys
    • multimasterPrimary

      These properties are specific to Multi-Master configuration and not required on the stand-alone server. Hence, need to be removed.
      Use the Delete Config Property REST Service to remove the properties.
  7. Configure the new cluster.
    1. Configure a new Multi-Master cluster. See Setting up a Multi-Master cluster.
    2. Add master servers to the new cluster. See Adding a non-HADR master server to a cluster.
    3. Ensure that the new cluster is up and running. See Multi-Master cluster configuration status.
    4. Ensure that any new or incoming client device communication certificates appear as pending for acceptance to allow secure communication between the device and the server. To do so, navigate to Configuration > Key Serving Parameters page and ensure that Keep pending client device communication certificates check box is selected.
  8. Configure the clients (for example, storage devices) that are using the source cluster.
    1. Configure the clients to use the new cluster.
    2. Verify that the clients are working with the new cluster.
  • (Optional): After you ensure that the clients are able to work with the new cluster, you can remove the source cluster. For instructions, see the uninstallation topic for that version in the IBM Documentation. For example, to uninstall version 4.1, see Uninstallation of IBM Security Guardium Key Lifecycle Manager.

Db2 patching on Multi-Master cluster

Complete the following steps to use Db2 patching on Multi-Master cluster.

  1. Stop Agent on primary and standby

    /opt/IBM/GKLMV50/agent/stopAgent.sh /opt/IBM/WebShpere/Liberty

    For more information, see Stop Agent.

  2. Stop Liberty on primary and standby

    /opt/IBM/WebSphere/Liberty/bin/stopServer.sh

  3. Stop HADR on primary and standby

    db2 stop hadr on db klmdb50

  4. Stop database on primary and standby

    db2stop force
db2 terminate

  5. Start patching on primary

    Extract the file installFixPack and run the following command:

    ./installFixPack -b /opt/IBM/DB2GKLMV50

    Optional: Check Db version by the following command:

    db2level

  6. Start patching on standby

    Extract the file installFixPack and run the following command:

    ./installFixPack -b /opt/IBM/DB2GKLMV50

    Optional: Check Db version by the following command:

    db2level

  7. Start Db2 and HADR on standby

    db2start
db2 start hadr on db klmdb50 as standby

  8. Start Db2 and HADR on primary

    db2start
db2 start hadr on db klmdb50 as primary

  9. Start Liberty and Agent on standby and primary

    /opt/IBM/GKLMV50/agent/startAgent.sh /opt/IBM/WebShpere/Liberty

    For more information, see Start Agent.

To configure IBM Security Guardium Key Lifecycle Manager for Suite B compliance

To configure IBM Security Guardium Key Lifecycle Manager to comply with Suite B standards when Multi-Master cluster is set up, complete these steps on every master server of the cluster:

  1. Log in as the database user. For example, in V4.1.0, sklmdb41.
  2. Enable Suite B compliance. See Agent Starter.
  3. Select a certificate that uses the ECDSA algorithm. If a certificate with the ECDSA algorithm is not available, create a new certificate.
    Suite B compliance requires ECDSA certificate for the SSL communication to work.
  4. Create a keystore and add the ECDSA certificate in it. Stop the agent. See Agent Starter.
  5. Browse to the SKLM_INSTALL_HOME/agent directory and run the agentImportKS script to add the certificate in the agent keystore.

    For Windows:
    agentImportKS.bat <WAS_HOME> <Keystore_Type> <Keystore_Path> <Keystore_Password>
    For example: agentImportKS.bat /opt/IBM/WebSphere/AppServer JCEKS /home/user/testks.jks password

    For Linux:
    agentImportKS.sh <WAS_HOME> <Keystore_Type> <Keystore_Path> <Keystore_Password>
    For example: agentImportKS.sh /opt/IBM/WebSphere/AppServer JCEKS /home/user/testks.jks password

    Where,
    <WAS_HOME> - The WebSphere Application Server home directory.
    <Keystore_Type> - Possible values are: JCEKS, PKCS12
    <Keystore_Path> - Path to the keystore file in which you have stored the certificate.
    <Keystore_Password> - Password of the keystore file.
  6. Start the agent. See Agent Starter.

To restore backup of a primary master server of a Multi-Master cluster to a stand-alone server

  • (Only applicable for Version 4.0.0 and later)

    Prerequisite: Ensure that the version of IBM Security Guardium Key Lifecycle Manager on the stand-alone server and the primary master server is the same.
    To restore the backup:
    1. Take a backup of the primary master server or you can use the existing backup file, if it exists.
    2. Restore the backup on the stand-alone server. For more information, see Restoring a backup file.
     

  • (Only applicable for Version 3.0.0 and 3.0.1)

    Prerequisite:
    • Ensure that the backup of the primary master server exists.
    • Ensure that the version of IBM Security Guardium Key Lifecycle Manager on the stand-alone server and the primary master server is the same.
     

    To restore the backup, complete these steps:

    1. On the stand-alone server, take a backup of the sklmconfig.properties file.
    2. Restore backup from the primary master server of the cluster to the stand-alone server. See Restoring a backup file.
    3. In the sklmconfig.properties file on the stand-alone server, remove the following properties:
      • multimasterNodes
      • multimasterRole
      • multimasterSetupState
      • multimasterStandbys
      • multimasterPrimary

        These properties are specific to Multi-Master configuration and not required on the stand-alone server. Hence, need to be removed.
        Use the Delete Config Property REST Service to remove the properties.
    4. If you plan to use the stand-alone server as a primary master server of a cluster, complete the following steps:
      1. Create a new server certificate. For more information, see Creating a server certificate.
      2. Restart the IBM Security Guardium Key Lifecycle Manager server. For more information, see Restarting the server.
      3. Restart the agent service. For more information, see Restarting the agent service.

To restore backup of a stand-alone IBM Security Guardium Key Lifecycle Manager server to the primary master server of a Multi-Master cluster

Prerequisite:
  • Ensure that the backup of the stand-alone server exists.
  • Ensure that the version of IBM Security Guardium Key Lifecycle Manager on the stand-alone server and the primary master server is the same.
 

To restore the backup, complete these steps:

  1. Only for IBM Security Key Lifecycle Manager version 3.0 and 3.0.1: On the primary master server of the cluster, take a backup of the sklmconfig.properties file.
  2. Restore backup from the stand-alone server to the primary master server of the cluster. See Restoring a backup file.
  3. Only for IBM Security Key Lifecycle Manager version 3.0 and 3.0.1: In the sklmconfig.properties file on the primary master server, add the following properties. Use values from the sklmconfig.properties file that you backed up in step 1:
  4. Run the Master Key Transfer REST API. See Master Key Transfer REST Service.
  5. Navigate to the SKLM_HOME/agent/ directory and ensure that the agentStarter.properties file is not empty. If it is empty, wait for the property file to get populated.
    Note: The agentStarter.properties file is populated depending on the value of agent.invoker.polling.interval property. Its default value is 600s.
  6. On all the master servers of the cluster:
    1. Kill the agent process.
    2. Rename the existing agentks.jks and agentts.jks files.
    3. Start agent. See Agent Starter.
  7. On all the master servers of the cluster:
    1. Ensure that the SSL server certificate (for example, cert1) is the same as the one that exists on the primary master server.
    2. Run Update Config Property REST Service to mark the SSL server certificate (for example, cert1) as “In-Use” by setting the property as shown.
      {"config.keystore.ssl.certalias": "cert1"}
    3. Restart the server.

To resolve the issue of expired agent certificate

The Multi-Master section on the Welcome page of the graphical user interface and the Multi-Master page might display Db2 HADR status as red (Db2 HADR status is red) although the Multi-Master cluster is up and running.  Also, the following exception is seen in the agent log, which occurs when the agent certificate has expired:

Thread[Thread-6,5,main] all:com.ibm.sklm.agent.listener.AgentTrustManager isTrusted  ALL:  java.security.cert.CertificateExpiredException: NotAfter:   CertificateExpiredException: NotAfter: 
To resolve this issue:
  1. Stop the agent service. See Stop Agent.
  2. Navigate to SKLM_INSTALL_HOME\agent\ and rename the agentks.jks file.
  3. Start the agent service. See Agent Starter.
A new keystore (agentks.jks file) is created that has the new agent certificate and the Multi-Master configuration status changes to green (Db2 HADR status as green).
 
 
 

(Only applicable for Versions 4.0.x.x and 4.1.0.0) Split cluster scenario: To merge clusters when a standby master server is unreachable

In case of a Split-cluster scenario, you cannot merge the clusters when any of the standby master servers is unreachable or down.

You must resolve the issue with the standby master server that is down, ensure that the server is up and running, and then try to merge the clusters.

Until the issue is unresolved, you can synchronize the data in the two clusters. To do so, complete the following steps:
  1. Complete the steps given in Exporting a device group on the primary master server of cluster 2.
  2. Complete the steps given in Importing a device group on the primary master server of cluster 1.
 

(Only applicable for Versions 4.0.x.x and 4.1.x.x) Split cluster scenario: To determine the time required for an auto takeover

In case of a Split cluster scenario in auto takeover, the time required for the auto takeover operation to complete depends on the values of the following properties and the time required to restart the IBM Security Guardium Key Lifecycle Manager server:

Time required to restart the server (ttr) depends on your system configuration.
 
Important: You experience downtime from the start of the auto takeover operation until it completes.
 

The following table provides the approximate time required for auto takeover operation to complete for the default, minimum, and a sample set of property values. The approximate time indicates the approximate downtime that you experience.

Property valuesApproximate downtime

Default:

  • takeoverRetryTimeInterval=300 (5 minutes)
  • takeoverRetryFrequency=3
(5x3) + ttr
= 15 + ttr minutes

Minimum:

  • takeoverRetryTimeInterval=60 (1 minute)
  • takeoverRetryFrequency=1
(1x1) + ttr
= 1 + ttr minutes

Sample:

  • takeoverRetryTimeInterval=180 (3 minutes)
  • takeoverRetryFrequency=2
(3x2) + ttr
= 6 + ttr minutes
 

(Only applicable for Version 4.1.0.x) Recovering a Multi-Master cluster from a possible split-cluster scenario

Scenario:

When the primary master server of a cluster is out of network for some time, it becomes unreachable, the primary database is unreachable, and the cluster goes in read-only mode. You promote the principal standby master server to primary, and the cluster is in read/write mode. After connectivity with the original primary master server is restored, Db2 disables HADR on the server. As a result, the status of Db2 HADR on the new primary master server displays an error or a warning symbol.

Solution:
Complete the following recovery steps:

  1. On the new primary master server: Take an online backup of the database. For example: DB2 BACKUP DATABASE SKLMDB41 ONLINE TO "C:\Users\sklmdb41\AppData\Local\Temp"
  2. On the original primary master server:
    1. Copy the backup file that is created in the earlier step to any directory. Let's consider the directory to be: C:\

    2. Restore the backup file.
      For example:

      db2 restore database sklmdb41 from C:\

    3. Restore the HADR configuration.

      For example:
      db2 update db cfg for sklmdb41 using HADR_LOCAL_HOST host1 db2 update db cfg for sklmdb41 using HADR_LOCAL_SVC 60028 db2 update db cfg for sklmdb41 using HADR_REMOTE_HOST host2 
      Where,

      host1 is the host name of the original primary master server

      host2 is the host name of the new primary master server

The HADR service on the original primary master server is restored, and it is added to the cluster as a standby master server. The HADR status on the primary master server is refreshed in a few minutes to display the Green status.

 

[{"Type":"MASTER","Line of Business":{"code":"LOB76","label":"Data Platform"},"Business Unit":{"code":"BU048","label":"IBM Software"},"Product":{"code":"SSTJE47","label":"IBM Security Guardium Key Lifecycle Manager"},"ARM Category":[{"code":"a8m0z000000cvdLAAQ","label":"SKLM"}],"ARM Case Number":"","Platform":[{"code":"PF025","label":"Platform Independent"}],"Version":"All Versions"}]

Document Information

Modified date:
23 December 2025

UID

ibm11072470

Page Feedback