Applying rolling updates to a Pacemaker-managed Db2 high availability disaster recovery (HADR) environment

When you use the integrated High Availability (HA) feature with Pacemaker to automate HADR, extra steps are required to update the operating system or the Db2 database system software, upgrade the hardware, or change the database configuration parameters.

Before you begin

Note: The following update procedures are for a Pacemaker automated HADR environment. If you want to perform rolling updates on an HADR environment that is not automated, see Applying rolling updates to a Db2® high availability disaster recovery (HADR) environment.

Before running the rolling update procedure, ensure that the following prerequisites are met:

You have configured HADR for your Pacemaker-managed Linux cluster, either on two hosts or on a two-sites multiple standby cluster with same-site failover automation, on four hosts in two sites.
The HADR pair are in PEER state.

Restrictions

Use this procedure to perform a rolling update on your Db2 database system and update the Db2 database product software to a new fix pack level in a Pacemaker automated HADR environment. For example, applying a fix pack to a Db2 database product software.

A rolling update cannot be used to upgrade a Db2 database system from an earlier version to a later version. For example, you cannot use this procedure to upgrade from Db2 11.5 to Db2 12.1. To upgrade a Pacemaker automated HADR environment from a previous Db2 version, see Upgrading Db2 servers in a Pacemaker automated HADR environment.

You cannot use this procedure to update the Db2 HADR configuration parameters. Updates to the HADR configuration parameters must be made separately. Because HADR requires the parameters on the primary and standby to be the same, both the primary and standby databases might need to be deactivated and updated at the same time.

The following procedure cannot be used to convert an existing Db2 HADR system using Tivoli® SA MP (TSA) as a cluster manager to a newer Db2 level using Pacemaker as a cluster manager in a single step. Instead, the existing system should be first updated to a new Db2 level while maintaining TSA as the integrated manager. Once the update is complete, follow the steps outlined in Replacing an existing Tivoli SA MP-managed Db2 instance with a Pacemaker-managed HADR Db2 instance to use Pacemaker as the cluster manager.

The following procedure is only applicable when the existing Db2 HADR cluster is deployed using the Db2-provided Pacemaker cluster software stack. If the cluster to be updated uses Pacemaker provided by other vendors, all cluster resources should be removed following the procedures outlined from the Pacemaker supplier and recreated using the Db2 provided Pacemaker cluster software stack using the db2cm utility. See Configuring high availability with the Db2 cluster manager utility (db2cm).

Procedure

Log in as the root user, export the DB2INSTANCE environment variable of the instance name, and disable automation on the cluster:
```
<install_dir>/bin/db2cm -disable -all
```
On each standby host, ensure all databases have their HADR_ROLE set as STANDBY:
```
db2pd -hadr -db <database-name>
```
- If the databases do not have the right role, run the following command on the primary host, for all databases that do not have the correct role:
```
db2 takeover hadr on db <database-name>
```
On each standby host, deactivate all databases to stop HADR while retaining the role:
```
db2 deactivate db <database-name>
```
On each standby host, stop all Db2 processes:
```
db2stop force
```
Db2 12.1.4 or earlier versions, log in as the root user on each standby host and stop all Pacemaker and Coroysnc processes:
```
systemctl stop pacemaker
systemctl stop corosync
systemctl stop corosync-qdevice
```
Note: Only run the systemctl stop corosync-qdevice command if the Qdevice is configured.
Apply the update on each standby host.
- If you are not updating to a new Db2 fix pack and not upgrading to a new major version of the operating system (for example, from Red Hat Enterprise Linux version 8 to version 9), then proceed to step 6 after applying the change.
- If you are updating to a new major version of the operating system (for example, from Red Hat Enterprise Linux version 8 to version 9), run the following command, and then proceed to step 6.
```
db2installPCMK -i
```
- If you are updating to a new Db2 fix pack, follow the procedure Installing offline fix pack updates to existing Db2 database products (Linux® and UNIX).
Db2 12.1.4 or earlier versions, log in as the root user on each standby host and stop all Pacemaker and Coroysnc processes:
```
systemctl start pacemaker
systemctl start corosync
systemctl start corosync-qdevice
```
Note: Only run the systemctl start corosync-qdevice command if the Qdevice is configured.
As the root user on each standby host, check the configuration, either manually or by running the crm_verify tool, if available:
```
crm_verify -L -V
```
Note: This prints any error in the configuration. If there is nothing wrong, nothing is printed.
On each standby host, start all Db2 processes:
```
db2start
```
On each standby host, activate all databases:
```
db2 activate db <database-name>
```
On the principal standby host, run a role switch for all databases:
```
db2 takeover hadr on db <database-name>
```
- If applying a new Db2 fix pack, after the role switch, the old primary database disconnects because the new primary is running on a higher fix pack level.
On the old primary host, repeat step 2 to step 9 to apply the update on this host.
On the original primary host for each database, run a failback operation to set the HADR roles back to their original state:
```
db2 takeover hadr on db <database-name>
```
Verify that all the databases are in the PEER state:
```
db2pd -hadr -db <database-name>
```
Important: Step 14 through step 16 are only necessary if updating to a new Db2 fix pack and the Qdevice is configured.
If updating to a new Db2 fix pack, as the root user on the Qdevice host with the Qdevice configured, stop the corosync-qnetd process:
```
systemctl stop corosync-qnetd
```
If the Qdevice is configured, update the corosync-qnetd package provided by IBM® on the Qdevice host as the root user using the db2installPCMK utility:
```
<Db2_image>/db2/<platform>/pcmk/db2installPCMK -q
```
If updating to new Db2 fix pack, as the root user on the Qdevice host with the Qdevice configured, update the qnetd process:
```
systemctl start corosync-qnetd
```
As the root user, export the DB2INSTANCE environment variable of the instance name, and enable automation on the cluster:
```
<install_dir>/bin/db2cm -enable -all
```

Confirm that the cluster is in a healthy state:

<db2inst home>/sqllib/bin/db2cm -status

Note: This might take Pacemaker around a minute to complete.

The following example shows an output from running the db2cm -status command:

 db2_db2tea1_eth1 (ocf::heartbeat:db2ethmon):  Started
 db2_kedge1_eth1  (ocf::heartbeat:db2ethmon):  Started
 db2_kedge1_db2inst1_0  (ocf::heartbeat:db2inst):  Started
 db2_db2tea1_db2inst2_0 (ocf::heartbeat:db2inst):  Started
 db2_kedge1_db2inst2_0  (ocf::heartbeat:db2inst):  Started
 Clone Set: db2_db2inst2_db2inst2_CORAL-clone [db2_db2inst2_db2inst2_CORAL] (promotable)
     Promoted: [ db2tea1 ]
     Unpromoted: [ kedge1 ]
 Clone Set: db2_db2inst2_db2inst2_CORAL2-clone [db2_db2inst2_db2inst2_CORAL2] (promotable)
     Promoted: [ db2tea1 ]
     Unpromoted: [ kedge1 ]
 db2_db2tea1_db2inst1_0 (ocf::heartbeat:db2inst):  Started
 Clone Set: db2_db2inst1_db2inst1_CORAL-clone [db2_db2inst1_db2inst1_CORAL] (promotable)
     Promoted: [ db2tea1 ]
     Unpromoted: [ kedge1 ]
 Clone Set: db2_db2inst1_db2inst1_CORAL2-clone [db2_db2inst1_db2inst1_CORAL2] (promotable)
     Promoted: [ db2tea1 ]
     Unpromoted: [ kedge1 ]

No resources should be in the unmanaged state and all resources should be started on the expected role.