Upgrading Db2 servers in HADR multiple standby environments without standby reinitialization

Upgrading Db2 servers in HADR multiple standby environments without standby reinitialization maintains the database role and relies on the normal log shipping and log replaying characteristics that are common to the HADR functionality. This method does not stop HADR for upgrades and avoids the need to reinitialize HADR after the upgrades. The benefit of this method is that it reduces the window where no standby database exists and eliminates the cost of sending a backup image to the standby site for reinitialization.

The minimum requirement for using this procedure is Db2 version 10.5 Fix Pack 7 or later.

This procedure ensures all log data generated by the primary database prior to the upgrade are shipped and replayed by all standbys. This is enforced by the use of db2ckupgrade that is run during db2iupgrade, or any db2ckupgrade invocation that establishes an exclusive connection to the database to perform proper database consistency checking.

There are two methods to perform upgrades in multiple standby environment, and both are described in procedure below.

Method 1: Upgrades all the standbys and the primary to the new version.
Method 2: Keeping one or both auxiliary standbys in the current version in case there is a need to fallback if encountering complications after upgrading primary and principal standby to the new version.

Before you begin

Ensure that you are familiar with the following tasks:

Upgrading Db2 servers in HADR single standby environments without standby reinitialization.
In case of failures during the HADR upgrade procedure, ensure that you are familiar with Dealing with failures while upgrading Db2 servers in HADR environments without standby reinitialization.

About this task

This upgrade task describes the steps to upgrade databases in HADR multiple standby environment from Db2 version 10.5 Fix Pack 7 or later. The primary and all standby databases should be using the same fix pack level of Db2.

Procedure

Method 1: Upgrading all standbys and the primary to the latest Db2 version.
1. In the current Db2 version, using High availability disaster recovery (HADR) monitoring ensure that the primary database's log shipping functionality and all the standby database's log replay functionality are healthy.
2. Ensure that replay delay is turned off on all the standby databases by setting the database configuration parameter hadr_replay_delay to 0 to ensure the standby log replay position can catch up to the primary in a reasonable amount of time.
3. On the primary instance, deactivate all databases and stop the instance. Ensure that the standby database is active.
4. Upgrade the primary instance using db2iupgrade. db2iupgrade calls db2ckupgrade and for HADR databases this validates the log positions between the primary database and all standby databases. db2ckupgrade also validates that instance-level settings do not contain any discontinued values. If the log positions do not match for some database, then db2iupgrade/db2ckupgrade fails. If this happens, ensure the log shipping/replay functionality is still healthy for that database. If healthy, increase the hadr_timeout value for that database to give the log validation check more time for the log positions to match. If some standby is unable to catch up to the primary within enough time, then remove that standby from the hadr_target_list. The removed standby database needs to reinitialized post upgrade using a backup image. For a given database, if there are issues getting the log positions to match for principal and auxiliary standbys then you must proceed by stopping HADR on that database. That database needs to be upgraded then HADR reinitialized.
5. On the standby instances, deactivate all databases and stop the instance. This can be done across all instances in parallel.
6. Upgrade the standby instances using db2iupgrade. This can be done across all instances in parallel.
7. For each database in the standby instances, starting with the principal standby, issue the UPGRADE DATABASE command. This upgrades the database metadata objects and if log validation succeeded in the earlier steps, the principal standby starts up and waits for a connection from the primary. The replay functionality begins in the background and waits for upgrade log data to be sent by the primary. At this point, the principal standby is considered upgrade in progress and the UPGRADE DATABASE command will return with:
```
SQL1103W  The UPGRADE DATABASE command was completed successfully.
```
  No user connections are allowed while in this upgrade in progress state. The progress on the standby database can be monitored using db2pd -hadr with the Db2 diagnostics log. Connect attempts can also be issued and receives SQL1776N, reason code 9 as long as the standby is in upgrade in progress state.
  You can repeat the same step for each auxiliary standby whenever convenient for you. The key is that the principal standby is brought up first and sits waiting for the primary before any auxiliary can issue the UPGRADE DATABASE command.
8. For each database in the primary instance, issue the UPGRADE DATABASE command. This upgrades the database metadata objects and if log validation succeeded in the earlier steps, attempts to form a connection with the principal standby database. Upgrade will not be able to proceed unless a connection to a valid principal standby database is available. Normal upgrade processing takes place and all log data are shipped to the principal standby database for replay. At any point once the primary is able to form a connection with an auxiliary database, log data is shipped to the auxiliary database for replay.
9. When upgrade on the primary database is complete the database can be activated and normal operations can resume.
10. When a standby database has replayed all the upgrade log data, the standby is no longer considered in upgrade in progress state and stays activated and can resume normal operations. For standby databases enabled for reads on standby that means read only connections can take place again.
Method 2: Maintains one or both auxiliary standbys in the current version in case a fallback is needed if complications arise after upgrading the primary and principal standby to the new version.
1. Follow steps (a) through (e) under Method 1.
  Note: It is very important to deactivate the auxiliary standby databases that you want to perform the upgrade procedure later, and keep them deactivated while performing the upgrade of the principal standby and the primary database. Failing to do so will result in not being able to complete the upgrade procedure later for the auxiliary database and require them to be reinitialized.
2. Upgrade the principal standby instance using db2iupgrade.
3. For each database in the principal standby instance, issue the UPGRADE DATABASE command. This upgrades the database metadata objects and if log validation succeeded in the earlier steps, the principal standby starts up and waits for a connection from the primary. The replay functionality begins in the background and waits for upgrade log data to be sent by the primary. At this point, the principal standby is considered upgrade in progress and the UPGRADE DATABASE command returns with:
```
SQL1103W  The UPGRADE DATABASE command was completed successfully.
```
  No user connections are allowed while in this upgrade in progress state. The progress on the standby database can be monitored using db2pd -hadr in conjunction with the Db2 diagnostics log. Connect attempts can also be issued and receive SQL1776N, reason code 9 as long as the standby is in upgrade in progress state.
4. For each database in the primary instance, issue the UPGRADE DATABASE command. This upgrades the database metadata objects and if log validation succeeded in the earlier steps, attempts to form a connection with the principal standby database. Upgrade will not be able to proceed unless a connection to a valid principal standby database is available. Normal upgrade processing takes place and all log data are shipped to the principal standby database for replay.
5. When upgrade on the primary database is complete the database can be activated and normal operations can resume.
6. When a standby database has replayed all the upgrade log data, the standby is no longer considered in upgrade in progress state and stays activated and can resume normal operations. For standby databases enabled for reads on standby that means read only connections can take place again.
7. After confirming that everything is functioning as expected on both the primary and principal standby databases, you can repeat steps (b), (c) and (f) for each auxiliary database to upgrade them to the new Db2 version.

What to do next

Post upgrade, follow the same recommendations as outlined in Upgrading Db2 servers in HADR single standby environments without standby reinitialization.