HADR takeover operations in a Db2 pureScale environment

When an HADR standby database takes over as the primary database in a Db2 pureScale environment, there are a number of important differences from HADR in other environments.

With HADR, there are two types of takeover: role switch and failover. Role switch, sometimes called graceful takeover or non-forced takeover, can be performed only when the primary is available and it switches the role of primary and standby. Failover, or forced takeover, can be performed when the primary is not available. It is commonly used in primary failure cases to make the standby the new primary. The old primary remains in the primary role in a forced takeover, but the standby sends it a message to disable it. Both types of takeover are supported in a Db2 pureScale environment, and both can be issued from any of the standby database members and not just the current replay member. However, after the standby completes the transition to the primary role, the database is only started on the member that served as the replay member before the takeover. The database can be started on the other members by issuing an ACTIVATE DATABASE command or implicitly through a client connection.

In Db2 12.1.2 and earlier, the member topology must be the same in the two clusters. In Db2 12.1.3 and later, each of the clusters can have different member topologies, with the primary cluster having the same, more, or fewer members than the standby cluster. When the number of members in the database topology, the instance topology, or both differ between the standby cluster and the primary cluster, this is referred to as a mixed topology environment. In such environments, special care needs to be made before and after takeover to ensure the right member topology exists in both primary and standby clusters.

Role switch

After a role switch, which is initiated by issuing the TAKEOVER HADR command from any standby member, the standby cluster becomes the primary cluster and vice versa. Role switch helps ensure that no data is lost between the old primary and new primary. You can initiate a role switch in the following circumstances only:
  • Crash recovery is not happening on the primary cluster, including member crash recovery that is pending or in progress.
  • All the log streams are in peer or assisted remote catchup state.
  • All the log streams are in remote catchup state or in assisted remote catchup state, and the synchronization mode is SUPERASYNC.
Before you initiate a role switch in remote catchup or assisted remote catchup state, check the log gap between the primary and standby log streams. A large gap can result in a long takeover time because all of the logs in that gap must be shipped and replayed first.
During a role switch, the following steps happen on the primary:
  1. New connections are rejected on all members, any open transactions are rolled back, and all remaining logs are shipped to the standby.
  2. The database role of the primary cluster changes to standby.
  3. A member that has a direct connection to the standby is chosen as the replay member, with preference given to the preferred replay member. The preferred replay member is the member from which HADR was started.
  4. Log receiving and replay starts on the replay member.
  5. The database is shut down on the other non-replay members of the cluster.
The following steps happen on the standby:
  1. Log receiving is stopped on the replay member after the end of logs is reached on each log stream, helping ensure no data loss.
  2. The replay member finishes replaying all received logs.
  3. After it is confirmed that the primary cluster is now in the standby role, the replay member changes the role of the standby cluster to primary.
  4. The database is opened for client connections, but it is only activated on the member that was previously the standby replay member.

Failover

After a failover, which is initiated by issuing the TAKEOVER HADR command with the BY FORCE option from any standby member, the standby cluster becomes the primary cluster. The old primary cluster is sent a disabling message, but its role is not changed. Any member on the primary that receives this message disables the whole primary cluster. By initiating a failover, you are accepting the trade-off between potential data loss and having a working database. You cannot initiate a failover if the databases are in local catchup state.
Note: Unlike in previous releases, you can now initiate a failover even if log archive retrieval is in progress.
During a failover, the following steps happen on the primary cluster, assuming that the cluster is online and connected to the standby cluster:
  1. After the database receives the disabling message, the database is shut down and log writing is stopped.
Also, the following steps happen on the standby, all of which are carried out from the replay member:
  1. A disabling message is sent to the primary, if it is connected.
  2. Log shipping and log retrieval is stopped, which can result in data loss.
  3. The replay member finishes replaying all the logs that are stored in the log path.
  4. Any open transactions are rolled back.
  5. The replay member changes the role of the standby cluster to primary.
  6. The database is opened for client connections, but it is only activated on the member that was previously the standby replay member.

You can reintegrate the old primary as a new standby only if the log streams of the old primary did not diverge from the log streams of the new primary. Before you can start HADR, the database must be offline for all of the members in the old primary cluster; the cluster caching facilities, however, can stay online. If any members are online, shut them down instead of issuing the DEACTIVATE DATABASE command on them.

Pending topology changes

If a takeover is required, there are two options available to the user to deal with pending topology changes on the standby:
  • Update the instance topology to match the database topology, as from the primary, before the TAKEOVER command is issued.
  • Update the instance topology to match the database topology, as from the primary, after the TAKEOVER command is issued and then reconcile those changes by activating the database on all the members in the instance.

In general, topology changes need to be seamless to the user and based on the maintenance windows available. At takeover time, if the standby is in an add member pending state, any new member directories are created and the database is made available. The add pending member is added to the database topology only if, at a later time, a connection is made to that member. If the standby is in a drop member pending state after the takeover operation, the database is left deactivated and it returns a SQL1387W reason code 2, since drop member is an offline operation. A START HADR AS PRIMARY [BY FORCE] statement can be issued to activate the database and start offline member reconciliation, which removes the member from the database topology.

On the primary, after the takeover operation, the primary becomes the new standby. Because the new standby is in a recovery state, any pending topology changes are deferred and do not impact the new standby. A Replay operation starts up and the database topology is put in a rebuild state as it waits and replays log records from the new primary.

It is expected that on both the primary and the standby database, all members have their HADR_LOCAL_HOST and HADR_LOCAL_SVC database configuration values set correctly. It is also recommended that the hadr_target_list is updated to represent the correct member topology. Having any of these values incorrectly configured can cause takeover and ensuing operations to fail.

Takeover with pending replay of Add or Drop Member operations

There can be timing situations where an add or drop member log record has been written on the primary and shipped to the standby, but the standby has not yet replayed the log record. This is known as a pending replay. In such cases, due to timing of the replay and when the user issues the takeover operation, specific handling is enforced.

Graceful, or non-forced, takeover

A graceful takeover operation fails if any add or drop member log record is pending to be replayed.

Forced Takeover

If the replay of the add or drop member log record has begun, any forced takeover operation fails. If the forced takeover operation is issued and then the replay of an add or drop member log record is encountered, the following actions take place:
  1. The replay service raises an error.
  2. The replay operation stops at the point before the add or drop member log record entry.
  3. An undo operation starts, with data loss happening from the point that the replay operation stopped.