Database resource shows both HADR databases as standby

Running the crm status command on either host lists both databases as standby.

In this scenario, running the db2pd -hadr command on each host should still show the databases in the expected state, with one as the primary and one as the standby. HADR_STATE and HADR_CONNECT_STATUS should be PEER and CONNECTED respectively. However, running the crm status command on either host lists both databases as 'Slaves'.

Important: In Db2® 11.5.8 and later, Mutual Failover high availability is supported when using Pacemaker as the integrated cluster manager. In Db2 11.5.6 and later, the Pacemaker cluster manager for automated fail-over to HADR standby databases is packaged and installed with Db2. In Db2 11.5.5, Pacemaker is included and available for production environments. In Db2 11.5.4, Pacemaker is included as a technology preview only, for development, test, and proof-of-concept environments.

Identifying the problem

[root@svlxtorcpacemaker]# crm status
<...>
Clone Set: db2_rohant_rohant_GTDB-clone [db2_rohant_rohant_GTDB] (promotable)
    Slaves: [svlxtorc, svlxtord]

This is typically caused by an issue at the Pacemaker layer. If the HADR cluster suffers from compounding issues, then the double 'Slaves' state may be encountered.

Here is an example of the problem:

The primary gets demoted to standby due to an issue on the primary host (Reboot, Takeover by force, and others)
Ensuing promotion on corresponding standby host fails due to an issue on the standby (Takeover fails)
After the primary database recovers, running the crm status command still shows both databases in 'Slaves' state

In this scenario the database resource should be recovered by recreating the database resource. This should result in Pacemaker following the normal logic to get the resource back into the desired state.

Resolving the problem

Correct this state with the following steps:

On both hosts run db2support to collect diagnostics for future analysis.
Run the delete database resource command on one of the hosts, this will remove the database resource from the cluster:
```
db2cm -delete -db <database name> -instance <instance name>
```
Confirm that the database has the correct roles of primary and standby on the two hosts and has value PEER and CONNECTED for HADR_STATE and HADR_CONNECT_STATUS respectively in the db2pd -hadr output.
Run the create database resource command on one of the hosts:
```
db2cm -create -db <database name> -instance <instance name>
```

Check the crm status command output to confirm the database resource is in 'Masters' and 'Slaves' state on each host as expected:

[root@svlxtorcpacemaker]# crm status
...
Clone Set: db2_rohant_rohant_GTDB-clone [db2_rohant_rohant_GTDB] (promotable)
    Masters: [ svlxtorc ]
    Slaves: [ svlxtord ]