DR notification examples

This topic illustrates common sequences of notifications that might be issued by a Disaster Recovery (DR) configuration.

Note: When a high availability (HA) queue manager is configured with DR, the current HA secondary appliance will notify only of changes to the DR role (other DR messages do not appear on the secondary appliance).

Configuring primary DR queue manager

When an appliance queue manager is enabled for disaster recovery, the following sequence of notifications are produced to indicate progress.

The following notifications are seen on the appliance where the crtdrprimary command is issued:

Replication notifications

AMQ3580E: DR status for queue manager QM_DR is 'Inconsistent'

As the data replication resources are created they are marked as Inconsistent to indicate the data is yet to be replicated to the remote appliance.

AMQ3581W: DR status for queue manager QM_DR is 'Remote appliance not configured'

When the DR primary instance is first created the remote appliance is reported as not configured, this will remain so until the generated crtdrsecondary command is issued on the remote appliance.

Disaster recovery (DR) notifications

AMQ3900I: DR role for queue manager QM_DR is primary

The appliance on which the crtdrprimary command is issued is promoted to the DR primary role.

Configuring secondary DR queue manager

When the crtdrsecondary command is issued on the remote appliance the following sequence of notifications is seen:

Disaster recovery (DR) notifications

AMQ3901I: DR role for queue manager QM_DR is secondary

The remote appliance takes on the DR secondary role.

Replication notifications

AMQ3580E: DR status for queue manager QM_DR is 'Inconsistent'

The data replication resource is marked as inconsistent to indicate that the DR secondary appliance needs to synchronize the queue manager data from the DR primary appliance.

AMQ3585W: DR status for queue manager QM_DR is 'Synchronization in progress'

After the data replication resources on both appliances connect, they begin synchronizing the data from the primary appliance to the secondary appliance. If replication is interrupted, for example, by interruption of the network, the state returns to Inconsistent.

AMQ3586I: DR status for queue manager QM_DR is 'Normal'

After the secondary appliance has synchronized, all disk writes from the primary appliance are asynchronously replicated to the secondary appliance. This is the Normal state.

Losing communication with remote appliance

When communication is lost between DR appliances the following notifications are seen:

AMQ3574E: DR replication to remote appliance 'Appliance2' for queue manager 'QM_DR' using interface 'eth20' is unavailable
AMQ3582E: DR status for queue manager QM_DR is 'Remote appliance unavailable'

Losing communication for replication is reported for each affected queue manager and results in a state change to Remote appliance unavailable. From this point the data on the secondary appliance becomes increasingly out of date.

Regaining communication with remote appliance

As with losing data replication communication with the remote appliance, the following notification is seen when communication with the remote appliance becomes available:

AMQ3575I: DR replication to remote appliance 'Appliance2' for queue manager 'QM_DR' using interface 'eth20' is available

A notification is produced for each queue manager when the data replication connection is re-established.

Typically, this is shortly followed by the secondary appliance reporting that the queue manager data as Inconsistent then synchronizing from the primary.

Note:

When a high availability (HA) queue manager is configured with DR, if the DR recovery appliance issues notification AMQ3574E or AMQ3575I it will always give the name of the HA appliance on which the crtdrprimary command was issued as the remote appliance.

Switching over to a recovery appliance

Following the loss of the primary queue manager at the main site, you might need to start the secondary queue manager at the recovery site.

AMQ3900I: DR role for queue manager QM_DR is primary

The queue manager is promoted to the Primary role.

If the recovery site is inconsistent when the makedrprimary command is issued (that is, synchronization failed from the main site), the queue manager reverts to the previous saved snapshot of its data before it can be promoted to primary.

AMQ3579W: DR status for queue manager QM_DR is 'Reverting to snapshot'
AMQ3582E: DR status for queue manager QM_DR is 'Remote appliance unavailable'

After reversion to the previous snapshot, the status is updated to Remote appliance unavailable.

The queue manager data has now deviated from that on the main site appliance and is reported as Partitioned when the replication connection becomes available.

Partitioned behavior

If the replication link is lost but the remote appliance is still operating, the same sequence of states occurs on both appliances. It is then possible to issue the makedrprimary command on the secondary appliance, resulting in the queue manager running simultaneously on both. This situation results in the queue manager deviating in a manner that cannot be reconciled.

AMQ3584S: DR status for queue manager QM_DR is 'Partitioned'

When the replication link recovers, the data replication resources detect the deviation in queue manager data and report that the queue manager has become Partitioned. The partitioned state might be detected on only one appliance, or both.

On detecting the partitioned state, the replication connection becomes unavailable.

Resolving a partitioned state

After following the steps in Resolving a partitioned problem in a disaster recovery configuration to identify which appliance holds the 'winning' data for partition resolution, use the makedrsecondary command on the 'losing' appliance.

AMQ3901I: DR role for queue manager QM_DR is secondary
AMQ3582E: DR status for queue manager QM_DR is 'Remote appliance unavailable'

The losing appliance is demoted to the DR secondary role and the status updated to indicate Remote appliance unavailable.

To complete the partition resolution, issue the makedrprimary command on the 'winning' appliance.

AMQ3575I: DR replication to remote appliance 'Appliance2' for queue manager 'QM_DR' using interface 'eth20' is available

Data replication is then re-established between the appliances, with the data to be replaced marked as Inconsistent, allowing synchronization from the chosen primary node.

AMQ3580E: DR status for queue manager QM_DR is 'Inconsistent'
AMQ3585W: DR status for queue manager QM_DR is 'Synchronization in progress'
AMQ3586I: DR status for queue manager QM_DR is 'Normal'

After synchronization has completed, if the makedrprimary command was not issued on the preferred primary node, the queue manager can be moved back using the makedrsecondary and makedrprimary commands.