Fail back to the primary storage cluster

When the formerly primary storage cluster recovers, fail back to the primary storage cluster.

Note: If you have scheduled snapshots at the image level, then you need to re-add the schedule as image resync operations changes the RBD Image ID and the previous schedule becomes obsolete.

Prerequisites

Minimum of two running IBM Storage Ceph clusters.
Root-level access to the node.
Pool mirroring or image mirroring configured with one-way mirroring.

Procedure

Check the status of the images from a monitor node in the site-b cluster again. They should show a state of up-stopped and the description should say local image is primary:

Example

[root@rbd-client ~]# rbd mirror image status data/image1
image1:
  global_id:   08027096-d267-47f8-b52e-59de1353a034
  state:       up+stopped
  description: local image is primary
  last_update: 2019-04-22 17:37:48
[root@rbd-client ~]# rbd mirror image status data/image2
image2:
  global_id:   08027096-d267-47f8-b52e-59de1353a034
  state:       up+stopped
  description: local image is primary
  last_update: 2019-04-22 17:38:18

From a Ceph Monitor node on the site-a storage cluster determine if the images are still primary:

Syntax
```
rbd mirror pool info POOL_NAME/IMAGE_NAME
```
Example
```
[root@rbd-client ~]# rbd info data/image1
[root@rbd-client ~]# rbd info data/image2
```
In the output from the commands, look for mirroring primary: true or mirroring primary: false, to determine the state.
Demote any images that are listed as primary by running a command like the following from a Ceph Monitor node in the site-a storage cluster:

Syntax
```
rbd mirror image demote POOL_NAME/IMAGE_NAME
```
Example
```
[root@rbd-client ~]# rbd mirror image demote data/image1
```

Resynchronize the images ONLY if there was a non-orderly shutdown. Run the following commands on a monitor node in the site-a storage cluster to resynchronize the images from site-b to site-a:

Syntax

rbd mirror image resync POOL_NAME/IMAGE_NAME

Example

[root@rbd-client ~]# rbd mirror image resync data/image1
Flagged image for resync from primary
[root@rbd-client ~]# rbd mirror image resync data/image2
Flagged image for resync from primary

After some time, ensure resynchronization of the images is complete by verifying they are in the up+replaying state. Check their state by running the following commands on a monitor node in the site-a storage cluster:

Syntax
```
rbd mirror image status POOL_NAME/IMAGE_NAME
```
Example
```
[root@rbd-client ~]# rbd mirror image status data/image1
[root@rbd-client ~]# rbd mirror image status data/image2
```
Demote the images on the site-b storage cluster by running the following commands on a Ceph Monitor node in the site-b storage cluster:

Syntax
```
rbd mirror image demote POOL_NAME/IMAGE_NAME
```
Example
```
[root@rbd-client ~]# rbd mirror image demote data/image1
[root@rbd-client ~]# rbd mirror image demote data/image2
```
Note: If there are multiple secondary storage clusters, this only needs to be done from the secondary storage cluster where it was promoted.
Promote the formerly primary images located on the site-a storage cluster by running the following commands on a Ceph Monitor node in the site-a storage cluster:

Syntax
```
rbd mirror image promote POOL_NAME/IMAGE_NAME
```
Example
```
[root@rbd-client ~]# rbd mirror image promote data/image1
[root@rbd-client ~]# rbd mirror image promote data/image2
```

Check the status of the images from a Ceph Monitor node in the site-a storage cluster. They should show a status of up+stopped and the description should say local image is primary:

Syntax

rbd mirror image status POOL_NAME/IMAGE_NAME

Example

[root@rbd-client ~]# rbd mirror image status data/image1
image1:
  global_id:   08027096-d267-47f8-b52e-59de1353a034
  state:       up+stopped
  description: local image is primary
  last_update: 2019-04-22 11:14:51
[root@rbd-client ~]# rbd mirror image status data/image2
image2:
  global_id:   596f41bc-874b-4cd4-aefe-4929578cc834
  state:       up+stopped
  description: local image is primary
  last_update: 2019-04-22 11:14:51