Mirroring Ceph Block Devices

As a storage administrator, you can add another layer of redundancy to Ceph Block Devices by mirroring data images between IBM Storage Ceph clusters. Understanding and using Ceph Block Device mirroring can provide you protection against data loss, such as a site failure.

The rbd-mirror daemon can run either on a single Ceph storage cluster for one-way mirroring or two Ceph storage clusters for two-way mirroring.

The rbd-mirror daemon is responsible for synchronizing images from one Ceph storage cluster to another Ceph storage cluster by pulling changes from the remote primary image and writing those changes to the local, non-primary image. You can configure Ceph Block Device mirroring either by one-way or two-way mirroring. These mirroring configurations can be done either on pools or on individual images.

For Ceph Block Device mirroring to work, either by using one-way or two-way replication, a couple of assumptions are made:

A pool with the same name exists on both storage clusters.
For journal-based mirroring, a pool contains journal-enabled images that you want to mirror.

Important: In one-way or two-way replication, each instance of rbd-mirror must be able to connect to the other Ceph storage cluster simultaneously. Also, the network must have sufficient bandwidth between the two data center sites to handle mirroring.

Mirroring is configured on a per-pool basis with mirror peering storage clusters. IBM Storage Ceph supports two mirroring modes, depending on the type of images in the pool.

Pool mode: All images in a pool with the journaling feature enabled are mirrored.
Image mode: Only a specific subset of images within a pool are mirrored. You must enable mirroring for each image separately.

Whether or not an image can be modified depends on its state:

You can modify images in the primary state.
You can not modify images in the non-primary state.

Images are automatically promoted to primary when mirroring is enabled on an image. The promotion can happen:

Implicitly by enabling mirroring in pool mode and only for journal-based images.
Explicitly by enabling the mirroring of a specific image.

It is possible to demote primary images and promote non-primary images.

One-way replication

One-way mirroring implies that a primary image or pool of images in one storage cluster gets replicated to a secondary storage cluster. One-way mirroring also supports replicating to multiple secondary storage clusters.

On the secondary storage cluster, the image is the non-primary replicate; that is, Ceph clients cannot write to the image. When data is mirrored from a primary storage cluster to a secondary storage cluster, the rbd-mirror runs only on the secondary storage cluster.

For one-way mirroring to work, a couple of assumptions are made:

You have two IBM Storage Ceph storage clusters and you want to replicate images from a primary storage cluster to a secondary storage cluster.
The secondary storage cluster contains the rbd-mirror daemon that can run on one of the cluster nodes. The rbd-mirror daemon connects to the primary storage cluster to sync images to the secondary storage cluster.

Two-way replication

Two-way replication adds an rbd-mirror daemon on the primary cluster so images can be demoted on it and promoted on the secondary cluster. Changes can then be made to the images on the secondary cluster and they will be replicated in the reverse direction, from secondary to primary. Both clusters must have rbd-mirror running to allow promoting and demoting images on either cluster. Currently, two-way replication is only supported between two sites.

For two-way mirroring to work, a couple of assumptions are made:

You have two storage clusters and you want to be able to replicate images between them in either direction.
Both storage clusters have rbd-mirror daemon running. The images that are primary on cluster1 are synced by the rbd-mirror daemon on the remote cluster and the images on the remote cluster are synced to the primary.