Verifying stretch mode

Use this information to verify that stretch mode was created correctly with the implemented CRUSH rules.

Procedure

Verify that all pools are using the CRUSH rule that was created in the Ceph cluster.

In these examples, the CRUSH rule is set as stretch_rule, per the settings that were created in Configuring a CRUSH map for stretch mode.

for pool in $(rados lspools);do echo -n "Pool: ${pool}; ";ceph osd pool get ${pool} crush_rule;done

For example,

[ceph: root@host01 /]# for pool in $(rados lspools);do echo -n "Pool: ${pool}; ";ceph osd pool get ${pool} crush_rule;done 
Pool: device_health_metrics; crush_rule: stretch_rule
Pool: cephfs.cephfs.meta; crush_rule: stretch_rule
Pool: cephfs.cephfs.data; crush_rule: stretch_rule
Pool: .rgw.root; crush_rule: stretch_rule
Pool: default.rgw.log; crush_rule: stretch_rule
Pool: default.rgw.control; crush_rule: stretch_rule
Pool: default.rgw.meta; crush_rule: stretch_rule
Pool: rbdpool; crush_rule: stretch_rule

Verify that stretch mode is enabled.
Ensure that stretch_mode_enabled is set to true.
```
ceph osd dump
```
The output includes the following information:

stretch_mode_enabled

Set to true if stretch mode is enabled.

stretch_bucket_count

The number of data centers with OSDs.

degraded_stretch_mode

Output of 0 if not degraded. If the stretch mode is degraded, this outputs the number of up sites.

recovering_stretch_mode

Output of 0 if not recovering. If the stretch mode is recovering, the output is 1.

stretch_mode_bucket

A unique value set for each CRUSH bucket type. This value is usually set to 8, for data center.
For example,
```
stretch_mode_enabled true

stretch_bucket_count 2

degraded_stretch_mode 0

recovering_stretch_mode 0

stretch_mode_bucket 8
```

Verify that stretch mode is using the mon map, by using the ceph mon dump command.

Ensure the following:

stretch_mode_enabled is set to 1
The correct mon host is set as tiebreaker_mon
The correct mon host is set as disallowed_leaders

ceph mon dump

For example,

[ceph: root@host01 /]# ceph mon dump
epoch 16
fsid ff19789c-f5c7-11ef-8e1c-fa163e4e1f7e
last_changed 2025-02-28T12:12:51.089706+0000
created 2025-02-28T11:34:59.325503+0000
min_mon_release 19 (squid)
election_strategy: 3
stretch_mode_enabled 1
tiebreaker_mon host07
disallowed_leaders host07
0: [v2:10.0.56.37:3300/0,v1:10.0.56.37:6789/0] mon.host01; crush_location {datacenter=DC1}
1: [v2:10.0.59.188:3300/0,v1:10.0.59.188:6789/0] mon.host05; crush_location {datacenter=DC2}
2: [v2:10.0.59.35:3300/0,v1:10.0.59.35:6789/0] mon.host02; crush_location {datacenter=DC1}
3: [v2:10.0.56.189:3300/0,v1:10.0.56.189:6789/0] mon.host07; crush_location {datacenter=DC3}
4: [v2:10.0.56.13:3300/0,v1:10.0.56.13:6789/0] mon.host04; crush_location {datacenter=DC2}
dumped monmap epoch 16

What to do next

If needed, install and configure an NVMe over Fabrics (NVMe-oF) gateway. For more information, see Ceph NVMe-oF gateway.
Create an NFS cluster, customize it, and export Ceph File System namespace over the NFS protocol. For more information, see NFS cluster and export management.
Deploy, configure, and administer a Ceph Object Gateway. For more information, see Ceph Object Gateway.
Manage, create, configure, and use Ceph Block Devices. For more information, see Using Ceph Block Devices.
Create, mount, and work the Ceph File System (CephFS). For more information, see Ceph File Systems.