Manually setting the CRUSH location for daemons

Set the locations manually through ceph osd crush add-bucket and ceph osd crush move commands after the cluster is deployed.

Before you begin

Before you begin, be sure that you have root-level access to the nodes.

Procedure

Add two buckets to which you plan to set the location of your non-tiebreaker monitors to the CRUSH map.

Specify the bucket type as datacenter.

ceph osd crush add-bucket BUCKET_NAME datacenter

For example,

[ceph: root@host01 /]# ceph osd crush add-bucket DC1 datacenter
[ceph: root@host01 /]# ceph osd crush add-bucket DC2 datacenter

. root=default.

ceph osd crush move BUCKET_NAME root=default

For example,

[ceph: root@host01 /]# ceph osd crush move DC1 root=default
[ceph: root@host01 /]# ceph osd crush move DC2 root=default

Move the OSD hosts, according to the required CRUSH placement.

ceph osd crush move HOST datacenter=DATACENTER

For example,

[ceph: root@host01 /]# ceph osd crush move host01 datacenter=DC1

Verify the CRUSH locations for OSD hosts.

ceph osd tree

For example,

[ceph: root@host01 /]# ceph osd tree
ID   CLASS  WEIGHT   TYPE NAME               STATUS REWEIGHT PRI-AFF
-1          0.35010  root default                                
-3          0.17505      datacenter DC1                              
-2          0.05835        host host01                            
5    hdd    0.01459               osd.5      up     1.00000  1.00000
11   hdd    0.01459              osd.11      up     1.00000  1.00000
17   hdd    0.01459              osd.17      up     1.00000  1.00000
23   hdd    0.01459              osd.23      up     1.00000  1.00000
-4          0.05835        host host02
1    hdd    0.01459               osd.1      up     1.00000  1.00000
6    hdd    0.01459               osd.6      up     1.00000  1.00000
12   hdd    0.01459              osd.12      up     1.00000  1.00000
18   hdd    0.01459              osd.18      up     1.00000  1.00000
-5          0.05835        host host03
3    hdd    0.01459               osd.3      up     1.00000  1.00000
10   hdd    0.01459              osd.10      up     1.00000  1.00000
16   hdd    0.01459              osd.16      up     1.00000  1.00000
22   hdd    0.01459              osd.22      up     1.00000  1.00000
-7          0.17505      datacenter DC2
-6          0.05835         host host04
2    hdd    0.01459               osd.2      up     1.00000  1.00000
8    hdd    0.01459               osd.8      up     1.00000  1.00000
14   hdd    0.01459              osd.14      up     1.00000  1.00000
20   hdd    0.01459              osd.20      up     1.00000  1.00000
-8          0.05835         host host05
0    hdd    0.01459               osd.0      up     1.00000  1.00000
7    hdd    0.01459               osd.7      up     1.00000  1.00000
13   hdd    0.01459              osd.13      up     1.00000  1.00000
19   hdd    0.01459              osd.19      up     1.00000  1.00000
-9          0.05835         host host06
4    hdd    0.01459               osd.4      up     1.00000  1.00000
9    hdd    0.01459               osd.9      up     1.00000  1.00000
15   hdd    0.01459              osd.15      up     1.00000  1.00000
21   hdd    0.01459              osd.21      up     1.00000  1.00000

Set the location of each monitor, matching your CRUSH map.

ceph mon set_location HOST datacenter=DATACENTER

For example,

[ceph: root@host01 /]# ceph mon set_location host01 datacenter=DC1
[ceph: root@host01 /]# ceph mon set_location host02 datacenter=DC1
[ceph: root@host01 /]# ceph mon set_location host04 datacenter=DC2
[ceph: root@host01 /]# ceph mon set_location host05 datacenter=DC2

Verify the CRUSH locations for the MON hosts.

Check the mon map to ensure that each MON host has a crush_location specified.

ceph mon dump

The output displays details about the MON map, including the crush_location for each host.

For example,

[ceph: root@host01 /]# ceph mon dump

epoch 5

fsid 4158287e-169e-11f0-b1ad-fa163e98b991

last_changed 2025-04-11T06:32:20.332479+0000

created 2025-04-11T06:29:24.974553+0000

min_mon_release 19 (squid)

election_strategy: 1

0: [v2:10.0.57.33:3300/0,v1:10.0.57.33:6789/0] mon.host07

1: [v2:10.0.58.200:3300/0,v1:10.0.58.200:6789/0] mon.host05; crush_location {datacenter=DC2}

2: [v2:10.0.58.47:3300/0,v1:10.0.58.47:6789/0] mon.host02; crush_location {datacenter=DC1}

3: [v2:10.0.58.104:3300/0,v1:10.0.58.104:6789/0] mon.host04; crush_location {datacenter=DC2}

4: [v2:10.0.58.38:3300/0,v1:10.0.58.38:6789/0] mon.host01; crush_location {datacenter=DC1}