Bootstrapping the Ceph cluster with a specification file

Deploy the generalized stretch cluster by setting the CRUSH location to the daemons in the cluster with the spec configuration file.

Before you begin

Before you begin, be sure that you have root-level access to the nodes.

About this task

Set the CRUSH location to the daemons in the cluster with a service configuration file. Use the configuration file to add the hosts to the proper locations during deployment.

For more information about Ceph bootstrapping and different cephadm bootstrap command options, see Bootstrapping a new storage cluster.

Important: Run cephadm bootstrap on the node that you want to be the initial Monitor node in the cluster. The IP_ADDRESS option should be the IP address of the node you are using to run cephadm bootstrap.

Note:

If the storage cluster includes multiple networks and interfaces, be sure to choose a network that is accessible by any node that uses the storage cluster.
To deploy a storage cluster by using IPV6 addresses, use the IPV6 address format for the --mon-ip IP_ADDRESS option. For example: cephadm bootstrap --mon-ip 2620:52:0:880:225:90ff:fefc:2536 --registry-json /etc/mylogin.json.
To route the internal cluster traffic over the public network, omit the --cluster-network SUBNET option.

Within this procedure the network Classless Inter-Domain Routing (CIDR) is referred to as subnet.

Procedure

Create the service configuration YAML file.

The YAML file adds the nodes to the IBM Storage Ceph cluster and also sets specific labels for where the services run.

The following example depends on the specific OSD and Ceph Object Gateway (RGW) configuration that is needed.

---
service_type: host
hostname: HOST01
addr: IP_ADDRESS01
labels: ['alertmanager', 'osd', 'installer', '_admin', 'mon', 'prometheus', 'mgr', 'grafana']
location:
  root: default
  datacenter: DC1
---
service_type: host
hostname: HOST02
addr: IP_ADDRESS02
labels: ['osd', 'mon', 'mgr', 'rgw']
location:
  root: default
  datacenter: DC1
---
service_type: host
hostname: HOST03
addr: IP_ADDRESS03
labels: ['osd', 'mon', 'mds']
location:
  root: default
  datacenter: DC1
---
service_type: host
hostname: HOST04
addr: IP_ADDRESS04
labels: ['osd', '_admin', 'mon', 'mgr']
location:
  root: default
  datacenter: DC2
---
service_type: host
hostname: HOST05
addr: IP_ADDRESS05
labels: ['osd', 'mon', 'mgr', 'rgw']
location:
  root: default
  datacenter: DC2
---
service_type: host
hostname: HOST06
addr: IP_ADDRESS06
labels: ['osd', 'mon', 'mds']
location:
  root: default
  datacenter: DC2
---
service_type: host
hostname: HOST07
addr: IP_ADDRESS07
labels: ['osd', '_admin', 'mon', 'mgr']
location:
  root: default
  datacenter: DC3
---
service_type: host
hostname: HOST08
addr: IP_ADDRESS08
labels: ['osd', 'mon', 'mgr', 'rgw']
location:
  root: default
  datacenter: DC3
---
service_type: host
hostname: HOST09
addr: IP_ADDRESS09
labels: ['osd', 'mon', 'mds']
location:
  root: default
  datacenter: DC3
---
service_type: mon
service_name: mon
placement:
  label: mon
spec:
  crush_locations:
    HOST01:
    - datacenter=DC1
    HOST02:
    - datacenter=DC1
    HOST03:
    - datacenter=DC1
    HOST04:
    - datacenter=DC2
    HOST05:
    - datacenter=DC2
    HOST06:
    - datacenter=DC2
    HOST07:
    - datacenter=DC3
    HOST08:
    - datacenter=DC3
    HOST09:
    - datacenter=DC3

---
service_type: mgr
service_name: mgr
placement:
  label: mgr
------
service_type: osd
service_id: osds
placement:
  label: osd
spec:
  data_devices:
     all: true
---------
service_type: rgw
service_id: rgw.rgw.1
placement:
label: rgw
------------

For more information about changing the custom spec for OSD and Object Gateway, see the following deployment instructions::

Bootstrap the storage cluster with the --apply-spec option.

cephadm bootstrap --apply-spec CONFIGURATION_FILE_NAME --mon-ip MONITOR_IP_ADDRESS --ssh-private-key PRIVATE_KEY --ssh-public-key PUBLIC_KEY --registry-url REGISTRY_URL --registry-username USER_NAME --registry-password PASSWORD

For example,

[root@host01 ~]# cephadm bootstrap --apply-spec initial-config.yaml --mon-ip 10.10.128.68 --ssh-private-key /home/ceph/.ssh/id_rsa --ssh-public-key /home/ceph/.ssh/id_rsa.pub --registry-url registry.redhat.io --registry-username myuser1 --registry-password mypassword1

Important: You can use different command options with the cephadm bootstrap command but always include the --apply-spec option to use the service configuration file and configure the host locations.

Log in to the cephadm shell.
```
[root@host01 ~]# cephadm shell
```
Configure the public network with the subnet.
For more information about configuring multiple public networks to the cluster, see Configuring multiple public networks to the cluster.
```
ceph config set global public_network "SUBNET_1,SUBNET_2, ..."
```
For example,
```
[ceph: root@host01 /]# ceph config set global public_network "10.0.208.0/22,10.0.212.0/22,10.0.64.0/22,10.0.56.0/22"
```

Optional: Configure a cluster network.

For more information about configuring multiple cluster networks to the cluster, see Configuring a private network.

ceph config set global cluster_network "SUBNET_1,SUBNET_2, ..."

For example,

[ceph: root@host01 /]# ceph config set global cluster_network "10.0.208.0/22,10.0.212.0/22,10.0.64.0/22,10.0.56.0/22"

Optional: Verify the network configurations.

[ceph: root@host01 /]# ceph config dump | grep network

Restart the daemons.
Ceph daemons bind dynamically, so you do not have to restart the entire cluster at once if you change the network configuration for a specific daemon.
```
ceph orch restart mon
```
Optional: Restart the cluster on the admin node as a root user, by running the systemctl restart command.
Note: To get the FSID of the cluster, use the ceph fsid command.
```
systemctl restart ceph-FSID_OF_CLUSTER.target
```
For example,
```
[root@host01 ~]# systemctl restart ceph-1ca9f6a8-d036-11ec-8263-fa163ee967ad.target
```

What to do next

Verify the specification file details and that the bootstrap was installed successfully.

Verify that all hosts were placed in the expected data centers, as specified in step 1.

ceph osd tree

Check that there are three data centers under root and that the hosts are placed in each of the expected data centers.

Note: The hosts with OSDs will only be present after bootstrap if OSDs are deployed during bootstrap with the specification file.

For example,

[root@host01 ~]# ceph osd tree
ID   CLASS  WEIGHT   TYPE NAME                                         STATUS  REWEIGHT  PRI-AFF
 -1         0.87836  root default
 -3         0.29279      datacenter DC1
 -2         0.09760          host host01-installer
  0    hdd  0.02440              osd.0                                     up   1.00000  1.00000
 12    hdd  0.02440              osd.12                                    up   1.00000  1.00000
 21    hdd  0.02440              osd.21                                    up   1.00000  1.00000
 29    hdd  0.02440              osd.29                                    up   1.00000  1.00000
 -4         0.09760          host host02
  1    hdd  0.02440              osd.1                                     up   1.00000  1.00000
  9    hdd  0.02440              osd.9                                     up   1.00000  1.00000
 18    hdd  0.02440              osd.18                                    up   1.00000  1.00000
 28    hdd  0.02440              osd.28                                    up   1.00000  1.00000
 -5         0.09760          host host03
  8    hdd  0.02440              osd.8                                     up   1.00000  1.00000
 16    hdd  0.02440              osd.16                                    up   1.00000  1.00000
 24    hdd  0.02440              osd.24                                    up   1.00000  1.00000
 34    hdd  0.02440              osd.34                                    up   1.00000  1.00000
 -7         0.29279      datacenter DC2
 -6         0.09760          host host04
  4    hdd  0.02440              osd.4                                     up   1.00000  1.00000
 13    hdd  0.02440              osd.13                                    up   1.00000  1.00000
 20    hdd  0.02440              osd.20                                    up   1.00000  1.00000
 27    hdd  0.02440              osd.27                                    up   1.00000  1.00000
 -8         0.09760          host host05
  3    hdd  0.02440              osd.3                                     up   1.00000  1.00000
 10    hdd  0.02440              osd.10                                    up   1.00000  1.00000
 19    hdd  0.02440              osd.19                                    up   1.00000  1.00000
 30    hdd  0.02440              osd.30                                    up   1.00000  1.00000
 -9         0.09760          host host06
  7    hdd  0.02440              osd.7                                     up   1.00000  1.00000
 17    hdd  0.02440              osd.17                                    up   1.00000  1.00000
 26    hdd  0.02440              osd.26                                    up   1.00000  1.00000
 35    hdd  0.02440              osd.35                                    up   1.00000  1.00000
-11         0.29279      datacenter DC3
-10         0.09760          host host07
  5    hdd  0.02440              osd.5                                     up   1.00000  1.00000
 14    hdd  0.02440              osd.14                                    up   1.00000  1.00000
 23    hdd  0.02440              osd.23                                    up   1.00000  1.00000
 32    hdd  0.02440              osd.32                                    up   1.00000  1.00000
-12         0.09760          host host08
  2    hdd  0.02440              osd.2                                     up   1.00000  1.00000
 11    hdd  0.02440              osd.11                                    up   1.00000  1.00000
 22    hdd  0.02440              osd.22                                    up   1.00000  1.00000
 31    hdd  0.02440              osd.31                                    up   1.00000  1.00000
-13         0.09760          host host09
  6    hdd  0.02440              osd.6                                     up   1.00000  1.00000
 15    hdd  0.02440              osd.15                                    up   1.00000  1.00000
 25    hdd  0.02440              osd.25                                    up   1.00000  1.00000
 33    hdd  0.02440              osd.33                                    up   1.00000  1.00000

From the cephadm shell, verify that the mon daemons are deployed with CRUSH locations, as specified in step 1.

ceph mon dump

Check that all mon daemons are in the output and that the correct CRUSH locations are added.

[root@host01 ~]# ceph mon dump
epoch 19
fsid b556497a-693a-11ef-b9d1-fa163e841fd7
last_changed 2024-09-03T12:47:08.419495+0000
created 2024-09-02T14:50:51.490781+0000
min_mon_release 19 (squid)
election_strategy: 3
0: [v2:10.0.67.43:3300/0,v1:10.0.67.43:6789/0] mon.host01-installer; crush_location {datacenter=DC1}
1: [v2:10.0.67.20:3300/0,v1:10.0.67.20:6789/0] mon.host02; crush_location {datacenter=DC1}
2: [v2:10.0.64.242:3300/0,v1:10.0.64.242:6789/0] mon.host03; crush_location {datacenter=DC1}
3: [v2:10.0.66.17:3300/0,v1:10.0.66.17:6789/0] mon.host06; crush_location {datacenter=DC2}
4: [v2:10.0.66.228:3300/0,v1:10.0.66.228:6789/0] mon.host09; crush_location {datacenter=DC3}
5: [v2:10.0.65.125:3300/0,v1:10.0.65.125:6789/0] mon.host05; crush_location {datacenter=DC2}
6: [v2:10.0.66.252:3300/0,v1:10.0.66.252:6789/0] mon.host07; crush_location {datacenter=DC3}
7: [v2:10.0.64.145:3300/0,v1:10.0.64.145:6789/0] mon.host08; crush_location {datacenter=DC3}
8: [v2:10.0.64.125:3300/0,v1:10.0.64.125:6789/0] mon.host04; crush_location {datacenter=DC2}
dumped monmap epoch 19

Verify that the service spec and all location attributes are added correctly.

Check the service name for mon daemons on the cluster, by using the ceph orch ls command.

For example,

[root@host01 ~]# ceph orch ls
NAME                       PORTS        RUNNING  REFRESHED  AGE  PLACEMENT
alertmanager               ?:9093,9094      1/1  8m ago     6d   count:1
ceph-exporter                               9/9  8m ago     6d   *
crash                                       9/9  8m ago     6d   *
grafana                    ?:3000           1/1  8m ago     6d   count:1
mds.cephfs                                  3/3  8m ago     6d   label:mds
mgr                                         6/6  8m ago     6d   label:mgr
 mon                                         9/9  8m ago     5d   label:mon
node-exporter              ?:9100           9/9  8m ago     6d   *
osd.all-available-devices                    36  8m ago     6d   label:osd
prometheus                 ?:9095           1/1  8m ago     6d   count:1
rgw.rgw.1                  ?:80             3/3  8m ago     6d   label:rgw

Confirm the mon daemon services, by using the ceph orch ls mon --export command.

For example,

[root@host01 ~]# ceph orch ls mon --export
service_type: mon
service_name: mon
placement:
  label: mon
spec:
  crush_locations:
    host01-installer:
    - datacenter=DC1
    host02:
    - datacenter=DC1
    host03:
    - datacenter=DC1
    host04:
    - datacenter=DC2
    host05:
    - datacenter=DC2
    host06:
    - datacenter=DC2
    host07:
    - datacenter=DC3
    host08:
    - datacenter=DC3
    host09:
    - datacenter=DC3

Verify that the bootstrap was installed successfully, by running the cephadm shell ceph -s command.
For more information, see Verifying the cluster installation.