Cluster bootstrapping and service deployment with Cephadm

The cephadm utility installs and starts a single Ceph Monitor daemon and a Ceph Manager daemon for a new IBM Storage Ceph cluster on the local node where the cephadm bootstrap command is run. Use this information to bootstrap the cluster and deploy all the needed IBM Storage Ceph services in one step using a cluster specification yaml file.

About this task

If you find issues during the deployment, troubleshoot the errors by dividing the deployment into two steps:

Bootstrap
Service deployment

Note: For more information about the bootstrapping process, see Installing > IBM Storage Ceph installation > Bootstrapping a new storage cluster within IBM Storage Ceph documentation.

Procedure

Create a JSON file to authenticate against the container registry.

For example:

cat <<EOF > /root/registry.json
{
 "url":"registry.redhat.io",
 "username":"User",
 "password":"Pass"
}
EOF

Create a cluster-spec.yaml that adds the nodes to the IBM Storage Ceph cluster and also sets specific labels for where the services should run.

Note: Be sure to follow specifications as detailed in Table 1.

cat <<EOF > /root/cluster-spec.yaml
service_type: host
addr: 10.0.40.78  ## <XXX.XXX.XXX.XXX>
hostname: ceph1   ##  <ceph-hostname-1>
location:
  root: default
  datacenter: DC1
labels:
  - osd
  - mon
  - mgr
---
service_type: host
addr: 10.0.40.35
hostname: ceph2
location:
  datacenter: DC1
labels:
  - osd
  - mon
---
service_type: host
addr: 10.0.40.24
hostname: ceph3
location:
  datacenter: DC1
labels:
  - osd
  - mds
  - rgw
---
service_type: host
addr: 10.0.40.185
hostname: ceph4
location:
  root: default
  datacenter: DC2
labels:
  - osd
  - mon
  - mgr
---
service_type: host
addr: 10.0.40.88
hostname: ceph5
location:
  datacenter: DC2
labels:
  - osd
  - mon
---
service_type: host
addr: 10.0.40.66
hostname: ceph6
location:
  datacenter: DC2
labels:
  - osd
  - mds
  - rgw
---
service_type: host
addr: 10.0.40.221
hostname: ceph7
labels:
  - mon
---
service_type: mon
placement:
  label: "mon"
---
service_type: mds
service_id: cephfs
placement:
  label: "mds"
---
service_type: mgr
service_name: mgr
placement:
  label: "mgr"
---
service_type: osd
service_id: all-available-devices
service_name: osd.all-available-devices
placement:
  label: "osd"
spec:
  data_devices:
    all: true
---
service_type: rgw
service_id: objectgw
service_name: rgw.objectgw
placement:
  count: 2
  label: "rgw"
spec:
  rgw_frontend_port: 8080
EOF

Retrieve the IP for the NIC with the IBM Storage Ceph public network configured from the bootstrap node.
After substituting 10.0.40.0 with the subnet that you have defined in your ceph public network, run the following command:
```
ip a | grep 10.0.40
```
Example output: 10.0.40.78
Run the Cephadm bootstrap command as the root user on the node that will be the initial Monitor node in the cluster.
The IP_ADDRESS option is the node’s IP address that you are using to run the cephadm bootstrap command.
Note: If you have configured a different user instead of root for passwordless SSH access, then use the--ssh-user= flag with the cepadm bootstrap command.
If you are using non default or id_rsa ssh key names, then use --ssh-private-key and --ssh-public-key options with cephadm command.
```
cephadm  bootstrap --ssh-user=deployment-user --mon-ip 10.0.40.78 --apply-spec /root/cluster-spec.yaml --registry-json /root/registry.json
```
Important: If the local node uses fully-qualified domain names (FQDN), then add the --allow-fqdn-hostname option to cephadm bootstrap on the command line.
Once the bootstrap finishes, you will see the following output from the previous cephadm bootstrap command:
```
You can access the Ceph CLI with:

        sudo /usr/sbin/cephadm shell --fsid dd77f050-9afe-11ec-a56c-029f8148ea14 -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring

Please consider enabling telemetry to help improve Ceph:

        ceph telemetry on

For more information see:

        https://docs.ceph.com/docs/pacific/mgr/telemetry/
```

Verify the status of IBM Storage Ceph cluster deployment using the Ceph CLI client from ceph1.

ceph -s

Example output:

cluster:
  id:     3a801754-e01f-11ec-b7ab-005056838602
  health: HEALTH_OK

services:
  mon: 5 daemons, quorum ceph1,ceph2,ceph4,ceph5,ceph7 (age 4m)
  mgr: ceph1.khuuot(active, since 5m), standbys: ceph4.zotfsp
  osd: 12 osds: 12 up (since 3m), 12 in (since 4m)
  rgw: 2 daemons active (2 hosts, 1 zones)

data:
  pools:   5 pools, 107 pgs
  objects: 191 objects, 5.3 KiB
  usage:   105 MiB used, 600 GiB / 600 GiB avail
           105 active+clean

Note: It may take several minutes for all the services to start. It is normal to get a global recovery event while you don’t have any OSDs configured. You can use ceph orch ps and ceph orch ls commands to further check the status of the services.

Verify if all the nodes are part of the cephadm cluster.
```
ceph orch host ls
```
Example output:
```
HOST   ADDR          LABELS  STATUS
ceph1  10.0.40.78    _admin osd mon mgr
ceph2  10.0.40.35    osd mon
ceph3  10.0.40.24    osd mds rgw
ceph4  10.0.40.185   osd mon mgr
ceph5  10.0.40.88    osd mon
ceph6  10.0.40.66    osd mds rgw
ceph7  10.0.40.221   mon
```
Note: You can run Ceph commands directly from the host because ceph1 was configured in the cephadm-ansible inventory as part of the [admin] group. The Ceph admin keys were copied to the host during the cephadm bootstrap process.

Check the current placement of the Ceph monitor services on the datacenters.

ceph orch ps | grep mon | awk '{print $1 " " $2}'

Example output:

mon.ceph1 ceph1
mon.ceph2 ceph2
mon.ceph4 ceph4
mon.ceph5 ceph5
mon.ceph7 ceph7

Check the current placement of the Ceph manager services on the datacenters.

ceph orch ps | grep mgr | awk '{print $1 " " $2}'

Example output:

mgr.ceph2.ycgwyz ceph2
mgr.ceph5.kremtt ceph5

Check the ceph osd crush map layout to ensure that each host has one OSD configured and its status is UP.

Note: Be sure to check that each node is under the correct datacenter bucket, as detailed in Table 1.

ceph osd tree

Example output:

ID   CLASS  WEIGHT   TYPE NAME           STATUS  REWEIGHT  PRI-AFF
-1          0.87900  root default
-16         0.43950      datacenter DC1
-11         0.14650          host ceph1
  2    ssd  0.14650              osd.2       up   1.00000  1.00000
 -3         0.14650          host ceph2
  3    ssd  0.14650              osd.3       up   1.00000  1.00000
-13         0.14650          host ceph3
  4    ssd  0.14650              osd.4       up   1.00000  1.00000
-17         0.43950      datacenter DC2
 -5         0.14650          host ceph4
  0    ssd  0.14650              osd.0       up   1.00000  1.00000
 -9         0.14650          host ceph5
  1    ssd  0.14650              osd.1       up   1.00000  1.00000
 -7         0.14650          host ceph6
  5    ssd  0.14650              osd.5       up   1.00000  1.00000

Create and enable a new RDB block pool.
```
ceph osd pool create rbdpool 32 32
```
```
ceph osd pool application enable rbdpool rbd
```
Note: The number 32 at the end of the command is the number of PGs assigned to this pool. The number of PGs can vary depending on several factors like the number of OSDs in the cluster, expected % used of the pool, etc. You can use the following calculator to determine the number of PGs needed: Ceph Placement Groups (PGs) per Pool Calculator.
Verify that the RBD pool has been created.
```
ceph osd lspools | grep rbdpool
```
Example output: 3 rbdpool

Verify that MDS services are active and has located one service on each datacenter.

ceph orch ps | grep mds

Example output:

mds.cephfs.ceph3.cjpbqo    ceph3               running (17m)   117s ago  17m    16.1M        -  16.2.9
mds.cephfs.ceph6.lqmgqt    ceph6               running (17m)   117s ago  17m    16.1M        -  16.2.9

Create the CephFS volume.
```
ceph fs volume create cephfs
```
Note: The ceph fs volume create command also creates the needed data and meta CephFS pools.

Check the Ceph status to verify how the MDS daemons have been deployed.

Ensure that the state is active where ceph6 is the primary MDS for this filesystem and ceph3 is the secondary MDS.

ceph fs status

Example output:

cephfs - 0 clients
======
RANK  STATE           MDS             ACTIVITY     DNS    INOS   DIRS   CAPS
 0    active  cephfs.ceph6.ggjywj  Reqs:    0 /s    10     13     12      0
       POOL           TYPE     USED  AVAIL
cephfs.cephfs.meta  metadata  96.0k   284G
cephfs.cephfs.data    data       0    284G
    STANDBY MDS
cephfs.ceph3.ogcqkl

Verify that RGW services are active.

ceph orch ps | grep rgw

Example output:

rgw.objectgw.ceph3.kkmxgb  ceph3  *:8080       running (7m)      3m ago   7m    52.7M        -  16.2.9
rgw.objectgw.ceph6.xmnpah  ceph6  *:8080       running (7m)      3m ago   7m    53.3M        -  16.2.9