Cluster bootstrapping and service deployment with Cephadm

The cephadm utility installs and starts a single Ceph Monitor daemon and a Ceph Manager daemon for a new IBM Storage Ceph cluster on the local node where the cephadm bootstrap command is run. Use this information to bootstrap the cluster and deploy all the needed IBM Storage Ceph services in one step using a cluster specification yaml file.

About this task

If you find issues during the deployment, troubleshoot the errors by dividing the deployment into two steps:
  • Bootstrap
  • Service deployment
Note: For more information about the bootstrapping process, see Installing > IBM Storage Ceph installation > Bootstrapping a new storage cluster within IBM Storage Ceph documentation.

Procedure

  1. Create a JSON file to authenticate against the container registry.
    For example:
    cat <<EOF > /root/registry.json
    {
     "url":"registry.redhat.io",
     "username":"User",
     "password":"Pass"
    }
    EOF
  2. Create a cluster-spec.yaml that adds the nodes to the IBM Storage Ceph cluster and also sets specific labels for where the services should run.
    Note: Be sure to follow specifications as detailed in Table 1.
    cat <<EOF > /root/cluster-spec.yaml
    service_type: host
    addr: 10.0.40.78  ## <XXX.XXX.XXX.XXX>
    hostname: ceph1   ##  <ceph-hostname-1>
    location:
      root: default
      datacenter: DC1
    labels:
      - osd
      - mon
      - mgr
    ---
    service_type: host
    addr: 10.0.40.35
    hostname: ceph2
    location:
      datacenter: DC1
    labels:
      - osd
      - mon
    ---
    service_type: host
    addr: 10.0.40.24
    hostname: ceph3
    location:
      datacenter: DC1
    labels:
      - osd
      - mds
      - rgw
    ---
    service_type: host
    addr: 10.0.40.185
    hostname: ceph4
    location:
      root: default
      datacenter: DC2
    labels:
      - osd
      - mon
      - mgr
    ---
    service_type: host
    addr: 10.0.40.88
    hostname: ceph5
    location:
      datacenter: DC2
    labels:
      - osd
      - mon
    ---
    service_type: host
    addr: 10.0.40.66
    hostname: ceph6
    location:
      datacenter: DC2
    labels:
      - osd
      - mds
      - rgw
    ---
    service_type: host
    addr: 10.0.40.221
    hostname: ceph7
    labels:
      - mon
    ---
    service_type: mon
    placement:
      label: "mon"
    ---
    service_type: mds
    service_id: cephfs
    placement:
      label: "mds"
    ---
    service_type: mgr
    service_name: mgr
    placement:
      label: "mgr"
    ---
    service_type: osd
    service_id: all-available-devices
    service_name: osd.all-available-devices
    placement:
      label: "osd"
    spec:
      data_devices:
        all: true
    ---
    service_type: rgw
    service_id: objectgw
    service_name: rgw.objectgw
    placement:
      count: 2
      label: "rgw"
    spec:
      rgw_frontend_port: 8080
    EOF
  3. Retrieve the IP for the NIC with the IBM Storage Ceph public network configured from the bootstrap node.
    After substituting 10.0.40.0 with the subnet that you have defined in your ceph public network, run the following command:
    ip a | grep 10.0.40

    Example output: 10.0.40.78

  4. Run the Cephadm bootstrap command as the root user on the node that will be the initial Monitor node in the cluster.
    The IP_ADDRESS option is the node’s IP address that you are using to run the cephadm bootstrap command.
    Note: If you have configured a different user instead of root for passwordless SSH access, then use the--ssh-user= flag with the cepadm bootstrap command.

    If you are using non default or id_rsa ssh key names, then use --ssh-private-key and --ssh-public-key options with cephadm command.

    cephadm  bootstrap --ssh-user=deployment-user --mon-ip 10.0.40.78 --apply-spec /root/cluster-spec.yaml --registry-json /root/registry.json
    Important: If the local node uses fully-qualified domain names (FQDN), then add the --allow-fqdn-hostname option to cephadm bootstrap on the command line.
    Once the bootstrap finishes, you will see the following output from the previous cephadm bootstrap command:
    You can access the Ceph CLI with:
    
            sudo /usr/sbin/cephadm shell --fsid dd77f050-9afe-11ec-a56c-029f8148ea14 -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring
    
    Please consider enabling telemetry to help improve Ceph:
    
            ceph telemetry on
    
    For more information see:
    
            https://docs.ceph.com/docs/pacific/mgr/telemetry/
  5. Verify the status of IBM Storage Ceph cluster deployment using the Ceph CLI client from ceph1.
    ceph -s
    Example output:
    cluster:
      id:     3a801754-e01f-11ec-b7ab-005056838602
      health: HEALTH_OK
    
    services:
      mon: 5 daemons, quorum ceph1,ceph2,ceph4,ceph5,ceph7 (age 4m)
      mgr: ceph1.khuuot(active, since 5m), standbys: ceph4.zotfsp
      osd: 12 osds: 12 up (since 3m), 12 in (since 4m)
      rgw: 2 daemons active (2 hosts, 1 zones)
    
    data:
      pools:   5 pools, 107 pgs
      objects: 191 objects, 5.3 KiB
      usage:   105 MiB used, 600 GiB / 600 GiB avail
               105 active+clean
    Note: It may take several minutes for all the services to start. It is normal to get a global recovery event while you don’t have any osds configured. You can use ceph orch ps and ceph orch ls commands to further check the status of the services.
  6. Verify if all the nodes are part of the cephadm cluster.
    ceph orch host ls
    Example output:
    HOST   ADDR          LABELS  STATUS
    ceph1  10.0.40.78    _admin osd mon mgr
    ceph2  10.0.40.35    osd mon
    ceph3  10.0.40.24    osd mds rgw
    ceph4  10.0.40.185   osd mon mgr
    ceph5  10.0.40.88    osd mon
    ceph6  10.0.40.66    osd mds rgw
    ceph7  10.0.40.221   mon
    Note: You can run Ceph commands directly from the host because ceph1 was configured in the cephadm-ansible inventory as part of the [admin] group. The Ceph admin keys were copied to the host during the cephadm bootstrap process.
  7. Check the current placement of the Ceph monitor services on the datacenters.
    ceph orch ps | grep mon | awk '{print $1 " " $2}'
    Example output:
    mon.ceph1 ceph1
    mon.ceph2 ceph2
    mon.ceph4 ceph4
    mon.ceph5 ceph5
    mon.ceph7 ceph7
  8. Check the current placement of the Ceph manager services on the datacenters.
    ceph orch ps | grep mgr | awk '{print $1 " " $2}'
    Example output:
    mgr.ceph2.ycgwyz ceph2
    mgr.ceph5.kremtt ceph5
  9. Check the ceph osd crush map layout to ensure that each host has one OSD configured and its status is UP.
    Note: Be sure to check that each node is under the correct datacenter bucket, as detailed in Table 1.
    ceph osd tree
    Example output:
    ID   CLASS  WEIGHT   TYPE NAME           STATUS  REWEIGHT  PRI-AFF
    -1          0.87900  root default
    -16         0.43950      datacenter DC1
    -11         0.14650          host ceph1
      2    ssd  0.14650              osd.2       up   1.00000  1.00000
     -3         0.14650          host ceph2
      3    ssd  0.14650              osd.3       up   1.00000  1.00000
    -13         0.14650          host ceph3
      4    ssd  0.14650              osd.4       up   1.00000  1.00000
    -17         0.43950      datacenter DC2
     -5         0.14650          host ceph4
      0    ssd  0.14650              osd.0       up   1.00000  1.00000
     -9         0.14650          host ceph5
      1    ssd  0.14650              osd.1       up   1.00000  1.00000
     -7         0.14650          host ceph6
      5    ssd  0.14650              osd.5       up   1.00000  1.00000
  10. Create and enable a new RDB block pool.
    ceph osd pool create rbdpool 32 32
    ceph osd pool application enable rbdpool rbd
  11. Verify that the RBD pool has been created.
    ceph osd lspools | grep rbdpool

    Example output: 3 rbdpool

  12. Verify that MDS services are active and has located one service on each datacenter.
    ceph orch ps | grep mds
    Example output:
    mds.cephfs.ceph3.cjpbqo    ceph3               running (17m)   117s ago  17m    16.1M        -  16.2.9
    mds.cephfs.ceph6.lqmgqt    ceph6               running (17m)   117s ago  17m    16.1M        -  16.2.9
  13. Create the CephFS volume.
    ceph fs volume create cephfs
    Note: The ceph fs volume create command also creates the needed data and meta CephFS pools.
  14. Check the Ceph status to verify how the MDS daemons have been deployed.
    Ensure that the state is active where ceph6 is the primary MDS for this filesystem and ceph3 is the secondary MDS.
    ceph fs status
    Example output:
    cephfs - 0 clients
    ======
    RANK  STATE           MDS             ACTIVITY     DNS    INOS   DIRS   CAPS
     0    active  cephfs.ceph6.ggjywj  Reqs:    0 /s    10     13     12      0
           POOL           TYPE     USED  AVAIL
    cephfs.cephfs.meta  metadata  96.0k   284G
    cephfs.cephfs.data    data       0    284G
        STANDBY MDS
    cephfs.ceph3.ogcqkl
  15. Verify that RGW services are active.
    ceph orch ps | grep rgw
    Example output:
    rgw.objectgw.ceph3.kkmxgb  ceph3  *:8080       running (7m)      3m ago   7m    52.7M        -  16.2.9
    rgw.objectgw.ceph6.xmnpah  ceph6  *:8080       running (7m)      3m ago   7m    53.3M        -  16.2.9