Deploying Ceph OSDs using advanced service specifications

The service specification of type OSD is a way to describe a cluster layout using the properties of disks. It gives the user an abstract way to tell Ceph which disks should turn into an OSD with the required configuration without knowing the specifics of device names and paths.

You can deploy the OSD for each device and each host by defining a YAML or JSON file.

Prerequisites

  • A running IBM Storage Ceph cluster.
  • Hosts are added to the cluster.
  • All manager and monitor daemons are deployed.

Procedure

  1. On the monitor node, create the osd_spec.yaml file:

    Example 

    [root@host01 ~]# touch osd_spec.yaml
  2. Edit the osd_spec.yaml file to include the following details: 
    service_type: osd
service_id: SERVICE_ID
placement:
  host_pattern: * # optional
data_devices: # optional
  model: DISK_MODEL_NAME # optional
  paths:
  - /DEVICE_PATH
osds_per_device: NUMBER_OF_DEVICES # optional
db_devices: # optional
  size: # optional
  all: true # optional
  paths:
   - /DEVICE_PATH
encrypted: true
    1. Simple scenarios: In these cases, all the nodes have the same set-up.

      Example 

      service_type: osd
service_id: osd_spec_default
placement:
  host_pattern: '*'
data_devices:
  all: true
  paths:
  - /dev/sdb
encrypted: true

      Example 

      service_type: osd
service_id: osd_spec_default
placement:
  host_pattern: '*'
data_devices:
  size: '80G'
db_devices:
  size: '40G:'
  paths:
   - /dev/sdc
    2. Simple scenario: In this case, all the nodes have the same setup with OSD devices created in raw mode, without an LVM layer.

      Example 

      service_type: osd
service_id: all-available-devices
encrypted: "true"
method: raw
placement:
  host_pattern: "*"
data_devices:
  all: "true"
    3. Advanced scenario: This would create the desired layout by using all HDDs as data_devices with two SSD assigned as dedicated DB or WAL devices. The remaining SSDs are data_devices that have the NVMe vendors assigned as dedicated DB or WAL devices.

      Example 

      service_type: osd
service_id: osd_spec_hdd
placement:
  host_pattern: '*'
data_devices:
  rotational: 0
db_devices:
  model: Model-name
  limit: 2
---
service_type: osd
service_id: osd_spec_ssd
placement:
  host_pattern: '*'
data_devices:
  model: Model-name
db_devices:
  vendor: Vendor-name
    4. Advanced scenario with non-uniform nodes: This applies different OSD specs to different hosts depending on the host_pattern key.

      Example 

      service_type: osd
service_id: osd_spec_node_one_to_five
placement:
  host_pattern: 'node[1-5]'
data_devices:
  rotational: 1
db_devices:
  rotational: 0
---
service_type: osd
service_id: osd_spec_six_to_ten
placement:
  host_pattern: 'node[6-10]'
data_devices:
  model: Model-name
db_devices:
  model: Model-name
    5. Advanced scenario with dedicated WAL and DB devices:

      Example 

      service_type: osd
service_id: osd_using_paths
placement:
  hosts:
    - host01
    - host02
data_devices:
  paths:
    - /dev/sdb
db_devices:
  paths:
    - /dev/sdc
wal_devices:
  paths:
    - /dev/sdd
    6. Advanced scenario with multiple OSDs per device:

      Example 

      service_type: osd
service_id: multiple_osds
placement:
  hosts:
    - host01
    - host02
osds_per_device: 4
data_devices:
  paths:
    - /dev/sdb
    7. For pre-created volumes, edit the osd_spec.yaml file to include the following details: 
       service_type: osd
 service_id: SERVICE_ID
 placement:
   hosts:
     - HOSTNAME
 data_devices: # optional
   model: DISK_MODEL_NAME # optional
   paths:
   - /DEVICE_PATH
 db_devices: # optional
   size: # optional
   all: true # optional
   paths:
   - /DEVICE_PATH

      Example 

       service_type: osd
 service_id: osd_spec
 placement:
   hosts:
     - machine1
 data_devices:
   paths:
     - /dev/vg_hdd/lv_hdd
 db_devices:
   paths:
     - /dev/vg_nvme/lv_nvmes
  3. Mount the YAML file under a directory in the container:

    Example 

    [root@host01 ~]# cephadm shell --mount osd_spec.yaml:/var/lib/ceph/osd/osd_spec.yaml
  4. Navigate to the directory:

    Example 

    [ceph: root@host01 /]# cd /var/lib/ceph/osd/
  5. Before deploying OSDs, do a dry run:
    Note: This step gives a preview of the deployment, without deploying the daemons.

    Example 

    [ceph: root@host01 osd]# ceph orch apply -i osd_spec.yaml --dry-run
  6. Deploy OSDs using service specification: 
    ceph orch apply -i _FILE_NAME_.yml

    Example 

    [ceph: root@host01 osd]# ceph orch apply -i osd_spec.yaml

Verification

  • List the service:

    Example 

    [ceph: root@host01 /]# ceph orch ls osd
  • View the details of the node and devices:

    Example 

    [ceph: root@host01 /]# ceph osd tree