Deploying the NVMe-oF gateway

The Ceph NVMe-oF gateway is the NVMe-oF target node and also a Ceph client node. The Ceph NVMe-oF gateway can be a stand-alone node or be collocated on a Ceph Object Storage Daemon (OSD) node.

Before you begin

Installing the NVMe-oF gateway requires a storage system that is running IBM Storage Ceph 8.0 or later cluster.

Use this information to install the Ceph NVMe-oF gateway with the cephadm command-line interface. To deploy and manage the NVMe-oF gateway with the Ceph Dashboard, see Managing the Ceph NVMe-oF gateway.

For multi-gateway group deployment, repeat any deployment method to configure newer NVMe-oF services.

About this task

Note: When planning to use in-band authentication, install the NVMe-oF service by using the service specification.

For more information, see Deploying by using a service specification file.

If deployment is not done through the service specification, the specification file can be modified at any point to include the encryption key.

For more information, see step 2 of Deploying by using a service specification file. Be sure to use the ceph orch apply command to apply the changes.

Procedure

Complete the following steps to install the Ceph NVMe-oF gateway.
Deploy NVMe-oF daemons.
Important:
  • High Availability is enabled by default. To use High Availability, a minimum of two gateways and listeners must be defined.
  • Ensure an NVMe-oF gateway node is always part of single gateway group.
  • Gateway group names must be unique.
Deploy NVMe-oF daemons by using one of the following methods:

Deploying by specifying the number of daemons and hostnames

Deploy an nvmeof daemon by specifying the number of daemons and hostnames, using the ceph orch apply nvmeof command. 
ceph orch apply nvmeof NVME-OF_GROUP_NAME --placement="NUMBER_OF_NODES NODE1 NODE2,..."
For example, 
[ceph: root@host01 /]# ceph orch apply nvmeof nvmeof_group01 --placement="2 host01 host02"
Similarly, add the more gateway groups by deploying nvmeof services on Ceph nodes that are not part of any other gateway group. The following example is for nvmeof_group2. 
[ceph: root@host01 /]# ceph orch apply nvmeof nvmeof_group2 --placement="2 host04 host05"

What to do next

The ceph orch apply command does not automatically redeploy the service specification changes. Manually redeploy the NVMe-oF service specification changes on the Ceph Orchestrator to complete implementing the changes. 
ceph orch redeploy nvmeof.SERVICE_NAME
For example, 
[ceph: root@host01 /]# ceph orch redeploy nvmeof.nvmeof-pool
Scheduled to redeploy nvmeof.nvmeof-pool.a-rhel9-node2.blhrht on host 'a-rhel9-node2'

Deploying by adding labels to the hosts

Add labels to the hosts and set up the daemons by using the labels.

About this task

For more information about adding and removing host labels, see Managing NVMe-oF gateway host labels.

When planning to deploy multiple gateway groups, maintain identical labels on gateways that will be part of same gateway group and change the label information in these steps, as needed.

Procedure

  1. Add a label to the host. 
    ceph orch host label add NODE LABEL
    For example, 
    [ceph: root@host01 /]# ceph orch host label add host01 nvmeof_label
[ceph: root@host01 /]# ceph orch host label add host02 nvmeof_label
    In this example, the NODE name is host01 and host02 and the LABEL is nvmeof_label.
  2. Display the labels associated with the host. 
    ceph orch host ls
    For example, 
    [ceph: root@host01 /]# ceph orch host ls
HOST    ADDR  LABELS                       STATUS
host01        mon,mgr,_admin,nvmeof_label
host02        mon,mgr,_admin,nvmeof_label
  3. Apply the daemons with the labels added in step 1. 
    ceph orch apply nvmeof NVME-OF_GROUP_NAME --placement="label:LABEL"
    For example, 
    [ceph: root@host01 /]# ceph orch apply nvmeof nvmeof_group01 --placement="label:nvmeof_label"
  4. Redeploy the NVMe-oF service. 
    ceph orch redeploy nvmeof.POOL_NAME.GROUP_NAME
    For example, 
    [ceph: root@host01 /]# ceph orch redeploy nvmeof.nvmeof-pool
Scheduled to redeploy nvmeof.nvmeof-pool.a-rhel9-node2.blhrht on host 'a-rhel9-node2'

Deploying by using a service specification file

Install NVMe daemons by using the service specification file.

About this task

For more information about adding and removing hosts with the service specification file, see Managing gateways with a specification file. To obtain optimal performance when first starting the gateway, consider the Ceph NVMe-oF gateway best practices per NVMe-oF gateway performance best practices.

If modifying cores, during initial deployment start from step 2 of Modifying the cores within NVMe-oF gateway performance best practices.
Note: mTLS can be configured during deployment or at a later stage. To add mTLS during deployment, see Configuring mTLS authentication.

Procedure

  1. Create a specification file. 
    service_type: nvmeof
service_id: .nvmeof.NVME-OF_GROUP_NAME
service_name: SERVICE_TYPE.nvmeof.NVME-OF_GROUP_NAME
placement:
  hosts:
    - HOST_NAME1
    - HOST_NAME2
spec:
  group: NVME-OF_GROUP_NAME
    For example, 
    [root@host01 ~]# cat file.yaml
service_type: nvmeof
service_id: .nvmeof.nvmeof_group01
service_name: nvmeof..nvmeof.nvmeof_group01
placement:
  hosts:
    - host01
    - host02
spec:
  pool: .nvmeof
  group: nvmeof_group01
  2. When using in-band authentication, create an encryption key.
    1. Create the encryption key. 
      openssl req -newkey rsa:512 -noenc -noout -keyout encryption.key -batch 2> /dev/null
    2. Copy the encryption and place it in specification file. 
      service_type: nvmeof
service_id: .nvmeof.NVME-OF_GROUP_NAME
service_name: SERVICE_TYPE..nvmeof.NVME-OF_GROUP_NAME
placement:
  hosts:
    - HOST_NAME1
    - HOST_NAME2
spec:
  encryption_key: |
    -----BEGIN PRIVATE KEY-----
    ENCRYPTION_KEY
    -----END PRIVATE KEY-----
  pool: .nvmeof
  group: nvmeof_group01
  3. Deploy the NVMe daemons on the admin node. 
    ceph orch apply -i NVME_FILE
  4. Redeploy the NVMe-oF service. 
    ceph orch redeploy nvmeof.POOL_NAME.GROUP_NAME
    For example, 
    [ceph: root@host01 /]# ceph orch redeploy nvmeof.nvmeof-pool
Scheduled to redeploy nvmeof.nvmeof-pool.a-rhel9-node2.blhrht on host 'a-rhel9-node2'