Configuring mTLS authentication

Configure mutual TLS (mTLS) to ensure secure connections between the command-line interface gRPC client and the Ceph NVMe-oF gateway gRPC server.

Before you begin

Before configuring mTLS, ensure that the following are on each Ceph node that the NVMe-oF gateway will run on.

Server certificate and key files.
Important: When creating the server certificates, use the subjectAltName parameter to specify the IP addresses of all gateways in the group. For example,
```
openssl req -x509 -newkey rsa:4096 -nodes -keyout server.key -out server.crt -days 3650 -subj /CN=my.server -addext "subjectAltName=IP:10.243.64.15,IP:10.243.64.17,IP:10.243.64.13,IP:10.243.64.4"
```
For more information about creating server certificates, see Creating and managing TLS keys and certificates within Securing networks in the Red Hat Enterprise Linux documentation.
Client certificate and key files.
A running IBM Storage Ceph 7.1z1 or later cluster.
Ceph NVMe-oF gateway installed. For more information, see Deploying the NVMe-oF gateway.
Important: Due to known issue BZ 2282560, when using mTLS together with the IBM Storage Ceph Plugin for vSphere a workaround is required. For more information see Known issues in the Release notes of IBM Storage Ceph Plugin for vSphere documentation.

About this task

Configuring mTLS authentication is optional but recommended for secure connectivity.

Use the following procedures to configure mTLS authentication, by using the command-line interface:

Configuring mTLS on the NVMe-oF server
Configuring mTLS on the NVMe-oF client

The mTLS configuration can be disabled at any time. To disable, see Disabling the mTLS secure connection.

Note: To configure mTLS by using the Ceph Dashboard, see Managing the NVMe-oF service.

Configuring mTLS on the NVMe-oF server

About this task

Start the mTLS configuration setup from the NVMe-oF server side.

Procedure

Export the NVMe-oF gateway specification configuration to a YAML file.

ceph orch ls nvmeof --export > gw-conf.yaml

For example,

[ceph: root@host01 /]# ceph orch ls nvmeof --export > gw-conf.yaml

Example for an NVMe-oF gateway specification configuration file.

service_type: nvmeof
service_id: nvmeof-pool
service_name: nvmeof.nvmeof-pool.group1
placement:
  hosts:
  - a-rhel9-node2
spec:
  allowed_consecutive_spdk_ping_failures: 1
  bdevs_per_cluster: 32
  conn_retries: 10
  discovery_port: 8009
  enable_monitor_client: true
  enable_prometheus_exporter: true
  group: group1
  log_directory: /var/log/ceph/
  log_files_enabled: true
  log_files_rotation_enabled: true
  log_level: DEBUG
  max_log_directory_backups: 10
  max_log_file_size_in_mb: 10
  max_log_files_count: 20
  monitor_timeout: 1.0
  omap_file_lock_duration: 20
  omap_file_lock_retries: 30
  omap_file_lock_retry_sleep_interval: 1.0
  omap_file_update_reloads: 10
  pool: nvmeof-pool
  port: 5500
  rpc_socket_dir: /var/tmp/
  rpc_socket_name: spdk.sock
  spdk_log_level: WARNING
  spdk_path: /usr/local/bin/nvmf_tgt
  spdk_ping_interval_in_seconds: 2.0
  spdk_timeout: 60.0
  state_update_interval_sec: 5
  state_update_notify: true
  tgt_cmd_extra_args: --cpumask=0xF
  tgt_path: /usr/local/bin/nvmf_tgt
  transport_tcp_options:
    in_capsule_data_size: 8192
    max_io_qpairs_per_ctrlr: 7
  transports: tcp
  verbose_log_messages: true
  verify_nqns: true

Copy the NVMe-oF exported specification file to a new mTLS file.

cp gw-conf.yaml mTLS_SPECIFICATION_FILE.yaml

For example,

[ceph: root@host01 /]# cp gw-conf.yaml gw-conf-with-mtls.yaml

Edit the NVMe-oF mTLS specification file.
Add the following values into the spec section of the specification file.
```
enable_mtls: true
```
For example,
```
vi gw-conf-with-mtls.yaml

service_type: nvmeof
service_id: nvmeof-pool
service_name: nvmeof.nvmeof-pool.group1
placement:
  hosts:
  - a-rhel9-node2
spec:
  pool: nvmeof-pool
  enable_mtls: true
```
Certificates are automatically created and managed by cephadm. Manually adding certificate content to the specification file is not required. The certificates generated by cephadm are located under /var/lib/ceph/FSID/nvmeof.CERTIFICATE

Apply the specification file changes.

ceph orch apply -i mTLS_SPECIFICATION_FILE.yaml

For example,

[ceph: root@host01 /]# ceph orch apply -i gw-conf-with-mtls.yaml
Scheduled nvmeof.nvmeof-pool update...

Redeploy the NVMe-oF service.

ceph orch redeploy nvmeof.POOL_NAME.GROUP_NAME

For example,

[ceph: root@host01 /]# ceph orch redeploy nvmeof.nvmeof-pool
Scheduled to redeploy nvmeof.nvmeof-pool.a-rhel9-node2.blhrht on host 'a-rhel9-node2'

Configuring mTLS on the NVMe-oF client

About this task

After the NVMe-oF server is configured, configure mTLS on the NVMe-oF client side.

The ceph nvmeof commands automatically handle mTLS without specifying certificates.

podman run -it --rm NVME-OF_CLI_CONTAINER_IMAGE:TAG --server-address NVME-OF_SERVER_IP gw info

Note: When using nvmeof-cli commands, certificates are also automatically generated by cephadm, but must be provided.

Locate certificates under: /var/lib/ceph/FSID/nvmeof.CERTIFICATE/

Copy them locally and run:

docker run -v ./server_cert:/root/server_cert:z \
-v ./client_key:/root/client_key:z \
-v ./client_cert:/root/client_cert:z --rm \

NVME-OF_CLI_CONTAINER_IMAGE \
--server-address NVME-OF_SERVER_IP \
--server-port 5500 \
--client-cert /root/client_cert \
--server-cert /root/server_cert \
--client-key /root/client_key \

CLI_COMMAND

Disabling the mTLS secure connection

About this task

To disable the mTLS secure connection to the server, the NVMe-oF gateway specification file needs to be reconfigured.

Procedure

Edit the NVMe-oF mTLS specification file.
Remove or set enable_mtls to false.

Apply the specification file changes.

ceph orch apply -i mTLS_SPECIFICATION_FILE.yaml

For example,

[ceph: root@host01 /]# ceph orch apply -i gw-conf-with-mtls.yaml
Scheduled nvmeof.nvmeof-pool update...

Redeploy the NVMe-oF service.

ceph orch redeploy nvmeof.POOL_NAME.GROUP_NAME

For example,

[ceph: root@host01 /]# ceph orch redeploy nvmeof.nvmeof-pool.group1
Scheduled to redeploy nvmeof.nvmeof-pool.a-rhel9-node2.blhrht on host 'a-rhel9-node2'

The mTLS configuration is removed and the connection is insecure. The NVMe-oF command-line interface can now be used without the certificate and keys.