Ceph on-wire encryption

You can enable encryption for all Ceph traffic over the network with the introduction of the messenger version 2 protocol. The secure mode setting for messenger v2 encrypts communication between Ceph daemons and Ceph clients, giving you end-to-end encryption.

The second version of Ceph’s on-wire protocol, msgr2, includes several new features:

  • A secure mode encrypting all data moving through the network.

  • Encapsulation improvement of authentication payloads.

  • Improvements to feature advertisement and negotiation.

The Ceph daemons bind to multiple ports allowing both the legacy, v1-compatible, and the new, v2-compatible, Ceph clients to connect to the same storage cluster. Ceph clients or other Ceph daemons connecting to the Ceph Monitor daemon will try to use the v2 protocol first, if possible, but if not, then the legacy v1 protocol will be used. By default, both messenger protocols, v1 and v2, are enabled. The new v2 port is 3300, and the legacy v1 port is 6789, by default.

The messenger v2 protocol has two configuration options that control whether the v1 or the v2 protocol is used:

ms_bind_msgr1

This option controls whether a daemon binds to a port speaking the v1 protocol; it is true by default.

ms_bind_msgr2

This option controls whether a daemon binds to a port speaking the v2 protocol; it is true by default.

Similarly, two options control based on IPv4 and IPv6 addresses used:

ms_bind_ipv4

This option controls whether a daemon binds to an IPv4 address; it is true by default.

ms_bind_ipv6

This option controls whether a daemon binds to an IPv6 address; it is true by default.

The msgr2 protocol supports two connection modes:

  • crc

    • Provides strong initial authentication when a connection is established with cephx.

    • Provides a crc32c integrity check to protect against bit flips.

    • Does not provide protection against a malicious man-in-the-middle attack.

    • Does not prevent an eavesdropper from seeing all post-authentication traffic.

  • secure

    • Provides strong initial authentication when a connection is established with cephx.

    • Provides full encryption of all post-authentication traffic.

    • Provides a cryptographic integrity check.

The default mode is crc.

Ensure that you consider cluster CPU requirements when you plan the IBM Storage Ceph deployment to include encryption overhead.

Important: Using secure mode is supported by Ceph clients using librbd, such as OpenStack Nova, Glance, and Cinder.

Address Changes

For both versions of the messenger protocol to co-exist in the same storage cluster, the address formatting has changed:

Syntax for old address format 

IP_ADDR:PORT/CLIENT_ID

For example, 1.2.3.4:5678/91011

Syntax for new address format 

PROTOCOL_VERSION:IP_ADDR:PORT/CLIENT_ID

For example, v2:1.2.3.4:5678/91011, where PROTOCOL_VERSION can be either v1 or v2.

Because the Ceph daemons now bind to multiple ports, the daemons display multiple addresses instead of a single address. Here is an example from a dump of the monitor map:

epoch 1
fsid 50fcf227-be32-4bcb-8b41-34ca8370bd17
last_changed 2021-12-12 11:10:46.700821
created 2021-12-12 11:10:46.700821
min_mon_release 14 (nautilus)
0: [v2:10.0.0.10:3300/0,v1:10.0.0.10:6789/0] mon.a
1: [v2:10.0.0.11:3300/0,v1:10.0.0.11:6789/0] mon.b
2: [v2:10.0.0.12:3300/0,v1:10.0.0.12:6789/0] mon.c

Also, the mon_host configuration option and specifying addresses on the command line using -m supports the new address format.

Connection Phases

There are four phases for making an encrypted connection:

Banner

On connection, both the client and the server send a banner. Currently, the Ceph banner is ceph 0 0n.

Authentication Exchange

All data, sent or received, is contained in a frame for the duration of the connection. The server decides if authentication has completed, and what the connection mode will be. The frame format is fixed, and can be in three different forms depending on the authentication flags being used.

Message Flow Handshake Exchange

The peers identify each other and establish a session. The client sends the first message, and the server will reply with the same message. The server can close connections if the client talks to the wrong daemon. For new sessions, the client and server proceed to exchanging messages. Client cookies are used to identify a session, and can reconnect to an existing session.

Message Exchange

The client and server start exchanging messages, until the connection is closed.

Reference

See the Data security and hardening for details on enabling the msgr2 protocol.