Ceph network configuration options
These are the common network configuration options for Ceph.
public_networkDescription
The IP address and netmask of the public (front-side) network (for example,192.168.0.0/24). Set in[global]. You can specify comma-delimited subnets.public_addrDescription
The IP address for the public (front-side) network. Set for each daemon.cluster_networkDescription
The IP address and netmask of the cluster network (for example,10.0.0.0/24). Set in[global]. You can specify comma-delimited subnets.cluster_addrDescription
The IP address for the cluster network. Set for each daemon.ms_typeDescription
The messenger type for the network transport layer. IBM supports thesimpleand theasyncmessenger type usingposixsemantics.ms_public_typeDescription
The messenger type for the network transport layer of the public network. It operates identically toms_type, but is applicable only to the public or front-side network. This setting enables Ceph to use a different messenger type for the public or front-side and cluster or back-side networks.ms_cluster_typeDescription
The messenger type for the network transport layer of the cluster network. It operates identically toms_type, but is applicable only to the cluster or back-side network. This setting enables Ceph to use a different messenger type for the public or front-side and cluster or back-side networks.
Host options
You must declare at least one Ceph Monitor in the Ceph configuration file, with a mon addr setting under each declared monitor. Ceph expects a host setting under each declared monitor, metadata server and OSD in the Ceph configuration file.
IMPORTANT: Do not use localhost. Use the short name of the node, not the fully-qualified domain name (FQDN). Do not specify any value for host when using a third party deployment system that retrieves the node name for you.
mon_addrDescription
A list of<hostname>:<port>entries that clients can use to connect to a Ceph monitor. If not set, Ceph searches[mon.*]sections.hostDescription
The host name. Use this setting for specific daemon instances (for example,[osd.0]).
TCP options Ceph disables TCP buffering by default.
ms_tcp_nodelayDescription
Ceph enablesms_tcp_nodelayso that each request is sent immediately (no buffering). Disabling Nagle’s algorithm increases network traffic, which can introduce congestion. If you experience large numbers of small packets, you may try disablingms_tcp_nodelay, but be aware that disabling it will generally increase latency.ms_tcp_rcvbufDescription
The size of the socket buffer on the receiving end of a network connection. Disabled by default.ms_tcp_read_timeoutDescription
If a client or daemon makes a request to another Ceph daemon and does not drop an unused connection, thetcp read timeoutdefines the connection as idle after the specified number of seconds.
Bind options
The bind options configure the default port ranges for the Ceph OSD daemons. The default range is 6800:7100. You can also enable Ceph daemons to bind to IPv6 addresses.
IMPORTANT: Verify that the firewall configuration allows you to use the configured port range.
ms_bind_port_minDescription
The minimum port number to which an OSD daemon will bind.ms_bind_port_maxDescription
The maximum port number to which an OSD daemon will bind.ms_bind_ipv6Description
Enables Ceph daemons to bind to IPv6 addresses.
Asynchronous messenger options
These Ceph messenger options configure the behavior of AsyncMessenger.
ms_async_transport_typeDescription
Transport type used by theAsyncMessenger. IBM supports theposixsetting, but does not support thedpdkorrdmasettings at this time. POSIX uses standard TCP/IP networking and is the default value. Other transport types are experimental and are NOT supported.ms_async_op_threadsDescription
Initial number of worker threads used by eachAsyncMessengerinstance. This configuration setting SHOULD equal the number of replicas or erasure code chunks, but it may be set lower if the CPU core count is low or the number of OSDs on a single server is high.ms_async_max_op_threadsDescription
The maximum number of worker threads used by eachAsyncMessengerinstance. Set to lower values if the OSD host has limited CPU count, and increase if Ceph is underutilizing CPUs are underutilized.ms_async_set_affinityDescription
Set totrueto bindAsyncMessengerworkers to particular CPU cores.ms_async_affinity_coresDescription
Whenms_async_set_affinityistrue, this string specifies howAsyncMessengerworkers are bound to CPU cores. For example,0,2will bind workers \#1 and \#2 to CPU cores \#0 and \#2, respectively.NOTE: When manually setting affinity, make sure to not assign workers to virtual CPUs created as an effect of hyper threading or similar technology, because they are slower than physical CPU cores.
ms_async_send_inlineDescription
Send messages directly from the thread that generated them instead of queuing and sending from theAsyncMessengerthread. This option is known to decrease performance on systems with a lot of CPU cores, so it’s disabled by default.