Configuring multiple public networks to the cluster

Edit online

When the user wants to place the Ceph Monitor daemons on hosts belonging to multiple network subnets, configuring multiple public networks to the cluster is necessary.

An example of usage is a stretch cluster mode used for Advanced Cluster Management (ACM) in Metro DR for OpenShift Data Foundation.

You can configure multiple public networks to the cluster during bootstrap and once bootstrap is complete.

Prerequisites

Before adding a host be sure that you have a running IBM Storage Ceph cluster.

Procedure

Bootstrap a Ceph cluster configured with multiple public networks.
1. Prepare a ceph.conf file containing a mon public network section.
  
  Important: At least one of the provided public networks must be configured on the current host used for bootstrap.
  
  Syntax
```
 [mon]
 public_network = PUBLIC_NETWORK1, PUBLIC_NETWORK2
```
  Example
```
 [mon]
 public_network = 10.40.0.0/24, 10.41.0.0/24, 10.42.0.0/24
```
  This is an example with three public networks to be provided for bootstrap.
2. Bootstrap the cluster by providing the ceph.conf file as input.
  
  Note: During the bootstrap you can include any other arguments that you want to provide.
  
  Syntax
```
 cephadm --image IMAGE_URL bootstrap --mon-ip MONITOR_IP -c PATH_TO_CEPH_CONF
```
  Note: Alternatively, an IMAGE_ID (such as, 13ea90216d0be03003d12d7869f72ad9de5cec9e54a27fd308e01e467c0d4a0a) can be used instead of IMAGE_URL.
  
  Example
```
 [root@host01 ~]# cephadm –image cp.icr.io/cp/ibm-ceph/ceph-7-rhel9:latest bootstrap –mon-ip 10.40.0.0/24 -c /etc/ceph/ceph.conf   
```
Add new hosts to the subnets.

Note: The host being added must be reachable from the host that the active manager is running on.
1. Install the cluster’s public SSH key in the new host’s root user’s authorized_keys file:
  
  Syntax
```
 # ssh-copy-id -f -i /etc/ceph/ceph.pub root@NEW-HOST
```
  Example
```
 [root@host01 ~]# ssh-copy-id -f -i /etc/ceph/ceph.pub root@host02 
 [root@host01 ~]# ssh-copy-id -f -i /etc/ceph/ceph.pub root@host03
```
2. Add the new node to the Ceph cluster:
  
  Syntax
```
ceph orch host add NEW_HOST IP [LABEL1 ...]
```
  Example
```
 ceph orch host add host02 10.10.0.102 label1 
 ceph orch host add host03 10.10.0.103 label2
```
  Note:
  - It is best to explicitly provide the host IP address. If an IP is not provided, then the host name will be immediately resolved via DNS and that IP will be used.
  - One or more labels can also be included to immediately label the new host. For example, by default the _admin label will make cephadm maintain a copy of the ceph.conf file and a client.admin keyring file in /etc/ceph.
Add the networks configurations for the public network parameters to a running cluster. Be sure that the subnets are separated by commas and that the subnets are listed in subnet/mask format.

Syntax
```
 ceph config set mon public_network "SUBNET_1,SUBNET_2, ..."
```
Example
```
 [root@host01 ~]# ceph config set mon public_network "192.168.0.0/24, 10.42.0.0/24, ..."
```
If necessary, update the mon specifications to place the mon daemons on hosts within the specified subnets.

Reference

For more information about adding hosts, see Adding hosts.
For more information about stretch clusters, see Stretch clusters for Ceph storage.