Configuring high availability for the Ceph Object Gateway

To configure high availability (HA) for the Ceph Object Gateway you write a YAML configuration file, and the Ceph orchestrator does the installation, configuration, and management of the ingress service. The ingress service uses the haproxy and keepalived daemons to provide high availability for the Ceph Object Gateway.

Before you begin

Before you begin, make sure that you have the following prerequisites in place:

A minimum of two hosts running Red Hat Enterprise Linux 9 or higher, for installing the ingress service on.
A healthy running IBM Storage Ceph cluster.
A minimum of two Ceph Object Gateway daemons running on different hosts.
Root-level access to the host running the ingress service.
If using a firewall, then open port 80 for HTTP and port 443 for HTTPS traffic.

About this task

You can deploy an ingress service with Ceph Object Gateway as the backend, where the use_tcp_mode_over_rgw option is set to true in the spec section of the ingress specification.

For more information, see High availability service, and Performing a standard RHEL 9 installation within the product documentation for your supported Red Hat Enterprise Linux version on Red Hat Documentation.

Note: In cephadm deployments, HTTPS is configured by using certificate management. The ingress service handles SSL termination rather than configuring SSL directly on the Ceph Object Gateway frontend.

Procedure

Create a new ingress.yaml file.

touch ingress.yaml

For example,

[root@host01 ~] touch ingress.yaml

Open the ingress.yaml file for editing.

Add in the following options, and add values applicable to the environment.

service_type: ingress
service_id: SERVICE_ID    # adjust to match your existing RGW service
placement:
 hosts:
   - HOST1
   - HOST2
   - HOST3
spec:
 backend_service: SERVICE_ID              # adjust to match your existing RGW service
 virtual_ip: IP_ADDRESS/CIDR              # ex: 192.168.20.1/24
 frontend_port: INTEGER                   # ex: 8080
 monitor_port: INTEGER                    # ex: 1967, used by haproxy for load balancer status
 virtual_interface_networks: [ ... ]      # optional: list of CIDR networks
 use_keepalived_multicast: TRUE/FALSE     # optional: Default is False.
 vrrp_interface_network: IP_ADDRESS/CIDR  # optional: ex: 192.168.20.0/24
 health_check_interval: TIME              # optional: Default is 2s.
 ssl: true
 certificate_source: inline               # optional: Default is cephadm-signed
 ssl_cert: |                              # optional: SSL certificate and key
   -----BEGIN CERTIFICATE-----
   ...
   -----END CERTIFICATE-----
 ssl_key: |
   -----BEGIN PRIVATE KEY-----
   ...
   -----END PRIVATE KEY-----
 enable_stats: true
 monitor_ssl: BOOLEAN
 monitor_cert_source: inline               # optional: default is reuse_service_cert
 monitor_ssl_cert: |                       # optional: SSL certificate and key
   -----BEGIN CERTIFICATE-----
   ...
   -----END CERTIFICATE-----
 monitor_ssl_key: |
   -----BEGIN PRIVATE KEY-----
   ...
   -----END PRIVATE KEY-----
 monitor_networks: [..]
 monitor_ip_addrs:
   host: IP_ADDRESSSS/CIDR

Table 1 provides the ingress service configuration parameters and descriptions.

Table 1. Ingress service configuration
Parameter	Description
service_type	Mandatory and set to `ingress`.
service_id	Name of the service, for example, `rgw.foo`.
placement hosts	Hosts for HA daemons (haproxy and keepalived). Need not match RGW nodes.
virtual_ip	Virtual IP in CIDR format for ingress service.
virtual_ips_list	List of virtual IPs in CIDR format. Each IP is primary on one ingress node.
virtual_interface_networks	Networks to identify ethernet interface for virtual IP.
frontend_port	Port used to access the ingress service.
ssl_cert	SSL certificate in .pem format. Required when `certificate_source` is `inline` or `reference`.
ssl	Enable SSL for ingress service.
certificate_source	Source of certificate: `inline`, `reference`, or `cephadm-signed`. Default is `cephadm-signed`.
ssl_key	SSL key in .pem format if certificate_source is not `cephadm-signed`.
use_keepalived_multicast	Default is `False`. Uses unicast IPs for keepalived config.
health_check_interval	Interval between haproxy health checks. Default is 2 seconds.
enable_stats	Enable haproxy stats. Default is `False`.
monitor_ssl	Enable SSL for monitoring. Requires service SSL enabled.
monitor_cert_source	Source of monitor certificate: `reuse_service_cert`, `inline`, `reference`, or `cephadm-signed`. Default is `reuse_service_cert`.
monitor_ssl_cert	Monitor SSL certificate in .pem format if monitor_cert_source is not`cephadm-signed`.
monitor_ssl_key	Monitor SSL key in .pem format if monitor_cert_source is not`cephadm-signed`.
monitor_ip_addrs	If assigned to host, it will be used. Otherwise, monitor_networks will be checked.
monitor_networks	If specified, an IP matching one of the networks will be used. Otherwise, default host IP is used.

Following is an example of providing an SSL cert:

service_type: ingress
service_id: rgw.foo
placement:
  hosts:
    - host01.example.com
    - host02.example.com
    - host03.example.com
spec:
  backend_service: rgw.foo
  virtual_ip: 192.168.1.2/24
  frontend_port: 8080
  monitor_port: 1967
  virtual_interface_networks:
    - 10.10.0.0/16
  ssl_cert: |
    -----BEGIN CERTIFICATE-----
    MIIEpAIBAAKCAQEA+Cf4l9OagD6x67HhdCy4Asqw89Zz9ZuGbH50/7ltIMQpJJU0
    gu9ObNtIoC0zabJ7n1jujueYgIpOqGnhRSvsGJiEkgN81NLQ9rqAVaGpadjrNLcM
    bpgqJCZj0vzzmtFBCtenpb5l/EccMFcAydGtGeLP33SaWiZ4Rne56GBInk6SATI/
    JSKweGD1y5GiAWipBR4C74HiAW9q6hCOuSdp/2WQxWT3T1j2sjlqxkHdtInUtwOm
    j5Ism276IndeQ9hR3reFR8PJnKIPx73oTBQ7p9CMR1J4ucq9Ny0J12wQYT00fmJp
    -----END CERTIFICATE-----
    -----BEGIN PRIVATE KEY-----
    MIIEBTCCAu2gAwIBAgIUGfYFsj8HyA9Zv2l600hxzT8+gG4wDQYJKoZIhvcNAQEL
    BQAwgYkxCzAJBgNVBAYTAklOMQwwCgYDVQQIDANLQVIxDDAKBgNVBAcMA0JMUjEM
    MAoGA1UECgwDUkhUMQswCQYDVQQLDAJCVTEkMCIGA1UEAwwbY2VwaC1zc2wtcmhj
    czUtOGRjeHY2LW5vZGU1MR0wGwYJKoZIhvcNAQkBFg5hYmNAcmVkaGF0LmNvbTAe
    -----END PRIVATE KEY-----

Following is an example of an ingress file without providing an SSL cert:

service_type: ingress

service_id: rgw.ssl    # adjust to match your existing RGW service

placement:
hosts:
- hostname1
- hostname2
spec:
backend_service: rgw.rgw.ssl.ceph13   # adjust to match your existing RGW service
virtual_ip: IP_ADDRESS/CIDR           # ex: 192.168.20.1/24
frontend_port: INTEGER                # ex: 443
monitor_port: INTEGER                 # ex 1969
use_tcp_mode_over_rgw: True

Launch the cephadm shell.

cephadm shell --mount ingress.yaml:FILEPATH/ingress.yaml

For example,

[root@host01 ~]# cephadm shell --mount ingress.yaml:/var/lib/ceph/radosgw/ingress.yaml

Configure the latest haproxy and keepalived images.

ceph config set mgr mgr/cephadm/container_image_haproxy HAPROXY_IMAGE_ID
ceph config set mgr mgr/cephadm/container_image_keepalived KEEPALIVED_IMAGE_ID

For example,

[ceph: root@host01 /]# ceph config set mgr mgr/cephadm/container_image_haproxy cp.icr.io/cp/ibm-ceph/haproxy-rhel10:latest
[ceph: root@host01 /]# ceph config set mgr mgr/cephadm/container_image_keepalived cp.icr.io/cp/ibm-ceph/keepalived-rhel10:latest

Install and configure the new ingress service using the Ceph orchestrator.

ceph orch apply -i FILEPATH/ingress.yaml

For example,

[ceph: root@host01 /]# ceph orch apply -i /var/lib/ceph/radosgw/ingress.yaml

After the Ceph orchestrator completes, verify the HA configuration.

On the host running the ingress service, check that the virtual IP address displays.
```
ip addr show
```
For example,
```
[root@host01 ~]# ip addr show
```

Try reaching the Ceph Object Gateway from a Ceph client.

wget HOST_NAME

For example,

[root@client ~]# wget host01.example.com

The HA configuration for the Ceph Object Gateway is working properly if this returns an index.html file, similar to the following example:

<?xml version="1.0" encoding="UTF-8"?>
        <ListAllMyBucketsResult xmlns="http://s3.amazonaws.com/doc/2006-03-01/">
                <Owner>
                        <ID>anonymous</ID>
                        <DisplayName></DisplayName>
                </Owner>
                <Buckets>
                </Buckets>
        </ListAllMyBucketsResult>