Configuring with IBM Storage Ceph

Configure Fusion Data Foundation in external mode with an external IBM Storage Ceph cluster.

1. Log in to the Red Hat® OpenShift® Container Platform web console as an administrator, and navigate to Storage > External systems.
2. Click Connect external systems.
3. Select Red Hat/IBM Ceph Cluster and click Next.
4. In the Connect Red Hat Ceph storage page, do as follows:
   1. Click on the Download script link to download the Python script for extracting Ceph cluster details.
   2. To extract the IBM Storage Ceph cluster details, contact the IBM Storage Ceph administrator to run the downloaded Python script on an IBM Storage Ceph node by using the admin key.
      1. Run the following command on the IBM Storage Ceph node to view the list of available arguments:

         python3 ceph-external-cluster-details-exporter.py --help

         Important: Use python instead of python3 if the IBM Storage Ceph cluster is deployed on a Red Hat Enterprise Linux 7.x (RHEL 7.x) cluster.

         You can also run the script from inside a MON container (for containerized deployments) or from a MON node (for RPM deployments).

         Note: Use the yum install cephadm command, followed by the cephadm command to deploy your IBM Storage Ceph cluster using containers. You must pull the IBM Storage Ceph cluster container images by using the cephadm command, rather than installing Ceph packages on nodes by using the yum command. For more information, see IBM Storage Ceph documentation.
      2. Retrieve the external cluster details from the IBM Storage Ceph cluster by using one of the following options:

         Configuration file flag

         Use the config-file flag that stores the parameters used during deployment.

         In new deployments, you can save the deployment parameters in a configuration file. This file can also be used during upgrades to preserve existing parameters and add any new parameters. Use the config-file flag to specify the path to the configuration file.

         An example of a configuration saved in the /config.ini file is as follows:

         [Configurations]
         format = bash
         cephfs-filesystem-name = <filesystem-name>
         rbd-data-pool-name = <pool_name>
         ...

         Run the following command to set the path to the /config.ini by using the config-file flag:

         python3 ceph-external-cluster-details-exporter.py --config-file /config.ini

         Command line flag

         Retrieve the external cluster details from the IBM Storage Ceph cluster and configure the required parameters mentioned in Table 1 for your deployment.

         Command example:

         python3 ceph-external-cluster-details-exporter.py --rbd-data-pool-name ceph-rbd --monitoring-endpoint xxx.xxx.xxx.xxx --monitoring-endpoint-port xxxx --rgw-endpoint xxx.xxx.xxx.xxx:xxxx --run-as-user client.ocs

         Table 1. List of parameters

         Parameter type	Parameter name	Description
         RBD parameters	rbd-data-pool-name	A mandatory parameter that is used for providing block storage in Fusion Data Foundation.
         	rados-namespace	Divides an RBD data pool into separate logical namespaces, used for creating RBD PVC in a radosNamespace. Flags required with rados-namespace are restricted-auth-permission and k8s-cluster-name.
         	rbd-metadata-ec-pool-name	(Optional) The name of the erasure coded RBD metadata pool.
         RGW parameters	rgw-endpoint	(Optional) This parameter is required only if the object storage is to be provisioned through Ceph Rados Gateway for Fusion Data Foundation. Provide the endpoint in the following format: <ip_address>:<port> Note: A fully-qualified domain name (FQDN) is also supported in the format <FQDN>:<PORT>.
         	rgw-pool-prefix	(Optional) The prefix of the RGW pools. If not specified, the default prefix is default.
         	rgw-tls-cert-path	(Optional) The file path of the RADOS Gateway endpoint TLS certificate. To provide the TLS certificate and RGW endpoint details to the helper script, ceph-external-cluster-details-exporter.py, run the following command: # python3 ceph-external-clustergw-endpoint r-details-exporter.py --rbd-data-pool-name <rbd block pool name> --rgw-endpoint <ip_address>:<port> --rgw-tls-cert-path <file path containing cert> This creates a resource to create a Ceph Object Store CR such as Kubernetes secret containing the TLS certificate. All the intermediate certificates including private keys need to be stored in the certificate file
         	rgw-skip-tls	(Optional) This parameter ignores the TLS certification validation when a self-signed certificate is provided (not recommended).
         Monitoring parameters	monitoring-endpoint	(Optional) This parameter accepts comma-separated list of IP addresses of active and standby mgrs reachable from the OpenShift Container Platform cluster. If not provided, the value is automatically populated.
         	monitoring-endpoint-port	(Optional) It is the port associated with the ceph-mgr Prometheus exporter specified by --monitoring-endpoint. If not provided, the value is automatically populated.
         Ceph parameters	ceph-conf	(Optional) The name of the Ceph configuration file.
         	run-as-user	(Optional) This parameter is used for providing name for the Ceph user which is created by the script. If this parameter is not specified, a default user name client.healthchecker is created. The permissions for the new user is set as: caps: [mgr] allow command config caps: [mon] allow r, allow command quorum_status, allow command version caps: [osd] allow rwx pool=RGW_POOL_PREFIX.rgw.meta, allow r pool=.rgw.root, allow rw pool=RGW_POOL_PREFIX.rgw.control, allow rx pool=RGW_POOL_PREFIX.rgw.log, allow x pool=RGW_POOL_PREFIX.rgw.buckets.index
         CephFS parameters	cephfs-metadata-pool-name	(Optional) The name of the CephFS metadata pool.
         	cephfs-data-pool-name	(Optional) The name of the CephFS data pool.
         	cephfs-filesystem-name	(Optional) The name of the CephFS filesystem.
         Output parameters	dry-run	(Optional) This parameter helps to print the executed commands without running them.
         	output	(Optional) The file where the output is required to be stored.
         Multicluster parameters	k8s-cluster-name	(Optional) Kubernetes cluster name.
         	cluster-name	(Optional) The Ceph cluster name.
         	restricted-auth-permission	(Optional) This parameter restricts cephCSIKeyrings auth permissions to specific pools and clusters. Mandatory flags that need to be set with this are rbd-data-pool-name and cluster-name. You can also pass the cephfs-filesystem-name flag if there is CephFS user restriction so that permission is restricted to a particular CephFS filesystem. Note: This parameter must be applied only for the new deployments. To restrict csi-usersper pool and per cluster, you need to create new csi-users and new secrets for those csi-users.

         Command example with restricted auth permission:

         python3 /etc/ceph/create-external-cluster-resources.py --cephfs-filesystem-name myfs --rbd-data-pool-name replicapool --cluster-name rookStorage --restricted-auth-permission true

         Command example of JSON output generated by using the Python script:

         [{"name": "rook-ceph-mon-endpoints", "kind": "ConfigMap", "data": {"data": "xxx.xxx.xxx.xxx:xxxx", "maxMonId": "0", "mapping": "{}"}}, {"name": "rook-ceph-mon", "kind": "Secret", "data": {"admin-secret": "admin-secret", "fsid": "<fs-id>", "mon-secret": "mon-secret"}}, {"name": "rook-ceph-operator-creds", "kind": "Secret", "data": {"userID": "<user-id>", "userKey": "<user-key>"}}, {"name": "rook-csi-rbd-node", "kind": "Secret", "data": {"userID": "csi-rbd-node", "userKey": "<user-key>"}}, {"name": "ceph-rbd", "kind": "StorageClass", "data": {"pool": "<pool>"}}, {"name": "monitoring-endpoint", "kind": "CephCluster", "data": {"MonitoringEndpoint": "xxx.xxx.xxx.xxx", "MonitoringPort": "xxxx"}}, {"name": "rook-ceph-dashboard-link", "kind": "Secret", "data": {"userID": "ceph-dashboard-link", "userKey": "<user-key>"}}, {"name": "rook-csi-rbd-provisioner", "kind": "Secret", "data": {"userID": "csi-rbd-provisioner", "userKey": "<user-key>"}}, {"name": "rook-csi-cephfs-provisioner", "kind": "Secret", "data": {"adminID": "csi-cephfs-provisioner", "adminKey": "<admin-key>"}}, {"name": "rook-csi-cephfs-node", "kind": "Secret", "data": {"adminID": "csi-cephfs-node", "adminKey": "<admin-key>"}}, {"name": "cephfs", "kind": "StorageClass", "data": {"fsName": "cephfs", "pool": "cephfs_data"}}, {"name": "ceph-rgw", "kind": "StorageClass", "data": {"endpoint": "xxx.xxx.xxx.xxx:xxxx", "poolPrefix": "default"}}, {"name": "rgw-admin-ops-user", "kind": "Secret", "data": {"accessKey": "<access-key>", "secretKey": "<secret-key>"}}]

      3. Save the JSON output to a file with .json extension.
         Note: For Fusion Data Foundation to work seamlessly, ensure that the parameters (RGW endpoint, CephFS details, RBD pool, and so on) uploaded by using the JSON file remain unchanged on the external IBM Storage Ceph cluster after the storage cluster is created.
      4. Run the command in a multi-tenant deployment where the IBM Storage Ceph cluster is already connected to an IBM Storage Ceph deployment with an earlier version.

         python3 ceph-external-cluster-details-exporter.py --upgrade

   3. Click Browse to select and upload the JSON file.
      The JSON file content is populated and displayed in the text box.
5. Optional: Select In-transit encryption to enable encryption for entire Ceph traffic including data using Ceph msgrv2.
   Note: Verify that the external IBM Storage Ceph cluster is configured for in‑transit encryption. Before enabling this setting, use the documentation link in the UI to complete the required configuration steps.

   1. 1. 1. (Optional) Select the advanced configuration options that apply to your environment:

      2. Review the selections and click Connect.
6. Optional: Select the required Advanced configuration option that apply to your environment:

   • Use Ceph RBD as the default StorageClass

     Sets the RBD StorageClass as the cluster default.

   • Use external PostgreSQL (Technology preview)

     Uses an external PostgreSQL instance instead of the in‑cluster database.

     Note: Additional advanced options may be available in the UI. Select only the options that are required for the deployment policy and environment.

   • Enable Automatic backup

     Allows automatic backup for the Multicloud Object Gateway metadata database.

     • Select the backup frequency: Daily, Weekly, or Monthly.

     • Specify the number of backups to retain.

       Note: This option is disabled if you select Use external PostgreSQL.
7. Review the details, and then click Connect.
8. Validate whether Fusion Data Foundation is successfully configured with an external IBM Storage Ceph cluster in external mode by following the instructions that are mentioned in Verifying Fusion Data Foundation configuration with IBM Storage Ceph.