Deploying Data Foundation in external mode

Fusion Data Foundation can make services from an external IBM Storage Ceph cluster available for consumption through OpenShift Container Platform clusters.

1. Install the IBM Fusion Data Foundation operator.
   1. From the Red Hat OpenShift Container Platform web management console, go to Operators > Operator Hub and search for IBM Fusion Data Foundation.
   2. Click IBM Fusion Data Foundation and then click Install.
   3. Ensure that the Enable option is selected for the Console plugin.
   4. Retain all the other default settings:
      • Select the Update Channel.
      • Set Installation Mode to A specific namespace on the cluster.
      • Set Installed Namespace as Operator recommended namespace, openshift-storage. If Namespace openshift-storage does not exist, it is created during the operator installation.
      • Select Approval Strategy as Automatic or Manual.

        Automatic

        If you select Automatic updates, then the Operator Lifecycle Manager (OLM) automatically upgrades the running instance of your Operator without any intervention.

        Manual

        If you select Manual updates, then the OLM creates an update request. As a cluster administrator, you must then manually approve that update request to update the Operator to a newer version.

   5. Click Install.
2. Verify the operator installation.
   1. After the operator is successfully installed, a pop-up with a message, Web console update is available appears on the user interface. Click Refresh web console from this pop-up for the console changes to reflect.
   2. From the Web Console do one of the following:
      • Navigate to Installed Operators and verify that the IBM Fusion Data Foundation Operator shows a green tick indicating successful installation.
      • Navigate to Storage and verify if Data Foundation dashboard is available.
3. Create a StorageSystem.
   1. View all installed operators.
      Operators > Installed Operators. Ensure that the Project selected is openshift-storage.
   2. Click the installed operator IBM Fusion Data Foundation and then click Create StorageSystem.
   3. In the Backing storage page, select the following options:
      • Select Full deployment for the Deployment type option.
      • Select Connect an external storage platform from the available options.
      • Select IBM Storage Ceph for Storage platform.
   4. Click Next.
   5. In the Connection details page, provide the necessary information:
      1. Click on the Download Script link to download the python script for extracting Ceph cluster details.
      2. For extracting the IBM Storage Ceph cluster details, contact the IBM Storage Ceph administrator to run the downloaded python script on a IBM Storage Ceph node with the admin key.
         1. Run the following command on the IBM Storage Ceph node to view the list of available arguments:

            python3 ceph-external-cluster-details-exporter.py --help

            Important: Use python instead of python3 if the Ceph Storage cluster is deployed on Red Hat Enterprise Linux 7.x (RHEL 7.x) cluster.

            You can also run the script from inside a MON container (containerized deployment) or from a MON node (RPM deployment).

            Note: Use the yum install cephadm command and then the cephadm command to deploy your IBM Storage Ceph cluster using containers. You must pull the IBM Storage Ceph cluster container images using the cephadm command, rather than using yum for installing the Ceph packages onto nodes.

            For more information, see IBM Storage Ceph documentation.

         2. To retrieve the external cluster details from the IBM Storage Ceph cluster, use one of the following options:

            • Use the config-file flag. This stores the parameters used during deployment.

              In new deployments, you can save the parameters used during deployment in a configuration file. This file can be used during the upgrade to preserve the parameters as well as add any additional parameters. Use the config-file flag to set the path to the configuration file.

              An example of a configuration saved in the /config.ini is as follows:

              [Configurations]
              format = bash
              cephfs-filesystem-name = <filesystem-name>
              rbd-data-pool-name = <pool_name>
              ...

              Run the following command to set the path to the /config.ini using the config-file flag:

              python3 ceph-external-cluster-details-exporter.py --config-file /config.ini

            • Retrieve the external cluster details from the IBM Storage Ceph cluster and pass the required parameters mentioned in Table 1 for your deployment.

              python3 ceph-external-cluster-details-exporter.py --rbd-data-pool-name <rbd block pool name>  [optional arguments]

              For example:

              python3 ceph-external-cluster-details-exporter.py --rbd-data-pool-name ceph-rbd --monitoring-endpoint xxx.xxx.xxx.xxx --monitoring-endpoint-port xxxx --rgw-endpoint xxx.xxx.xxx.xxx:xxxx --run-as-user client.ocs

              Table 1. List of parameters

              Parameter type	Parameter name	Description
              RBD parameters	rbd-data-pool-name	A mandatory parameter that is used for providing block storage in Fusion Data Foundation.
              	rados-namespace	Divides an RBD data pool into separate logical namespaces, used for creating RBD PVC in a radosNamespace. Flags required with rados-namespace are restricted-auth-permission and k8s-cluster-name.
              	rbd-metadata-ec-pool-name	(Optional) The name of the erasure coded RBD metadata pool.
              RGW parameters	rgw-endpoint	(Optional) This parameter is required only if the object storage is to be provisioned through Ceph Rados Gateway for Fusion Data Foundation. Provide the endpoint in the following format: <ip_address>:<port> Note: A fully-qualified domain name (FQDN) is also supported in the format <FQDN>:<PORT>.
              	rgw-pool-prefix	(Optional) The prefix of the RGW pools. If not specified, the default prefix is default.
              	rgw-tls-cert-path	(Optional) The file path of the RADOS Gateway endpoint TLS certificate. To provide the TLS certificate and RGW endpoint details to the helper script, ceph-external-cluster-details-exporter.py, run the following command: # python3 ceph-external-clustergw-endpoint r-details-exporter.py --rbd-data-pool-name <rbd block pool name> --rgw-endpoint <ip_address>:<port> --rgw-tls-cert-path <file path containing cert> This creates a resource to create a Ceph Object Store CR such as Kubernetes secret containing the TLS certificate. All the intermediate certificates including private keys need to be stored in the certificate file
              	rgw-skip-tls	(Optional) This parameter ignores the TLS certification validation when a self-signed certificate is provided (not recommended).
              Monitoring parameters	monitoring-endpoint	(Optional) This parameter accepts comma-separated list of IP addresses of active and standby mgrs reachable from the OpenShift Container Platform cluster. If not provided, the value is automatically populated.
              	monitoring-endpoint-port	(Optional) It is the port associated with the ceph-mgr Prometheus exporter specified by --monitoring-endpoint. If not provided, the value is automatically populated.
              Ceph parameters	ceph-conf	(Optional) The name of the Ceph configuration file.
              	run-as-user	(Optional) This parameter is used for providing name for the Ceph user which is created by the script. If this parameter is not specified, a default user name client.healthchecker is created. The permissions for the new user is set as: caps: [mgr] allow command config caps: [mon] allow r, allow command quorum_status, allow command version caps: [osd] allow rwx pool=RGW_POOL_PREFIX.rgw.meta, allow r pool=.rgw.root, allow rw pool=RGW_POOL_PREFIX.rgw.control, allow rx pool=RGW_POOL_PREFIX.rgw.log, allow x pool=RGW_POOL_PREFIX.rgw.buckets.index
              CephFS parameters	cephfs-metadata-pool-name	(Optional) The name of the CephFS metadata pool.
              	cephfs-data-pool-name	(Optional) The name of the CephFS data pool.
              	cephfs-filesystem-name	(Optional) The name of the CephFS filesystem.
              Output parameters	dry-run	(Optional) This parameter helps to print the executed commands without running them.
              	output	(Optional) The file where the output is required to be stored.
              Multicluster parameters	k8s-cluster-name	(Optional) Kubernetes cluster name.
              	cluster-name	(Optional) The Ceph cluster name.
              	restricted-auth-permission	(Optional) This parameter restricts cephCSIKeyrings auth permissions to specific pools and clusters. Mandatory flags that need to be set with this are rbd-data-pool-name and cluster-name. You can also pass the cephfs-filesystem-name flag if there is CephFS user restriction so that permission is restricted to a particular CephFS filesystem. Note: This parameter must be applied only for the new deployments. To restrict csi-usersper pool and per cluster, you need to create new csi-users and new secrets for those csi-users.

              Example with restricted auth permission:

              python3 /etc/ceph/create-external-cluster-resources.py --cephfs-filesystem-name myfs --rbd-data-pool-name replicapool --cluster-name rookStorage --restricted-auth-permission true

              Example of JSON output generated using the python script:

              [{"name": "rook-ceph-mon-endpoints", "kind": "ConfigMap", "data": {"data": "xxx.xxx.xxx.xxx:xxxx", "maxMonId": "0", "mapping": "{}"}}, {"name": "rook-ceph-mon", "kind": "Secret", "data": {"admin-secret": "admin-secret", "fsid": "<fs-id>", "mon-secret": "mon-secret"}}, {"name": "rook-ceph-operator-creds", "kind": "Secret", "data": {"userID": "<user-id>", "userKey": "<user-key>"}}, {"name": "rook-csi-rbd-node", "kind": "Secret", "data": {"userID": "csi-rbd-node", "userKey": "<user-key>"}}, {"name": "ceph-rbd", "kind": "StorageClass", "data": {"pool": "<pool>"}}, {"name": "monitoring-endpoint", "kind": "CephCluster", "data": {"MonitoringEndpoint": "xxx.xxx.xxx.xxx", "MonitoringPort": "xxxx"}}, {"name": "rook-ceph-dashboard-link", "kind": "Secret", "data": {"userID": "ceph-dashboard-link", "userKey": "<user-key>"}}, {"name": "rook-csi-rbd-provisioner", "kind": "Secret", "data": {"userID": "csi-rbd-provisioner", "userKey": "<user-key>"}}, {"name": "rook-csi-cephfs-provisioner", "kind": "Secret", "data": {"adminID": "csi-cephfs-provisioner", "adminKey": "<admin-key>"}}, {"name": "rook-csi-cephfs-node", "kind": "Secret", "data": {"adminID": "csi-cephfs-node", "adminKey": "<admin-key>"}}, {"name": "cephfs", "kind": "StorageClass", "data": {"fsName": "cephfs", "pool": "cephfs_data"}}, {"name": "ceph-rgw", "kind": "StorageClass", "data": {"endpoint": "xxx.xxx.xxx.xxx:xxxx", "poolPrefix": "default"}}, {"name": "rgw-admin-ops-user", "kind": "Secret", "data": {"accessKey": "<access-key>", "secretKey": "<secret-key>"}}]

         3. To retrieve the external cluster details from the IBM Storage Ceph cluster, run the following command:

            python3 ceph-external-cluster-details-exporter.py --rbd-data-pool-name <rbd block pool name>  [optional arguments]

            For example:

            python3 ceph-external-cluster-details-exporter.py --rbd-data-pool-name ceph-rbd --monitoring-endpoint xxx.xxx.xxx.xxx --monitoring-endpoint-port xxxx --rgw-endpoint xxx.xxx.xxx.xxx:xxxx --run-as-user client.ocs

            Example with restricted auth permission:

            python3 /etc/ceph/create-external-cluster-resources.py --cephfs-filesystem-name myfs --rbd-data-pool-name replicapool --cluster-name rookStorage --restricted-auth-permission true

            Example of JSON output generated using the python script:

            [{"name": "rook-ceph-mon-endpoints", "kind": "ConfigMap", "data": {"data": "xxx.xxx.xxx.xxx:xxxx", "maxMonId": "0", "mapping": "{}"}}, {"name": "rook-ceph-mon", "kind": "Secret", "data": {"admin-secret": "admin-secret", "fsid": "<fs-id>", "mon-secret": "mon-secret"}}, {"name": "rook-ceph-operator-creds", "kind": "Secret", "data": {"userID": "<user-id>", "userKey": "<user-key>"}}, {"name": "rook-csi-rbd-node", "kind": "Secret", "data": {"userID": "csi-rbd-node", "userKey": "<user-key>"}}, {"name": "ceph-rbd", "kind": "StorageClass", "data": {"pool": "<pool>"}}, {"name": "monitoring-endpoint", "kind": "CephCluster", "data": {"MonitoringEndpoint": "xxx.xxx.xxx.xxx", "MonitoringPort": "xxxx"}}, {"name": "rook-ceph-dashboard-link", "kind": "Secret", "data": {"userID": "ceph-dashboard-link", "userKey": "<user-key>"}}, {"name": "rook-csi-rbd-provisioner", "kind": "Secret", "data": {"userID": "csi-rbd-provisioner", "userKey": "<user-key>"}}, {"name": "rook-csi-cephfs-provisioner", "kind": "Secret", "data": {"adminID": "csi-cephfs-provisioner", "adminKey": "<admin-key>"}}, {"name": "rook-csi-cephfs-node", "kind": "Secret", "data": {"adminID": "csi-cephfs-node", "adminKey": "<admin-key>"}}, {"name": "cephfs", "kind": "StorageClass", "data": {"fsName": "cephfs", "pool": "cephfs_data"}}, {"name": "ceph-rgw", "kind": "StorageClass", "data": {"endpoint": "xxx.xxx.xxx.xxx:xxxx", "poolPrefix": "default"}}, {"name": "rgw-admin-ops-user", "kind": "Secret", "data": {"accessKey": "<access-key>", "secretKey": "<secret-key>"}}]

         4. Save the JSON output to a file with .json extension.
            Note: For Fusion Data Foundation to work seamlessly, ensure that the parameters (RGW endpoint, CephFS details, RBD pool, and so on) to be uploaded using the JSON file remains unchanged on the IBM Storage Ceph external cluster after the storage cluster creation.
         5. Run the command when there is a multi-tenant deployment in which IBM Storage Ceph cluster is already connected to Fusion Data Foundation deployment with a lower version.

            python3 ceph-external-cluster-details-exporter.py --upgrade

      3. Click Browse to select and upload the JSON file.

         The content of the JSON file is populated and displayed in the text box.

      4. Click Next

         The Next button is enabled only after you upload the JSON file.

   6. Review if all the details are correct from the Review and create page.
      To modify any configuration settings, click Back to go back to the previous configuration page.
   7. Click Create StorageSystem.
4. Verify the StorageSystem creation
   1. From the OpenShift Web Console, navigate to Installed Operators > IBM Fusion Data Foundation > Storage System > ocs-external-storagecluster-storagesystem > Resources.
   2. Verify that StorageCluster is in a Ready state and has a green tick.