Troubleshooting CephFS PVC creation in external mode

If you have updated the IBM Storage Ceph cluster from a version lower than 4.1.1 to the latest release and is not a freshly deployed cluster, you must manually set the application type for CephFS pool on the IBM Storage Ceph cluster to enable CephFS PVC creation in external mode.

1. Check for CephFS pvc stuck in Pending status.

   oc get pvc -n <namespace>

   Example output :

   NAME                      STATUS    VOLUME
   CAPACITY  ACCESS MODES    STORAGECLASS                        AGE
   ngx-fs-pxknkcix20-pod     Pending
                             ocs-external-storagecluster-cephfs  28h
   [...]

2. Check the describe output to see the events for respective PVC.
   Expected error message is:

   cephfs_metadata/csi.volumes.default/csi.volume.pvc-xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx: (1) Operation not permitted)

   oc describe pvc ngx-fs-pxknkcix20-pod -n nginx-file

   Example output:

   Name:          ngx-fs-pxknkcix20-pod
   Namespace:     nginx-file
   StorageClass:  ocs-external-storagecluster-cephfs
   Status:        Pending
   Volume:
   Labels:        <none>
   Annotations:   volume.beta.kubernetes.io/storage-provisioner: openshift-storage.cephfs.csi.ceph.com
   Finalizers:    [kubernetes.io/pvc-protection]
   Capacity:
   Access Modes:
   VolumeMode:    Filesystem
   Mounted By:    ngx-fs-oyoe047v2bn2ka42jfgg-pod-hqhzf
   Events:
    Type     Reason              Age                   From                                                                                                                      Message
    ----     ------              ----                  ----                                                                                                                      -------
    Warning  ProvisioningFailed  107m (x245 over 22h)  openshift-storage.cephfs.csi.ceph.com_csi-cephfsplugin-provisioner-5f8b66cc96-hvcqp_6b7044af-c904-4795-9ce5-bf0cf63cc4a4
    (combined from similar events): failed to provision volume with StorageClass "ocs-external-storagecluster-cephfs": rpc error: code = Internal desc = error (an error (exit status 1) occurred while
    running rados args: [-m 192.168.13.212:6789,192.168.13.211:6789,192.168.13.213:6789 --id csi-cephfs-provisioner --keyfile=stripped -c /etc/ceph/ceph.conf -p cephfs_metadata getomapval
    csi.volumes.default csi.volume.pvc-1ac0c6e6-9428-445d-bbd6-1284d54ddb47 /tmp/omap-get-186436239 --namespace=csi]) occurred, command output streams is ( error getting omap value
    cephfs_metadata/csi.volumes.default/csi.volume.pvc-1ac0c6e6-9428-445d-bbd6-1284d54ddb47: (1) Operation not permitted)

3. Check the settings for the <cephfs metadata pool name>.
   In this example, <cephfs metadata pool name> is cephfs_metadata and <cephfs data pool name> is cephfs_data. For running the command, you will need jq preinstalled in the IBM Storage Ceph client node.

   ceph osd pool ls detail --format=json | jq '.[] | select(.pool_name| startswith("cephfs")) | .pool_name, .application_metadata' "cephfs_data"
   {
     "cephfs": {}
   }
   "cephfs_metadata"
   {
      "cephfs": {}
   }

4. Set the application type for CephFS pool.
   Run the following commands on the IBM Storage Ceph client node:

   ceph osd pool application set <cephfs metadata pool name> cephfs metadata cephfs ceph osd pool application set <cephfs data pool name> cephfs data cephfs

5. Verify if the settings are applied.

   ceph osd pool ls detail --format=json | jq '.[] | select(.pool_name| startswith("cephfs")) | .pool_name, .application_metadata' "cephfs_data"
     

   { "cephfs": { "data": "cephfs" } } "cephfs_metadata" { "cephfs": { "metadata": "cephfs" } }

6. Check the CephFS PVC status again.
   The PVC should now be in Bound state.

   oc get pvc -n <namespace>

   Example output :

   NAME                      STATUS    VOLUME
   CAPACITY  ACCESS MODES    STORAGECLASS                        AGE
   ngx-fs-pxknkcix20-pod     Bound     pvc-1ac0c6e6-9428-445d-bbd6-1284d54ddb47
   1Mi       RWO             ocs-external-storagecluster-cephfs  29h
   [...]