Fusion Data Foundation service error scenarios

Problem statement

The Fusion Data Foundation storage cluster is in a degraded state.

Cause

This issue might occurs due to many reasons. Follow the steps to find the root cause few the issues:

1. Log in to the OpenShift Container Platform console.
2. Go to Workloads > Pods.
3. Select the Openshift-storage form the Projects drop down.
4. Click CephClusters CR instance and go to YAML tab.
5. Check status field to find the exact failure reason. If the issue occurs because of MDS trimming, then follow steps mentioned in resolution section to resolve the issue.

Resolution

1. Log in to the OpenShift Container Platform console.
2. Go to Workloads > Pods.
3. Select the Openshift-storage form the Projects drop down.
4. Select the rook-ceph-operator pod from the list.
5. Go to Terminal tab, and run the following commands:

   
   export CEPH_ARGS='-c /var/lib/rook/openshift-storage/openshift-storage.config'
   ceph -s

   Example output:

   cluster:
       id:     0e7a1d51-ad98-4b33-ab71-4577de1b3e7d
       health: HEALTH_WARN
               2 MDSs behind on trimming
    
     services:
       mon: 3 daemons, quorum e,f,g (age 5d)
       mgr: a(active, since 14h), standbys: b
       mds: 1/1 daemons up, 1 hot standby
       osd: 5 osds: 5 up (since 6d), 5 in (since 6d)
       rgw: 1 daemon active (1 hosts, 1 zones)
    
     data:
       volumes: 1/1 healthy
       pools:   12 pools, 265 pgs
       objects: 1.54M objects, 1.3 TiB
       usage:   4.3 TiB used, 31 TiB / 35 TiB avail
       pgs:     265 active+clean
    
     io:
       client:   121 MiB/s rd, 119 MiB/s wr, 125 op/s rd, 165 op/s wr

   ceph config get mds.{id} mds_log_max_segments 128
   ceph config set mds mds_log_max_segments 256
   ceph config get mds.{id} mds_log_max_segments 256
   ceph -s

   Example output:

   cluster:
       id:     0e7a1d51-ad98-4b33-ab71-4577de1b3e7d
       health: HEALTH_OK
    
     services:
       mon: 3 daemons, quorum e,f,g (age 5d)
       mgr: a(active, since 14h), standbys: b
       mds: 1/1 daemons up, 1 hot standby
       osd: 5 osds: 5 up (since 6d), 5 in (since 6d)
       rgw: 1 daemon active (1 hosts, 1 zones)
    
     data:
       volumes: 1/1 healthy
       pools:   12 pools, 265 pgs
       objects: 1.54M objects, 1.3 TiB
       usage:   4.3 TiB used, 31 TiB / 35 TiB avail
       pgs:     265 active+clean
    
     io:
       client:   101 MiB/s rd, 109 MiB/s wr, 80 op/s rd, 128 op/s wr

For more information about the issue, see Red Hat Customer Portal.

Admission webhook warning :- ObjectBucketClaim my-obj-bucket violates policy 299 - "unknown field "spec.ss||"' after object bucket claim"*

Problem statement

When you configure a Fusion Data Foundation cluster, you do not find any candidate storage nodes.

Cause

When you configure Fusion Data Foundation cluster, only compute nodes with available disks (SSD/NVMe or HDD) get displayed in the Data Foundation page of IBM Fusion user interface. The following nodes get filtered out and do not display on the screen:

• Nodes have SSD/NVMe or HDD disks but they are not in available state
• The selected disk properties are not present in current node. For example, disk size or disk type.
• The total disk count (with same disk size, disk type) is less than 3.

Steps to verify whether you have the correct storage node candidates

1. In Red Hat OpenShift Container Platform console, go to Operators > Installed Operators.
2. Verify whether the LocalStorage operator is installed successfully.
3. Run the following command to get all the worker nodes:

   oc get node -l node-role.kubernetes.io/worker=

4. Run the following command to check if discovery results are created for all worker nodes.

   oc get localvolumediscoveryresult -n openshift-local-storage

5. Run the following command to confirm that none of the nodes have a Fusion Data Foundation storage label:

   oc get node -l cluster.ocs.openshift.io/openshift-storage=

Note: In Linux on IBM Z platform, disks might be formatted and partitioned first. For more information about this behavior, see Red Hat OpenShift Data Foundation on IBM Z and IBM LinuxONE - Reference Architecture section 4.1.1 and 4.1.2.