Understanding the storage clusters usage stats

Edit online

To check a cluster’s data usage and data distribution among pools, use the df option. It is similar to the Linux df command.

The SIZE/AVAIL/RAW USED in the ceph df and ceph status command output are different if some OSDs are marked OUT of the cluster compared to when all OSDs are IN. The SIZE/AVAIL/RAW USED is calculated from sum of SIZE (osd disk size), RAW USE (total used space on disk), and AVAIL of all OSDs which are in IN state. You can see the total of SIZE/AVAIL/RAW USED for all OSDs in ceph osd df tree command output.

Example

    [ceph: root@host01 /]#ceph df
    --- RAW STORAGE ---
    CLASS   SIZE    AVAIL     USED  RAW USED  %RAW USED
    hdd    5 TiB  2.9 TiB  2.1 TiB   2.1 TiB      42.98                                            
    TOTAL  5 TiB  2.9 TiB  2.1 TiB   2.1 TiB      42.98                                            
    
    --- POOLS ---
    POOL                        ID  PGS   STORED  OBJECTS     USED  %USED  MAX AVAIL
    .mgr                         1    1  5.3 MiB        3   16 MiB      0    629 GiB
    .rgw.root                    2   32  1.3 KiB        4   48 KiB      0    629 GiB
    default.rgw.log              3   32  3.6 KiB      209  408 KiB      0    629 GiB
    default.rgw.control          4   32      0 B        8      0 B      0    629 GiB
    default.rgw.meta             5   32  1.7 KiB       10   96 KiB      0    629 GiB
    default.rgw.buckets.index    7   32  5.5 MiB       22   17 MiB      0    629 GiB
    default.rgw.buckets.data     8   32  807 KiB        3  2.4 MiB      0    629 GiB
    default.rgw.buckets.non-ec   9   32  1.0 MiB        1  3.1 MiB      0    629 GiB

source-ecpool-86 11 32 1.2 TiB 391.13k 2.1 TiB 53.49 1.1 TiB

The ceph df detail command gives more details about other pool statistics such as quota objects, quota bytes, used compression, and under compression.

The RAW STORAGE section of the output provides an overview of the amount of storage the storage cluster manages for data.

  • CLASS: The class of OSD device.

  • SIZE: The amount of storage capacity managed by the storage cluster.

    In the above example, if the SIZE is 90 GiB, it is the total size without the replication factor, which is three by default. The total available capacity with the replication factor is 90 GiB/3 = 30 GiB. Based on the full ratio, which is 0.85% by default, the maximum available space is 30 GiB * 0.85 = 25.5 GiB

  • AVAIL: The amount of free space available in the storage cluster.

    In the above example, if the SIZE is 90 GiB and the USED space is 6 GiB, then the AVAIL space is 84 GiB. The total available space with the replication factor, which is three by default, is 84 GiB/3 = 28 GiB

  • USED: The amount of raw storage consumed by user data.

    In the above example, 100 MiB is the total space available after considering the replication factor. The actual available size is 33 MiB.

  • RAW USED: The amount of raw storage consumed by user data, internal overhead, or reserved capacity.

  • % RAW USED: The percentage of RAW USED. Use this number in conjunction with the full ratio and near full ratio to ensure that you are not reaching the storage cluster’s capacity.

The POOLS section of the output provides a list of pools and the notional usage of each pool. The output from this section DOES NOT reflect replicas, clones or snapshots. For example, if you store an object with 1 MB of data, the notional usage will be 1 MB, but the actual usage may be 3 MB or more depending on the number of replicas for example, size = 3, clones and snapshots.

  • POOL: The name of the pool.

  • ID: The pool ID.

  • STORED: The actual amount of data stored by the user in the pool. This value changes based on the raw usage data based on (k+M)/K values, number of object copies, and the number of objects degraded at the time of pool stats calculation.

  • OBJECTS: The notional number of objects stored per pool. It is STORED size * replication factor.

  • USED: The notional amount of data stored in kilobytes, unless the number appends M for megabytes or G for gigabytes.

  • %USED: The notional percentage of storage used per pool.

  • MAX AVAIL: An estimate of the notional amount of data that can be written to this pool. It is the amount of data that can be used before the first OSD becomes full. It considers the projected distribution of data across disks from the CRUSH map and uses the first OSD to fill up as the target.

    In the above example, MAX AVAIL is 153.85 MB without considering the replication factor, which is three by default.

    See the Knowledgebase article titled ceph df MAX AVAIL is incorrect for simple replicated pool to calculate the value of MAX AVAIL.

  • QUOTA OBJECTS: The number of quota objects.

  • QUOTA BYTES: The number of bytes in the quota objects.

  • USED COMPR: The amount of space allocated for compressed data including his includes compressed data, allocation, replication and erasure coding overhead.

  • UNDER COMPR: The amount of data passed through compression and beneficial enough to be stored in a compressed form.

NOTE: The numbers in the POOLS section are notional. They are not inclusive of the number of replicas, snapshots or clones. As a result, the sum of the USED and %USED amounts will not add up to the RAW USED and %RAW USED amounts in the GLOBAL section of the output.

NOTE: The MAX AVAIL value is a complicated function of the replication or erasure code used, the CRUSH rule that maps storage to devices, the utilization of those devices, and the configured mon_osd_full_ratio.