Nearfull OSDs

Understand and troubleshoot nearfull OSDs.

If OSDs are considered nearfull, the ceph health detail command returns an error message, similar to the following example:

HEALTH_WARN 1 nearfull osds
osd.2 is near full at 95%

What this means

Ceph returns the nearfull osds message when the cluster reaches the capacity set by the ceph osd set-nearfull-ratio VALUE command.

For the latest default parameter settings and percentages, see Full OSDs.

Ceph distributes data based on the CRUSH hierarchy in the best possible way but it cannot guarantee equal distribution. The main causes of the uneven data distribution and the nearfull osds messages are:

The OSDs are not balanced among the OSD nodes in the cluster. That is, some OSD nodes host more aggregate capacity than others.
The placement group (PG) count is not proper as per the number of the OSDs, use case, target PGs per OSD, and OSD utilization.
The cluster uses inappropriate CRUSH tunables.
The back-end storage for OSDs is almost full.
The OSDs have different sizes.
Note: This is only an issue if the pool has too few PGs or if the balancer module is not enabled. Ensure that the target autoscaler ratio is changed from the default and set to 300.

For more information, see the following:

Troubleshooting this problem

Note: Ratios can trigger even if the ceph osd df command shows OSDs slightly below the thresholds. A buffer of 1 - 1.5% is common.

Verify that the PG count is sufficient, by using the ceph osd df command.
The PG column output should be between 100 and 300.

If needed, increase the count. For more information, see Increasing the placement group.
Note: When ratio values are changed it can take time to take effect. To expedite the value change, you can restart the affected OSDs in a safe rolling fashion.
Verify that you are using optimal CRUSH tunables to the cluster version. If needed, adjust the CRUSH tunables. For more information, see CRUSH tuning.
Use the Ceph Manager balancer module to balance the OSDs.
For more information, seeUsing the Ceph Manager balancer module .
Determine how much space remains on the disks that are used by OSDs.
- To view how much space OSDs use in general.
```
ceph osd df
```
  For example,
```
[ceph: root@host01 /]# ceph osd df
```
- To view how much space OSDs use on a particular node, use the df command from the node containing nearfull OSDs.
  For example,
```
[ceph: root@host01 /]# df
```
If needed, add an OSD node.

Additional considerations and best practices

If ratios are raised during an emergency, restore them afterward. Leaving them elevated increases the risk of outages.
Very large clusters may raise ratios by a few percent to reserve capacity.
Always check that the balancer is working and that there are no overlapping CRUSH roots. Overlaps often occur when some CRUSH rules specify a device class and others do not.
Run fstrim on Ceph Block Device clients weekly by using cron or one-time during crisis. Avoid using the discard mount option as it impacts performance continuously.
Check for unused RADOS pools, empty the Ceph Block Device trash, and orphaned rados bench objects. Remove carefully by using the rados rm command.