Ceph Scrubbing And Its Parameters

Answer

Ceph OSDs are responsible for storing, retrieving, and protecting data stored in the Reliable Autonomic Distributed Object Store (RADOS). In order to check the coherence of the data stored, all copies of data  must be periodically checked in order to verify that all are identical. The verification process proceeds at PG (placement group) granularity and is driven by the primary OSD assigned to each PG.  This is known as scrubbing.

Scrubbing types
1. Light (aka Shallow) scrubbing
Light scrubbing checks metadata: that each RADOS object replica or shard serving the PG has the same size and the same digest.
2. Deep Scrubbing
Deep scrubbing reads the entire payload data of each RADOS object within the PG on each OSD in order to recalculate the digest and compare the result among all OSDs that serve the PG.  This helps detect latent corruption.

Automated scrubbing
   Each PG is frequently light scrubbed. The options influencing the light scrubbing process are:
- osd_scrub_min_interval Minimum interval between two light scrubs when load is low. Defaults to 1 day.
- osd_scrub_max_interval Maximum interval between two light scrubs. Defaults to a week.
- osd_scrub_load_threshold Light scrubbing occurs if system load average is below this value. Defaults to 10. 
- osd_deep_scrub_interval The maximum tolerated interval between two deep scrubs irrespective of load. Defaults to a week.
- osd_max_scrubs Maximum number of concurrent PG scrub concurrent operations per OSD. Defaults to 1.
- osd_scrub_sleep Delay between two light or deep scrub operations. Defaults to 0.0.

It is possible to disable automated light and deep scrubbing operations in favor of manual scheduling mechanism in order to control the exact time these operations take place.
- ceph osd set noscrub Disable automated light scrubbing
- ceph osd set nodeep-scrub Disable automated light scrubbing

These flags when raised  are shown by the ceph status command. Most clusters are best served by allowing scrubs to be scheduled automatically.

    cluster 8c5d3515-4dab-436d-92b8-267bd4f1185c
     health HEALTH_WARN noscrub,nodeep-scrub flag(s) set
     monmap e1: 1 mons at {daisy=192.168.122.114:6789/0}, election epoch 1, quorum 0 daisy
     osdmap e22: 3 osds: 3 up, 3 in
            flags noscrub,nodeep-scrub
      pgmap v151: 192 pgs, 3 pools, 0 bytes data, 0 objects
            102 MB used, 27512 MB / 27614 MB avail
                 192 active+clean

Manual scrubbing
It is possible to perform a manual scrub operation at any time. You can manually trigger a scrub operation using the following commands.
- ceph pg scrub {pgid} Trigger a one shot light scrubbing operation on the specified PG.
- ceph pg deep-scrub {pgid} Trigger a one shot deep scrubbing operation on the specified PG.
- ceph osd scrub {osdid} Trigger a one shot light scrubbing operation of all the PGs managed by the specified OSD.
- ceph osd deep-scrub {osdid} Trigger a deep scrubbing operation of all PGs managed by the specified OSD.

Note that manually requested scrubs are still subject to reservation dynamics and parallelism limits; these commands influence scheduling priority.

Scheduled scrubbing operations via cron
If you have decided to manage scrubs manually, please refer to the following article. This approach is usually not recommended.