IBM Storage Ceph: BlueStore OSD bluestore_min_alloc_size, bluestore_min_alloc_size_hdd, bluestore_min_alloc_size_ssd values and ramifications

Steps

• BlueStore OSDs allocate and manage underlying raw storage in chunks of at smallest bluestore_min_alloc_size.
• bluestore_min_alloc_size_hdd is used for HDD based OSDs and bluestore_min_alloc_size_ssd is used for SSD and NVMe based OSDs.
• Default values are :

  # ceph config show-with-defaults osd.0 | grep bluestore_min_alloc_size
  bluestore_min_alloc_size         0                                                                                                                                                                                                                                                                                            
  bluestore_min_alloc_size_hdd     4096                                                                                                                                                                                                                                                                                         
  bluestore_min_alloc_size_ssd     4096         

• For example a 1 KB RADOS object will allocate 4KB of raw capacity, 3KB of which is padding.  We call this space amplification. Note that IBM Storage Ceph 9 brings improvements that partly mitigate this effect.
• The value of 0 for bluestore_min_alloc_size indicates that the value to use for a given OSD will be that of bluestore_min_alloc_size_hdd or bluestore_min_alloc_size_ssd, depending on the detected device type when each OSD is created. The device type is determined by the kernel's rotational attribute for the main OSD block device. For example, an HDD reports as below:

  # cat  /sys/block/sdp/queue/rotational
  1

• The 4KB defaults in recent IBM Storage Ceph releases work well for most deployments on HDDs or conventional SSDs.

• There are rare use-cases that may benefit from raising these values, for example to the historical setting of 64 KB.  The defaults were lowered because this larger size resulted in significant space amplification for RGW and CephFS pools storing large numbers of user objects or files smaller than several multiples of the min_alloc_size, especially for erasure coded (EC) pools.  IBM advises that the default 4 KB values not be modified.  If you suspect that you would benefit from adjusted values, please contact your IBM Storage Ceph support team for guidance.

• Recent and QLC and QLC-class SSDs may have a coarse IU (indirection unit) that is larger than 4KB.  Examples include the Intel / Solidigm P5316 and P5336 and the Micron 6550 ION.  These novel models perform best when writes are aligned to the IU boundaries.  When provisioning OSDs on such SSDs, be sure to set the below value before deployment:
   

  ceph config set global bluestore_use_optimal_io_size_for_min_alloc_size true

  This directs the OSD creation logic to query /sys/block/sdp/queue/optimal_io_size for each OSD main block device.  With recent kernels this value will reflect the appropriate value for coarse IU QLC-class SSDs.  If the value is 0, the logic will fall back to the above 4KB defaults.