Tuning considerations for IBM Storage Ceph

BIOS and BMC settings

Operating system tuning with TuneD

1. Install TuneD.

   # dnf install -y tuned
   # tuned-adm active
   Current active profile: balanced

2. Activate the network-latency profile.

   The network-latency profile prevents deep CPU C-state transitions. These transitions are slow and can cause performance degradation or packet loss.

   # tuned-adm profile network-latency
   # tuned-adm active
   Current active profile: network-latency

3. Verify the system C-states.

   On the idle stats tab, CPUs should report being in the following states most of the time: C0 , C1, or C1E. The CPU should never be in the C6 state. This improves throughput and latency, but may increase power consumption and heat generation.

4. After verifying that TuneD is running with the correct profile, install the powertop utility to verify that TuneD is running correctly.

   # dnf install -y powertop
   # powertop

Operating system tuning with sysctl

1. Create the /etc/sysctl.d/90-ibm-storage-ceph.conf file with the following contents:

   # Memory tuning
   vm.min_free_kbytes = 4194304
   vm.vfs_cache_pressure = 10
   vm.swappiness = 1
   vm.zone_reclaim_mode = 0
   vm.dirty_ratio = 80
   vm.dirty_background_ratio = 3
   
   # I/O performance
   fs.aio-max-nr = 50000000
   
   # Network performance
   net.ipv4.tcp_timestamps = 0
   net.ipv4.tcp_sack = 1
   net.ipv4.tcp_dsack = 0
   net.ipv4.tcp_fack = 0
   net.ipv4.tcp_congestion_control = bbr
   net.core.default_qdisc = fq
   net.ipv4.tcp_mtu_probing = 1
   net.ipv4.tcp_syncookies = 1
   net.core.somaxconn = 5000
   net.ipv4.conf.all.send_redirects = 0
   net.ipv4.conf.all.accept_source_route = 0
   net.ipv4.tcp_fin_timeout = 20
   net.ipv4.tcp_tw_reuse = 1
   net.ipv4.tcp_window_scaling = 1
   net.ipv4.tcp_low_latency = 1
   net.ipv4.tcp_adv_win_scale = 1
   net.ipv4.tcp_slow_start_after_idle = 0
   
   # High-impact memory and backlog tuning
   net.core.netdev_max_backlog = 250000
   net.ipv4.tcp_max_tw_buckets = 2000000
   net.ipv4.tcp_max_syn_backlog = 100000
   net.ipv4.udp_mem = 4096 87380 33554432
   net.ipv4.tcp_rmem = 4096 87380 33554432
   net.ipv4.tcp_wmem = 4096 65536 33554432
   net.core.rmem_max = 67108864
   net.core.wmem_max = 67108864
   
   # Connection tracking
   net.netfilter.nf_conntrack_max = 1048576
   net.nf_conntrack_max = 1048576

2. Complete the nf_conntrack table expansion by running the following commands:

   echo 131072 > /sys/module/nf_conntrack/parameters/hashsize
   echo "options nf_conntrack hashsize=131072" >/etc/modprobe.d/nf_conntrack.conf

   Ensure that your fleet automation persists these files by using Ansible, or a similar fleet configuration management tool.

Volatile HDD cache

Modern hard drives include up to 256 MB of cache. This cache is volatile and lacks Power Loss Protection (PLP). If power is lost, unwritten data may be corrupted. When volatile write cache is enabled, the Linux kernel flushes every write synchronously to media. This prevents data loss but often reduces OSD performance, especially average and tail write latencies.

You can improve performance by disabling the volatile cache. This cache is different from controller-level caches, such as those on RAID HBAs. The impact varies by drive and controller. Test changes in a lab or pre-GA environment before production deployment. Monitor hdparm and sdparm latency before and after changes by using your cluster dashboard.

Figure 1 an example OSD latency output on the dashboard.

Note: Persistence across reboot depends on drive firmware.

To disable volatile write cache reliably at boot, create a udev rule.

cat <<EOF >/etc/udev/rules.d/90-wce.rules SUBSYSTEM=="block", ENV{SCSI_VENDOR}=="SPINNERSRUS", ACTION=="add|change", RUN+="/usr/bin/sdparm --clear=WCE /dev/%k" EOF

udevadm control --reload