Setting readahead on a drive or volume for local storage

Cassandra is the main data store for Monitoring. Best practice for optimizing disk performance for the Cassandra database is to reduce the default disk readahead for the drive or partition where your Cassandra data is stored. By default, the Linux kernel reads additional file data so that subsequent reads can be satisfied from the cache. The file access patterns of Cassandra queries result in the readaheads mostly being unused, therefore polluting the cache, driving up I/O time and also results in excessive disk I/O levels.

Before you begin

You can view your current readahead settings with either of these commands:

lsblk --output NAME,KNAME,TYPE,MAJ:MIN,FSTYPE,SIZE,RA,MOUNTPOINT,LABEL

blockdev --report

Examples:

lsblk --output NAME,KNAME,TYPE,MAJ:MIN,FSTYPE,SIZE,RA,MOUNTPOINT,LABEL
NAME          KNAME TYPE MAJ:MIN FSTYPE       SIZE   RA MOUNTPOINT LABEL
fd0           fd0   disk   2:0                  4K  128
sda           sda   disk   8:0                 80G 4096
├─sda1        sda1  part   8:1   xfs            1G 4096 /boot
└─sda2        sda2  part   8:2   LVM2_member   79G 4096
  ├─rhel-root dm-0  lvm  253:0   xfs           75G 4096 /
  └─rhel-swap dm-1  lvm  253:1   swap           4G 4096 [SWAP]
sdb           sdb   disk   8:16  xfs          100G 4096 /docker
sdc           sde   disk   8:32                 2T  128

blockdev --report
RO    RA   SSZ   BSZ   StartSec            Size   Device
rw   256   512  4096          0            4096   /dev/fd0
rw  8192   512  4096          0     85899345920   /dev/sda
rw  8192   512   512       2048      1073741824   /dev/sda1
rw  8192   512  4096    2099200     84824555520   /dev/sda2
rw  8192   512   512          0    107374182400   /dev/sdb
rw  8192   512   512          0     80530636800   /dev/dm-0
rw  8192   512  4096          0      4290772992   /dev/dm-1
rw   256   512  4096          0   2148557389824   /dev/sdc

Looking at the RA and Size columns, the readahead of 8192 combined with the size of 512 results in a readahead of 4096 KB. That means any read on the / root drive results in 4 MB of disk I/O into the system cache. Best practice is to use a separate drive for the Cassandra data, as well as the other StatefulSet services requiring disk space.

About this task

The following steps provide two examples of how to modify the readahead settings for an existing drive by using the command line or the tuned service to ensure better performance for Cassandra. You can manually set the readahead on an existing drive or modify the tuned.services disk settings to make the readahead settings persistent. These steps need to be performed on each VM running Cassandra.

Procedure

Manually set readahead on an existing drive or volume.

With this method, the tuned service adjusts the configuration settings to optimize system performance. Tuned profiles overwrite the smaller readahead setting used in the LVM setup. The tuned service adjusts the configuration settings to optimize system performance. The service can modify settings such as disk device readahead. Tuned profiles overwrite the smaller readahead setting used in the LVM setup. To prevent the overwrite, add the setting to your tuned profile. For more information, see Performance Tuning with tuned and tuned-adm in the Red Hat documentation.