RocksDB cache behavior and metrics

Understand how RocksDB caching works in BlueStore OSDs, how to interpret cache performance counters, and how to identify and tune cache inefficiencies.

RocksDB is used by Ceph components, such as monitors and object storage daemons (OSDs). This topic describes RocksDB caching behavior for BlueStore OSDs.

RocksDB uses a block cache to store portions of SST files in memory. In BlueStore, IBM Storage Ceph uses an integrated cache implementation that combines the following elements:

RocksDB block cache
BlueStore metadata cache
BlueStore data cache

These caches share available memory and dynamically compete for resources. This design allows memory to be allocated where it is most beneficial for performance.

Cache sharding

To improve concurrency and reduce lock contention, the RocksDB cache is divided into multiple shards. Each cached object is assigned to a shard based on a hash of its key, enabling concurrent access across threads.

The number of shards is controlled by the rocksdb_cache_shard_bits configuration option. Set rocksdb_cache_shard_bits to select how many of bits from the key hash used to determine the cache shard. The shards input is an unsigned integer and the default is 4.

For example, a value of 4 results in 16 cache shards.

RocksDB cache performance counters

BlueStore exposes RocksDB cache activity through performance counters.

By default, two cache instances are created:

rocksdb-cache-O: metadata cache for onode structures
rocksdb-cache-default: cache for all other data

These counters can be used to evaluate cache efficiency and effectiveness.

Use the following command to view performance counters:

ceph tell osd.ID perf dump rocksdb-cache-O

The following is an example output:

"rocksdb-cache-O": {
  "capacity": 134217728,
  "usage": 134182832,
  "pinned": 0,
  "elems": 24502,
  "inserts": 25806978,
  "lookups": 150436987,
  "hits": 124629911,
  "misses": 25807076
  "read_bytes": 35265735
}

The following metrics are provided:

capacity: Total cache size
usage: Currently used memory
pinned: Number of temporarily pinned entries
elems: Number of cached elements
inserts: Number of items added to the cache
lookups: Number of cache lookup requests
hits: Number of successful lookups
misses: Number of failed lookups
read_bytes: Size of read operations being used, in bytes.

Viewing detailed cache statistics

Performance counters provide aggregated values. For deeper analysis, you can inspect cache statistics at the shard level.

Use per-shard statistics to help you identify imbalance or inefficient cache usage.

Use the following command to display shard statistics:
```
ceph tell osd.ID rocksdb show cache O
```
Use the following command to reset cache counters:
```
ceph tell osd.ID rocksdb reset cache O
```

Identifying cache inefficiencies

Imbalanced shard utilization can negatively affect performance.

Indicators of inefficiency include:

Very low element count in a shard compared to others.
High number of cache misses relative to hits.
Rapid increase in bluefs.read_bytes.
Note: To monitor read_bytes counters, use the perf dump commands. For example,
```
ceph tell osd.0 perf dump bluefs | jq -r '.bluefs.read_bytes'
```

For example,

# ceph tell osd.ID rocksdb show cache O
shard  capacity     usage   pinned   elems  inserts  lookups     hits   misses
  ...
    2  13631488  11060608        0    2232   135076   908468   773313   135155
    3  13631488   8166896        0       1   134006   427070   293147   133923
    4  13631488  11117984        0    2297   133367   700242   567318   132924
  ...

This pattern can indicate frequent eviction or ineffective cache usage within a shard.

Cache behavior during compaction

Large OSD deployments can store millions of objects, which can result in large RocksDB index blocks.

During compaction:

Multiple index blocks can be required at the same time.
Index blocks can be large compared to shard size.
If multiple large blocks are assigned to the same shard, repeated eviction can occur.

This behavior can reduce cache efficiency and negatively impact performance.

Tuning cache shard configuration

The default configuration of 16 shards (rocksdb_cache_shard_bits=4) is suitable for most environments.

In some cases, reducing the number of shards can improve cache behavior by increasing shard size. When larger size shards are required, decrease the value of rocksdb_cache_shard_bits.

Important: Reducing the number of shards can increase lock contention and might slightly affect baseline performance. Evaluate performance in a test environment before applying changes in production.

For more information about restructuring the RocksDB database to improve performance, see Resharding the RocksDB database.