Local read-only cache

Many applications benefit greatly from large local caches. Not only is the data available with very low latency, but the cache hit serves to reduce the load on the shared network and on the backend storage itself, thus benefiting all nodes, even those without large caches.

Local solid-state disks (SSDs) provide an economical way to create very large caches. The SSD cache serves as an extension to the local buffer pool. As user data or metadata is evicted from the buffer pool in memory, it can be stored in the local cache. A subsequent access retrieves the data from the local cache, rather than from the home location. The data stored in the local cache, like data that is stored in memory, remains consistent. If a conflicting access occurs, the data is invalidated from all caches. In a like manner, if a node is restarted, all data stored in the cache is discarded.

In theory, any data or metadata can be stored in the local SSD cache, but the cache works best for small random reads where latency is a primary concern. Since the local cache typically offers less bandwidth than the backend storage, it might be unsuitable for large sequential reads. The configuration options provide controls over what is stored in the cache. The default settings are targeted at small random I/O.

The local read-only cache (LROC) function is disabled by default. To enable it, the administrator must define an NSD for an LROC device. The LROC device is expected to be a solid-state disk (SSD) accessible via SCSI. The device is defined as a standard NSD by mmcrnsd, but the DiskUsage is set to localCache. The NSD must have a primary server and is not allowed to have other servers. The primary server must be the node where the physical LROC device is installed. The device is not exported to other nodes in the cluster. The storage pool and failure group that is defined for the NSD are ignored and must be set to null. The mmcrnsd command writes a unique NSD volume ID onto the device. LROC devices are not tied to filesystems and therefore NSD limits are not affected by LROC devices.

The minimum size of a local read-only cache device is 4 GB. The maximum size of a local read-only cache device is 4 TB. The local read-only cache requires memory equal to 1% of the capacity of the LROC device.

Once the LROC device is defined, the daemon code at the primary server node is automatically told to do device discovery. The daemon detects that localCache is defined for its use and determines the mapping to the local device. The daemon then informs the LROC code to begin using the device for caching. Currently, there is a limit of four localCache devices per node. Note that the daemon code does not need to be restarted to begin using the cache.

The LROC device can be deleted by using the mmdelnsd command. Both mmcrnsd and mmdelnsd can be issued while the daemon is running with file systems mounted and online. The call to delete the NSD first informs the daemon that the device is being deleted, which removes it from the list of active LROC devices. Any data that is cached on the device is immediately lost, but data that is cached on other local LROC devices is unaffected. Once the mmdelnsd command completes, the underlying SSD can be physically removed from the node.

The NSD name for the LROC device cannot be used in any other GPFS commands, such as mmcrfs, mmadddisk, mmrpldisk, mmchdisk, or mmchnsd. The device is shown by mmlsnsd as a localCache.

Note: To avoid a security exposure, by default IBM Storage Scale does not allow file data from encrypted files, which is held in memory as cleartext, to be copied into an LROC device. However, you can set IBM Storage Scale to allow cleartext from encrypted files to be copied into an LROC device with the following command:

mmchconfig lrocEnableStoringClearText=yes

You might choose this option if you have configured your system to remove the security exposure.

Warning: If you allow cleartext from an encrypted file to be copied into an LROC device, you must take steps to protect the cleartext while it is in LROC storage.

For more information, see the following links:

Note: To reference data stored in LROC, IBM Storage Scale might require more buffer descriptors than the default number of buffer descriptors on a particular node. The amount of Buffer Descriptors that are desired can be controlled by the maxBufferDescs configuration value. This can be set cluster wide or node local. Given that local read-only cache is a node local component, you can run the following command:

mmchconfig maxBufferDescs=DesiredNumber -N Node

Where Node is the node with the local read-only cache device that is attached to configure the desired Buffer Descriptors on that particular node.

The amount of buffer descriptors that are required can vary depending on multiple variables. These variables include the capacity of the LROC devices that are configured and the type of workloads that are running on the node with the LROC devices. In IBM Storage Scale version 5.1.1 or later, warnings are written to the mmfs.log if the amount of buffer descriptors that are allocated is deemed too low based on the LROC device capacity. This warning would be displayed in System Health as a TIPS event if System Health is enabled.

Example message in mmfs.log:

2021-02-18_18:49:37.087-0700: [W] This node has LROC devices with total capacity of 50.0 GB.
Optimal LROC performance requires setting the maxBufferDescs config option.
Based on an assumed 4 MB data block size, the recommended value for maxBufferDescs is 45567 on this node.

For more information, see Monitoring local read-only cache.