Local read-only cache

Many applications benefit greatly from large local caches. Not only is the data available with very low latency, but the cache hit serves to reduce the load on the shared network and on the backend storage itself, thus benefiting all nodes, even those without large caches.

Local solid-state disks (SSDs) provide an economical way to create very large caches. The SSD cache serves as an extension to the local buffer pool. As user data or metadata is evicted from the buffer pool in memory, it can be stored in the local cache. A subsequent access will retrieve the data from the local cache, rather than from the home location. The data stored in the local cache, like data stored in memory, remains consistent. If a conflicting access occurs, the data is invalidated from all caches. In a like manner, if a node is restarted, all data stored in the cache is discarded.

In theory, any data or metadata can be stored in the local SSD cache, but the cache works best for small random reads where latency is a primary concern. Since the local cache typically offers less bandwidth than the backend storage, it might be unsuitable for large sequential reads. The configuration options provide controls over what is stored in the cache. The default settings are targeted at small random I/O.

The local read-only cache (LROC) function is disabled by default. To enable it, the administrator must define an NSD for an LROC device. The LROC device is expected to be a solid-state disk (SSD) accessible via SCSI. The device is defined as a standard NSD by mmcrnsd, but the DiskUsage is set to localCache. The NSD must have a primary server and is not allowed to have other servers. The primary server must be the node where the physical LROC device is installed. The device is not exported to other nodes in the cluster. The storage pool and failure group defined for the NSD are ignored and should be set to null. The mmcrnsd command writes a unique NSD volume ID onto the device.

The minimum size of a local read-only cache device is 4 GB. The local read-only cache requires memory equal to 1% of the capacity of the LROC device.

Once the LROC device is defined, the daemon code at the primary server node is automatically told to do device discovery. The daemon detects that localCache is defined for its use and determines the mapping to the local device. The daemon then informs the local read-only cache code to begin using the device for caching. Currently, there is a limit of four localCache devices per node. Note that the daemon code does not need to be restarted to begin using the cache.

The LROC device can be deleted by using the mmdelnsd command. Both mmcrnsd and mmdelnsd can be issued while the daemon is running with file systems mounted and online. The call to delete the NSD first informs the daemon that the device is being deleted, which removes it from the list of active LROC devices. Any data cached on the device is immediately lost, but data cached on other local LROC devices is unaffected. Once the mmdelnsd command completes, the underlying SSD can be physically removed from the node.

The NSD name for the LROC device cannot be used in any other GPFS™ commands, such as mmcrfs, mmadddisk, mmrpldisk, mmchdisk or mmchnsd. The device is shown by mmlsnsd as a localCache.

Note: To avoid a security exposure, by default IBM Spectrum Scale™ does not allow file data from encrypted files, which is held in memory as cleartext, to be copied into an LROC device. However, you can set IBM Spectrum Scale to allow cleartext from encrypted files to be copied into an LROC device with the following command:

mmchconfig lrocEnableStoringClearText=yes

You might choose this option if you have configured your system to remove the security exposure.

Warning: If you allow cleartext from an encrypted file to be copied into an LROC device, you must take steps to protect the cleartext while it is in LROC storage.

For more information, see the following links: