Configuring D3N

You can configure the D3N data cache on an existing Ceph Object Gateway to improve the performance of big-data jobs running in IBM Storage Ceph clusters.

Before you begin

Before you begin, make sure that you have the following prerequisites in place:

A running IBM Storage Ceph cluster.
Ceph Object Gateway installed.
Root-level access to the admin node.
A fast NVMe drive in each RGW node to serve as the local cache storage.

About this task

For more information about using high-availability, see Ceph subsystems default logging level values and Understanding Ceph logs .

To enable D3N on an existing Ceph Object Gateway, use the following command to set the D3N-related configuration:

ceph config set CLIENT.RGW CONFIGURATION_OPTION VALUE

rgw_d3n_l1_local_datacache_enabled=true

ceph config set CLIENT.RGW rgw_d3n_l1_local_datacache_enabled TRUE/FALSE

For example,

[ceph: root@host01 /]# ceph config set client.rgw.host01 rgw_d3n_l1_local_datacache_enabled true

rgw_d3n_l1_datacache_persistent_path=PATH_TO_CACHE_DIRECTORY

ceph config set CLIENT.RGW rgw_d3n_l1_datacache_persistent_path PATH_TO_CACHE_DIRECTORY

For example,

[ceph: root@host01 /]# ceph config set client.rgw.host01 rgw_d3n_l1_datacache_persistent_path /mnt/nvme/rgw_datacache/

rgw_d3n_l1_datacache_size=max_size_of_cache_in_bytes

ceph config set CLIENT.RGW rgw_d3n_l1_datacache_size VALUE

For example,

[ceph: root@host01 /]# ceph config set client.rgw.host01 rgw_d3n_l1_datacache_size 10737418240

Example

Create a test object.

Note: The test object needs to be larger than 4 MB to cache.

[ceph: root@host01 /]# fallocate -l 1G ./1G.dat
[ceph: root@host01 /]# s3cmd mb s3://bkt
[ceph: root@host01 /]# s3cmd put ./1G.dat s3://bkt

Run GET of an object.

[ceph: root@host01 /]# s3cmd get s3://bkt/1G.dat /dev/shm/1G_get.dat
download: 's3://bkt/1G.dat' -> './1G_get.dat'  [1 of 1]
1073741824 of 1073741824   100% in   13s    73.94 MB/s  done

Verify cache creation. Cache is created with the name consisting of object key-name within a configured rgw_d3n_l1_datacache_persistent_path.
```
[ceph: root@host01 /]# ls -lh /mnt/nvme/rgw_datacache

rw-rr. 1 ceph ceph 1.0M Jun  2 06:18 cc7f967c-0021-43b2-9fdf-23858e868663.615391.1_shadow.ZCiCtMWeu_19wb100JIEZ-o4tv2IyA_1
```
After the cache is created for an object, the next GET operation for that object will access from cache resulting in faster access.
After the cache is created, you can see the cache acceleration by writing to the RAM drive, in this example /dev/shm.
```
[ceph: root@host01 /]# s3cmd get s3://bkt/1G.dat /dev/shm/1G_get.dat
download: 's3://bkt/1G.dat' -> './1G_get.dat'  [1 of 1]
1073741824 of 1073741824   100% in    6s   155.07 MB/s  done
```