Resharding the RocksDB database

Use the BlueStore administration tool to reshard the RocksDB database. Resharding splits the BlueStore block.db into several column families without redeploying the OSDs, which can be beneficial for write performance.

Before you begin

Before you begin, make sure that you have the following prerequisites in place:
  • A running IBM Storage Ceph cluster.
  • The object store configured as BlueStore.
  • OSD nodes deployed on the hosts.
  • Root level access to the all the hosts.
  • The ceph-common and cephadm packages installed on all the hosts.

About this task

Reshard the RocksDB database to split the BlueStore block.db into several columns. Column families have the same features as the whole database, but allows users to operate on smaller data sets and apply different options. It leverages the different expected lifetime of keys stored. The keys are moved during the transformation without creating new keys or deleting existing keys.

Reshard the OSD in one of the following ways:

For more information, see Initial installation.

Use the rocksdb-resharding.yml playbook

Reshard the database by using the rocksdb-resharding.yml playbook.

Procedure

  1. As a root user, on the administration node, navigate to the cephadm folder where the playbook is installed. 
    cd /usr/share/cephadm-ansible
  2. Run the playbook. 
    ansible-playbook -i hosts rocksdb-resharding.yml -e osd_id=OSD_ID -e admin_node=HOST_NAME
    For example, 
    [root@host01 ~]# ansible-playbook -i hosts rocksdb-resharding.yml -e osd_id=7 -e admin_node=host03

...............
TASK [stop the osd] ***********************************************************************************************************************************************************************************************
Wednesday 29 November 2023  11:25:18 +0000 (0:00:00.037)       0:00:03.864 **** 
changed: [localhost -> host03]
TASK [set_fact ceph_cmd] ******************************************************************************************************************************************************************************************
Wednesday 29 November 2023  11:25:32 +0000 (0:00:14.128)       0:00:17.992 **** 
ok: [localhost -> host03]

TASK [check fs consistency with fsck before resharding] ***********************************************************************************************************************************************************
Wednesday 29 November 2023  11:25:32 +0000 (0:00:00.041)       0:00:18.034 **** 
ok: [localhost -> host03]

TASK [show current sharding] **************************************************************************************************************************************************************************************
Wednesday 29 November 2023  11:25:43 +0000 (0:00:11.053)       0:00:29.088 **** 
ok: [localhost -> host03]

TASK [reshard] ****************************************************************************************************************************************************************************************************
Wednesday 29 November 2023  11:25:45 +0000 (0:00:01.446)       0:00:30.534 **** 
ok: [localhost -> host03]

TASK [check fs consistency with fsck after resharding] ************************************************************************************************************************************************************
Wednesday 29 November 2023  11:25:46 +0000 (0:00:01.479)       0:00:32.014 **** 
ok: [localhost -> host03]

TASK [restart the osd] ********************************************************************************************************************************************************************************************
Wednesday 29 November 2023  11:25:57 +0000 (0:00:10.699)       0:00:42.714 **** 
changed: [localhost -> host03]

What to do next

Verify that the sharding is complete.
  1. Stop the OSD that is resharded. 
    [ceph: root@host01 /]# ceph orch daemon stop osd.7
  2. Enter the OSD container. 
    [root@host03 ~]# cephadm shell --name osd.7
  3. Check for resharding. 
    [ceph: root@host03 /]# ceph-bluestore-tool --path /var/lib/ceph/osd/ceph-9/ show-sharding
    m(3) p(3,0-12) O(3,0-13) L P
  4. Start the OSD. 
    [ceph: root@host01 /]# ceph orch daemon start osd.7

Manually resharding the OSDs

Manually reshard the OSDs with the cephadm shell.

Procedure

  1. Log into the cephadm shell.
    For example, 
    [root@host01 ~]# cephadm shell
[ceph: root@host01 /]#
  2. Fetch the OSD_ID and the host details from the administration node. 
    ceph orch ps
  3. Log into the respective host as a root user and stop the OSD. 
    cephadm unit --name OSD_ID stop
    For example, 
    [root@host02 ~]# cephadm unit --name osd.0 stop
  4. Enter into the stopped OSD daemon container. 
    cephadm shell --name OSD_ID
    For example, 
    root@host02 ~]# cephadm shell --name osd.0
  5. Log into the cephadm shell and check the file system consistency. 
    ceph-bluestore-tool --path/var/lib/ceph/osd/ceph-OSD_ID/ fsck
    For example, 
    [ceph: root@host02 /]# ceph-bluestore-tool --path /var/lib/ceph/osd/ceph-0/ fsck

fsck success
  6. Check the sharding status of the node. 
    ceph-bluestore-tool --path /var/lib/ceph/osd/ceph-OSD_ID/ show-sharding
    For example, 
    [ceph: root@host02 /]# ceph-bluestore-tool --path /var/lib/ceph/osd/ceph-9/ show-sharding

m(3) p(3,0-12) O(3,0-13) L P
  7. Reshard by running the ceph-bluestore-tool command.
    Use the command, along with the following given parameters. 
    ceph-bluestore-tool --log-level 10 -l log.txt --path /var/lib/ceph/osd/ceph-OSD_ID/ --sharding="m(3) p(3,0-12) O(3,0-13)=block_cache={type=binned_lru} L P" reshard
    For example, 
    [ceph: root@host02 /]# ceph-bluestore-tool --path /var/lib/ceph/osd/ceph-9/ --sharding="m(3) p(3,0-12) O(3,0-13)=block_cache={type=binned_lru} L P" reshard

reshard success
  8. Use the show-sharding command to check the sharding status of the OSD node. 
    ceph-bluestore-tool --path /var/lib/ceph/osd/ceph-OSD_ID/ show-sharding
    For example, 
    [ceph: root@host02 /]# ceph-bluestore-tool --path /var/lib/ceph/osd/ceph-9/ show-sharding

m(3) p(3,0-12) O(3,0-13)=block_cache={type=binned_lru} L P
  9. Exit from the cephadm shell.
    For example, 
    [ceph: root@host02 /]# exit
[root@host02 ~]#
  10. Log into the respective host as a root user and start the OSD. 
    cephadm unit --name OSD_ID start
    For example, 
    [root@host02 ~]# cephadm unit --name osd.0 start