Migrating BlueFS data

You can migrate the BlueStore file system (BlueFS) data, that is the RocksDB data, from the source volume to the target volume using the migrate LVM subcommand.

When using the migrate LVM subcommand, the source volume, except the main one, is removed on success.

LVM volumes are primarily for the target only.

The new volumes are attached to the OSD, replacing one of the source drives.

Following are the placement rules for the LVM volumes:

  • If source list has DB or WAL volume, then the target device replaces it.

  • If source list has slow volume only, then explicit allocation using the new-db or new-wal command is needed.

The new-db and new-wal commands attaches the given logical volume to the given OSD as a DB or a WAL volume respectively.

Prerequisites

  • A running IBM Storage Ceph cluster.

  • Root-level access to the Ceph OSD node.

  • Ceph OSDs prepared by the ceph-volume utility.

  • Volume groups and Logical volumes are created.

Procedure

  1. Log in the cephadm shell:

    Example 

    [root@host01 ~]# cephadm shell

  2. Stop the OSD to which you have to add the DB or the WAL device:

    Example 

    [ceph: root@host01 /]# ceph orch daemon stop osd.1

  3. Mount the new devices to the container:

    Example 

    [root@host01 ~]# cephadm shell --mount /var/lib/ceph/72436d46-ca06-11ec-9809-ac1f6b5635ee/osd.1:/var/lib/ceph/osd/ceph-1

  4. Attach the given logical volume to OSD as a DB/WAL device:

    Note: This command fails if the OSD has an attached DB.

    Syntax 

    ceph-volume lvm new-db --osd-id OSD_ID --osd-fsid OSD_FSID --target VOLUME_GROUP_NAME/LOGICAL_VOLUME_NAME

    Example 

    [ceph: root@host01 /]# ceph-volume lvm new-db --osd-id 1 --osd-fsid 7ce687d9-07e7-4f8f-a34e-d1b0efb89921 --target vgname/new_db
[ceph: root@host01 /]# ceph-volume lvm new-wal --osd-id 1 --osd-fsid 7ce687d9-07e7-4f8f-a34e-d1b0efb89921 --target vgname/new_wal

  5. You can migrate BlueFS data in the following ways:

    • Move BlueFS data from main device to LV that is already attached as DB:

      Syntax 

      ceph-volume lvm migrate --osd-id OSD_ID --osd-fsid OSD_UUID --from data --target VOLUME_GROUP_NAME/LOGICAL_VOLUME_NAME

      Example 

      [ceph: root@host01 /]# ceph-volume lvm migrate --osd-id 1 --osd-fsid 0263644D-0BF1-4D6D-BC34-28BD98AE3BC8 --from data --target vgname/db

    • Move BlueFS data from shared main device to LV which shall be attached as a new DB:

      Syntax 

      ceph-volume lvm migrate --osd-id OSD_ID --osd-fsid OSD_UUID --from data --target VOLUME_GROUP_NAME/LOGICAL_VOLUME_NAME

      Example 

      [ceph: root@host01 /]# ceph-volume lvm migrate --osd-id 1 --osd-fsid 0263644D-0BF1-4D6D-BC34-28BD98AE3BC8 --from data --target vgname/new_db

    • Move BlueFS data from DB device to new LV, and replace the DB device:

      Syntax 

      ceph-volume lvm migrate --osd-id OSD_ID --osd-fsid OSD_UUID --from db --target VOLUME_GROUP_NAME/LOGICAL_VOLUME_NAME

      Example 

      [ceph: root@host01 /]# ceph-volume lvm migrate --osd-id 1 --osd-fsid 0263644D-0BF1-4D6D-BC34-28BD98AE3BC8 --from db --target vgname/new_db

    • Move BlueFS data from main and DB devices to new LV, and replace the DB device:

      Syntax 

      ceph-volume lvm migrate --osd-id OSD_ID --osd-fsid OSD_UUID --from data db --target VOLUME_GROUP_NAME/LOGICAL_VOLUME_NAME

      Example 

      [ceph: root@host01 /]# ceph-volume lvm migrate --osd-id 1 --osd-fsid 0263644D-0BF1-4D6D-BC34-28BD98AE3BC8 --from data db --target vgname/new_db

    • Move BlueFS data from main, DB, and WAL devices to new LV, remove the WAL device, and replace the DB device:

      Syntax 

      ceph-volume lvm migrate --osd-id OSD_ID --osd-fsid OSD_UUID --from data db wal --target VOLUME_GROUP_NAME/LOGICAL_VOLUME_NAME

      Example 

      [ceph: root@host01 /]# ceph-volume lvm migrate --osd-id 1 --osd-fsid 0263644D-0BF1-4D6D-BC34-28BD98AE3BC8 --from data db --target vgname/new_db

    • Move BlueFS data from main, DB, and WAL devices to the main device, remove the WAL and DB devices:

      Syntax 

      ceph-volume lvm migrate --osd-id OSD_ID --osd-fsid OSD_UUID --from db wal --target VOLUME_GROUP_NAME/LOGICAL_VOLUME_NAME

      Example 

      [ceph: root@host01 /]# ceph-volume lvm migrate --osd-id 1 --osd-fsid 0263644D-0BF1-4D6D-BC34-28BD98AE3BC8 --from db wal --target vgname/data