Increasing the placement group

Learn how to increase the placement group.

Before you begin

Before you begin, make sure that you have the following prerequisites in place:

A running IBM Storage Ceph cluster in a healthy state.
Root-level access to the node.

About this task

Insufficient placement group (PG) count impacts the performance of the Ceph cluster and data distribution. It is one of the main causes of the nearfull osds error messages.

The recommended ratio is between 100 and 300 PGs per OSD. This ratio can decrease when you add more OSDs to the cluster.

The pg_num and pgp_num parameters determine the PG count. These parameters are configured per each pool, and therefore, you must adjust each pool with low PG count separately.

Important: Increasing the PG count is the most intensive process that you can run on a Ceph cluster. This process might have a serious performance impact if not done in a slow and methodical way. After you increase pgp_num, you will not be able to stop or reverse the process and you must complete it. Consider increasing the PG count outside of business-critical processing time allocation, and alert all clients about the potential performance impact. Do not change the PG count if the cluster is in the HEALTH_ERR state.

For more information, see Nearfull OSDs and Monitoring placement group sets.

Procedure

Reduce the impact of data redistribution and recovery on individual OSDs and OSD hosts.

Lower the value of the osd max backfills, osd_recovery_max_active, and osd_recovery_op_priority parameters.

[ceph: root@host01 /]# ceph tell osd.* injectargs '--osd_max_backfills 1 --osd_recovery_max_active 1 --osd_recovery_op_priority 1'

Disable the shallow and deep scrubbing.

ceph osd set noscrub

ceph osd set nodeep-scrub

For example,

[ceph: root@host01 /]# ceph osd set noscrub
[ceph: root@host01 /]# ceph osd set nodeep-scrub

Calculate the optimal value of the pg_num and pgp_num parameters.
Use the Ceph Placement Groups (PGs) per Pool Calculator to calculate the optimal values.
Increase the pg_num value in small increments until you reach the wanted value.
1. Determine the starting increment value.
  Use a very low value that is a power of two, and increase it when you determine the impact on the cluster. The optimal value depends on the pool size, OSD count, and client I/O load.
2. Increment the pg_num value.
  Specify the pool name and the new value.
```
ceph osd pool set POOL pg_num VALUE
```
  For example,
```
[ceph: root@host01 /]# ceph osd pool set data pg_num 4
```
3. Monitor the status of the cluster by using the ceph -s command.
  The PGs state changes from creating to active+clean. Wait until all PGs are in the active+clean state.
Increase the pgp_num value in small increments until you reach the wanted value.
1. Determine the starting increment value.
  Use a very low value that is a power of two, and increase it when you determine the impact on the cluster. The optimal value depends on the pool size, OSD count, and client I/O load.
2. Increment the pgp_num value.
  Specify the pool name and the new value.
```
ceph osd pool set POOL pgp_num VALUE
```
  For example,
```
[ceph: root@host01 /]# ceph osd pool set data pgp_num 4
```
3. Monitor the status of the cluster by using the ceph -s command.
  The PGs state changes through peering, wait_backfill, backfilling, recover, and others. Wait until all PGs are in the active+clean state.
Repeat steps 1 though 4 for all pools with insufficient PG count.

Set osd max backfills, osd_recovery_max_active, and osd_recovery_op_priority to their default values.

For example,

[ceph: root@host01 /]# ceph tell osd.* injectargs '--osd_max_backfills 1 --osd_recovery_max_active 3 --osd_recovery_op_priority 3'

Enable the shallow and deep scrubbing.

ceph osd unset noscrub

ceph osd unset nodeep-scrub

For example,

[ceph: root@host01 /]# ceph osd unset noscrub
[ceph: root@host01 /]# ceph osd unset nodeep-scrub