IBM Storage Ceph: How to handle Inconsistent Placement Groups

Troubleshooting

Problem

ceph status or ceph -s reports inconsistent placement groups (PGs)

Symptom

Run a ceph status and/or ceph health detail and look at the pg states reporting as inconsistent:

$ ceph health detail
[...]
pg 11.eeef is active+clean+inconsistent, acting [106,427,854]
pg 5.ee92 is active+clean+inconsistent, acting [247,183,125]
[...]

Cause

There are multiple causes of inconsistent PGs For a detailed analysis of what caused this to happen, please open a support case with the IBM Storage Ceph support team.

That said, with supported releases, most inconsistent PGs are caused by persistent hardware failures.

Environment

IBM Storage Ceph

Diagnosing The Problem

For each affected Placement Group (PG) note the acting set reported by ceph health detail. We will use 11.eeef above as our example.

This PG is placed on OSDs with the IDs 106, 427, and 854. Ascertain the hosts on which each of these reside:

# for i in 106 42 85 ; do ceph osd find $i ; done
{
    "osd": 106,
...
    "host": "barney",
    "crush_location": {
        "host": "l18-28",
        "root": "default"
    }
}
{
    "osd": 427,
...
    "host": "fred",
    "crush_location": {
        "host": "fred",
        "root": "default"
    }
}
{
    "osd": 106,
...
    "host": "wilma",
    "crush_location": {
        "host": "wilma",
        "root": "default"
    }
}

Next, ascertain the storage drive(s) associated with each of these OSDs:

# ceph osd metadata 106 | jq .devices
"sdad,sdy"

Now from a shell on each OSD's host, search the output of dmesg -T and /var/log/messages* for each device

# zgrep sdad /var/log/messages*
#

Note the timestamps of any matches to ensure that a subsequent reboot has not re-enumerated the devices. If you find clear hardware errors, consider taking the affected OSD down and replacing the storage drive. You might also find it illuminating to examine the output of smartctl -a and smartctl -x for the affected device. HDDs with multiple grown defects, for example, are candidates for replacement.

Resolving The Problem

Info: Ceph offers the ability to repair inconsistent PGs with the ceph pg repair command. This should not be attempted before looking for a storage drive failure, as it may simply mask the root cause instead of fixing it.

Steps to repair inconsistent PGs:

Watch the ceph log for the result of the scrub.
```
# ceph -w | grep <pg.id>
```
In another terminal session trigger a deep-scrub on the placement group.
```
# ceph pg deep-scrub <pg.id>
```
Try repairing it:
```
# ceph pg repair <pg.id>
```

Sample session:

In the first terminal session, run ceph -w:

# ceph -w | grep 11.eeef

In another terminal, run the deep-scrub:

# ceph pg deep-scrub 11.eeef
instructing pg 11.eeef on osd.106 to deep-scrub

In the terminal session where you run ceph -w you should see error messages similar to:

2015-02-26 01:35:36.778215 osd.106 [ERR] 11.eeef deep-scrub stat mismatch, got 636/635 objects, 0/0 clones, 0/0 dirty, 0/0 omap, 0/0 hit_set_archive, 0/0 whiteouts, 1855455/1854371 bytes.
2015-02-26 01:35:36.788334 osd.106 [ERR] 11.eeef deep-scrub 1 errors

Repair the inconsistent PGs as follows:

# ceph pg repair 11.eeef
instructing pg 11.eeef on osd.106 to repair

Watch the terminal session with ceph -w to see something similar to:

2015-02-26 01:49:28.164677 osd.106 [ERR] 11.eeef repair stat mismatch, got 636/635 objects, 0/0 clones, 0/0 dirty, 0/0 omap, 0/0 hit_set_archive, 0/0 whiteouts, 1855455/1854371 bytes.
2015-02-26 01:49:28.164957 osd.106 [ERR] 11.eeef repair 1 errors, 1 fixed

There are multiple causes of inconsistent PGs. For a more detailed analysis of the underlying factors, please open a support case with the IBM Storage Ceph team. If there is any doubt as to the status of individual storage drives, hosts, or HBAs, please contact support *before* taking actions that may hinder restoration, especially removing or zapping a storage drive.

Document Location

Worldwide

[{"Type":"MASTER","Line of Business":{"code":"LOB69","label":"Storage TPS"},"Business Unit":{"code":"BU048","label":"IBM Software"},"Product":{"code":"SSEG27","label":"IBM Storage Ceph"},"ARM Category":[{"code":"a8m3p000000UoIUAA0","label":"Documentation"}],"ARM Case Number":"","Platform":[{"code":"PF025","label":"Platform Independent"}],"Version":"All Versions"}]

Tips