Understanding Ceph OSD status

Understand the different Ceph OSD states for the storage cluster.

A Ceph OSD’s status is either in the storage cluster, or out of the storage cluster. It is either up and running, or it is down and not running. If a Ceph OSD is in an up state, it can be either in the storage cluster, where data can be read and written, or it is out of the storage cluster. If it was in the storage cluster and recently moved out of the storage cluster, Ceph starts migrating placement groups to other Ceph OSDs. If a Ceph OSD is out of the storage cluster, CRUSH does not assign placement groups to the Ceph OSD. If a Ceph OSD is in a down state, it is also out.
Note: If a Ceph OSD is in the down and in state there is a problem and the storage cluster will not be in a healthy state.
Figure 1. OSD states
OSD states
If you run a command such as ceph health, ceph -s or ceph -w, the storage cluster does not always echo back HEALTH OK. Not displaying the HEALTH OK output is expected in the following circumstances:
  • You have not started the storage cluster yet, and it is not responding.
  • You have just started or restarted the storage cluster, and it is not ready yet, because the placement groups are getting created and the Ceph OSDs are in the process of peering.
  • You just added or removed a Ceph OSD.
  • You just modified the storage cluster map.

An important aspect of monitoring Ceph OSDs is to ensure that when the storage cluster is up and running that all Ceph OSDs that are in the storage cluster are up and running, too.

To see whether all OSDs are running, run one of the following commands: 
  • ceph osd stat
    For example, 
    [ceph: root@host01 /]# ceph osd stat
  • ceph osd dump
    For example, 
    [ceph: root@host01 /]# ceph osd dump
The command results shown as 
eNNNN: x osds: y up, z in
where,
  • eNNNN is the map epoch
  • x is the total number of OSDs
  • y is the number of OSDs that are in an up state
  • z is the number OSDs that are in an in state
If the number of Ceph OSDs that are in the storage cluster are more than the number of Ceph OSDs that are up. Run the ceph osd tree command to identify the ceph-osd daemons that are not running. 
[ceph: root@host01 /]# ceph osd tree

# id    weight  type name   up/down reweight
-1  3   pool default
-3  3       rack mainrack
-2  3           host osd-host
0   1               osd.0   up  1
1   1               osd.1   up  1
2   1               osd.2   up  1
Tip: The ability to search through a well-designed CRUSH hierarchy can help you troubleshoot the storage cluster by identifying the physical locations faster.

If a Ceph OSD is down, connect to the node and start it. Use either the IBM Storage Ceph Console or the command line to restart the Ceph OSD daemon.

From the command line: 
systemctl start CEPH_OSD_SERVICE_ID
For example, 
[root@host01 ~]# systemctl start ceph-499829b4-832f-11eb-8d6d-001a4a000635@osd.6.service