Mismatch of Heketi disk usage

Mismatch of actual disk space and the used disk space that is reported by Heketi.

When you create a persistent volume claim (PVC), you might see an error message that no disk space is available.

Resolving the issue

If you see a disk usage mismatch in the heketi topology information, run these commands to synchronize the disk usage.

1. Set up kubectl CLI. See Accessing your cluster from the kubectl CLI.

2. Get the Heketi deployment pod name.

   kubectl -n kube-system get pods | grep heketi
   

   Following is an example output of the command:

   storage-glusterfs-glusterfs-heketi-deployment-85844b495f-wbxhf   1/1       Running     0          18h
   

3. Log in to the Heketi pod and get the list of nodes.

   kubectl -n kube-system exec -it <Heketi-pod-name> -- heketi-cli --user admin --secret admin node list
   

   Following is an example output of the command:

   Id:10cc13f4cd18136fc9b6d2a6d1eac733     Cluster:34fe224a09d8022a3582da817c31a81b
   Id:157012bb7b0391b2c116b35de8d6e7ba     Cluster:34fe224a09d8022a3582da817c31a81b
   Id:95c72d452bc2c21b3a1e2fa87e63f902     Cluster:34fe224a09d8022a3582da817c31a81b
   

4. Get information about the devices on the nodes. Repeat this step for each node.

   kubectl -n kube-system exec -it <Heketi-pod-name> -- heketi-cli --user admin --secret admin node info <node-ID>
   

   Following is an example command and output:

   kubectl -n kube-system exec -it storage-glusterfs-glusterfs-heketi-deployment-85844b495f-wbxhf -- heketi-cli --user admin --secret admin node info 10cc13f4cd18136fc9b6d2a6d1eac733
   Node Id: 10cc13f4cd18136fc9b6d2a6d1eac733
   State: online
   Cluster Id: 34fe224a09d8022a3582da817c31a81b
   Zone: 1
   Management Hostname: 10.41.4.108
   Storage Hostname: 10.41.4.108
   Devices:
   Id:c7e93cd3d3d293a78dfb99cb2809f699   Name:/dev/disk/by-path/virtio-pci-0000_00_11_0 State:online    Size (GiB):699      Used (GiB):497       Free (GiB):202
              Bricks:
                        Id:33cb391e0113ec96ccc546a4e4288018   Size (GiB):100     Path: /var/lib/heketi/mounts/vg_d991f7315d9e13f9ef4d8044cd474569/brick_33cb391e0113ec96ccc546a4e4288018/brick
                        Id:41a508db035b042b1d35839f0d40f0c5   Size (GiB):20     Path: /var/lib/heketi/mounts/vg_d991f7315d9e13f9ef4d8044cd474569/brick_41a508db035b042b1d35839f0d40f0c5/brick
   

5. Get information about the disk usage on the devices. Repeat this step for each device on each node.

   kubectl -n kube-system exec -it <Heketi-pod-name> -- heketi-cli --user admin --secret admin device info <device-ID>
   

   Following is an example command and output:

   kubectl -n kube-system exec -it storage-glusterfs-glusterfs-heketi-deployment-85844b495f-wbxhf -- heketi-cli --user admin --secret admin device info c7e93cd3d3d293a78dfb99cb2809f699
   Device Id: c7e93cd3d3d293a78dfb99cb2809f699
   Name: /dev/disk/by-path/virtio-pci-0000_00_11_0
   State: online
   Size (GiB): 699
   Used (GiB): 497
   Free (GiB): 202
   Bricks:
   Id:33cb391e0113ec96ccc546a4e4288018   Size (GiB):100     Path: /var/lib/heketi/mounts/vg_d991f7315d9e13f9ef4d8044cd474569/brick_33cb391e0113ec96ccc546a4e4288018/brick
   Id:41a508db035b042b1d35839f0d40f0c5   Size (GiB):20     Path: /var/lib/heketi/mounts/vg_d991f7315d9e13f9ef4d8044cd474569/brick_41a508db035b042b1d35839f0d40f0c5/brick
   

6. Synchronize the device to reflect the actual disk usage. Repeat this step for each device on each node.

   kubectl -n kube-system exec -it <Heketi-pod-name> -- heketi-cli --user admin --secret admin device resync <device-ID>
   

   Following is an example command and output:

   kubectl -n kube-system exec -it storage-glusterfs-glusterfs-heketi-deployment-85844b495f-wbxhf -- heketi-cli --user admin --secret admin device resync c7e93cd3d3d293a78dfb99cb2809f699
   Device  updated
   

7. Verify that the disk usage information is synchronized.

   kubectl -n kube-system exec -it <Heketi-pod-name> -- heketi-cli --user admin --secret admin device info <device-ID>
   

   Following is an example command and output:

   kubectl -n kube-system exec -it storage-glusterfs-glusterfs-heketi-deployment-85844b495f-wbxhf -- heketi-cli --user admin --secret admin device info c7e93cd3d3d293a78dfb99cb2809f699
   Device Id: c7e93cd3d3d293a78dfb99cb2809f699
   Name: /dev/disk/by-path/virtio-pci-0000_00_11_0
   State: online
   Size (GiB): 699
   Used (GiB): 120
   Free (GiB): 579
   Bricks:
   Id:33cb391e0113ec96ccc546a4e4288018   Size (GiB):100     Path: /var/lib/heketi/mounts/vg_d991f7315d9e13f9ef4d8044cd474569/brick_33cb391e0113ec96ccc546a4e4288018/brick
   Id:41a508db035b042b1d35839f0d40f0c5   Size (GiB):20     Path: /var/lib/heketi/mounts/vg_d991f7315d9e13f9ef4d8044cd474569/brick_41a508db035b042b1d35839f0d40f0c5/brick