GlusterFS node status shows as peer rejected

GlusterFS node status might show as "Peer Rejected".

Note: If multiple nodes are in a "Peer Rejected" state, you might not be able to recover the GlusterFS nodes. You might need to set up the GlusterFS cluster again.

Resolving the problem

1. Ensure that kubectl CLI is set up. See Accessing your cluster from the Kubernetes CLI (kubectl).

2. Verify the status of GlusterFS pods.

   kubectl -n kube-system get pods -o wide | grep gluster
   

   The output resembles the following code:

   glusterfs-l95bp        1/1       Running     3          6d        192.168.0.184   worker-2
   glusterfs-n85vr        1/1       Running     0          4d        192.168.0.56    worker-1
   glusterfs-p66jq        1/1       Running     3          6d        192.168.0.96    worker-3
   

3. Check the peer status from any GlusterFS pod.

   kubectl -n kube-system exec <pod name> -- gluster peer status
   

   Following is an example command and output:

   kubectl -n kube-system exec glusterfs-l95bp -- gluster peer status
   Number of Peers: 2
   
   Hostname: 192.168.0.96
   Uuid: 3e518a43-b59f-45e7-a62c-5b213e0fece8
   State: Peer in Cluster (Connected)
   Other names:
   192.168.0.96
   
   Hostname: worker-1
   Uuid: c2a81937-e94f-4a22-86e7-bc9c8929c2d3
   State: Peer Rejected (Connected)
   

   If you see a "Peer Rejected" status, it implies that the volume configuration on that peer is out of sync with the rest of the cluster.

To synchronize with the GlusterFS cluster, complete these steps on the GlusterFS node that is in "Peer Rejected" state:

1. Access the shell inside the GlusterFS pod:

   kubectl -n kube-system exec -it <pod name> bash
   

2. Change to the /var/lib/glusterd directory.

   cd /var/lib/glusterd
   

   The folder might contain the following files and folders:

   .  ..  bitd  geo-replication  glusterd.info  glusterfind  glustershd  groups  hooks  nfs  options  peers  quotad  scrub  snaps  ss_brick  vols
   

3. Delete everything except glusterd.info, which is the universally unique identifier (UUID) file.

4. Restart the Gluster daemon.

   service glusterd restart
   

5. Probe a peer that is not in a "Peer Rejected" state.

   gluster peer probe <node name>
   

6. Check the peer status.

   gluster peer status
   

Note: You might need to repeat the steps until you no longer see the "Peer Rejected" status.