OpenShift Container Platform 4.19.9 Upgrade Issues

Problem:

Upgrades to OCP 4.19.9+ may take longer than expected, and in some cases, the post-upgrade ODF configuration does not complete successfully.

Cause:
During the upgrade, underlying Local Storage Operator (LSO) symlinks may unexpectedly change or disappear. This results in missing physical drive paths.

Symptom: 

Below pods stuck in Init:0/4 state in openshift-storage project: 

$ oc get pods -n openshift-storage | grep "rook-ceph-osd"
rook-ceph-osd-0-9cd56bd86-6hscc              0/2     Init:0/4   0          18h
rook-ceph-osd-2-57cf7598d9-cfzwm             0/2     Init:0/4   0          18h

Diagnosing The Problem:

Checking the events for the pod shows symlink resolution errors such as:

MapVolume.EvalHostSymlinks failed for volume "local-pv-d73fe259" : lstat /dev/disk/by-id/nvme-eui.354c4c30526012250025384500000001 no such file or directory

Resolution:

Note: This is an example snippet. Please modify the values according to your environment.

1. Login to Node compute-1-ru5 on which above osd pods is pending as seen in above screenshot

2. Check existing symlinks in /mnt/local-storage/ibm-spectrum-fusion-local/ 

   sh-5.1# ls -l /mnt/local-storage/ibm-spectrum-fusion-local/  
   total 0  
   lrwxrwxrwx. 1 root root 57 Oct 27 09:11 nvme-eui.354c4c30526001550025384500000001 -> /dev/disk/by-id/nvme-eui.354c4c30526001550025384500000001  
   lrwxrwxrwx. 1 root root 57 Oct 27 09:11 nvme-eui.354c4c30526012250025384500000001 -> /dev/disk/by-id/nvme-eui.354c4c30526012250025384500000001

3. Check disks present on the node.

   sh-5.1# lsblk -f | grep "ceph"  
   nvme0n1 ceph_bluestore  
   nvme1n1 ceph_bluestore

4. Verify that the corresponding device entries exist under /dev/disk/by-id/ for the disks present on node. 

   sh-5.1# ls -l /dev/disk/by-id/ | grep "nvme0n1"  
   lrwxrwxrwx. 1 root root 13 Jan 29 09:02 nvme-MZWLJ7T6HALA-000V5_S5LLNE0R600155 -> ../../nvme0n1 \
   lrwxrwxrwx. 1 root root 13 Jan 29 09:02 nvme-MZWLJ7T6HALA-000V5_S5LLNE0R600155_1 -> ../../nvme0n1 
   lrwxrwxrwx. 1 root root 13 Jan 29 09:02 nvme-nvme.144d-53354c4c4e453052363030313535-4d5a574c4a37543648414c412d3030305635-00000001 -> ../..
   

5. If the symlink names in /mnt/local-storage/ibm-spectrum-fusion-local/ do not match the current entries in /dev/disk/by-id/, the symlink is stale and must be corrected. Remove stale symlinks.

   sh-5.1# rm /mnt/local-storage/ibm-spectrum-fusion-local/nvme-eui.354c4c30526001550025384500000001

6. Recreate correct symlinks, Point each expected NVMe symlink to the correct device under /dev/disk/by-id/.

   sh-5.1# ln -s /dev/disk/by-id/nvme-MZWLJ7T6HALA-000V5_S5LLNE0R600155 /mnt/local-storage/ibm-spectrum-fusion-local/nvme-eui.354c4c30526001550025384500000001

Once symlinks are corrected, OCP should automatically re-initiate mounting of PVs and the OSD pods should move past the Init phase. 

This needs to be repeated for all affected disks and all OSD pods that are stuck in Init or Pending state across all impacted nodes.