Lenovo node issues

Unable to add node to OpenShift Container Platform

Problem statement

The node cannot be added to the OpenShift® Container Platform cluster.

Cause

The issue might be caused by a hung power operation (powerop) for the node. This can be identified by checking the logs on the system.

Symptoms

• The power operation is not progressing.
• The live powerop CR logs for the node are blank or do not show any activity.
• The powerop CR is not getting updated.

Resolution

To resolve this issue, follow these steps:

1. Check the live logs on the system for the powerop operation.
2. If the logs are blank or not updating, delete the powerop CR by running the following commands in the ibm-spectrum-fusion-ns namespace:

   oc get cpr

   oc delete cpr <node-powerop-cr-name>

   Important: Wait for the command to delete the powerop CR. Avoid forceful deletion as it may cause further issues.
3. After deleting the powerop CR, verify if the node can be added to the OpenShift Container Platform cluster.
4. If the issue persists, contact IBM support .

Gen2 node upsize failed on Gen 1 rack

Resolution

1. Log in to the BMC console of the node.
2. Press the space bar immediately when the blue screen appears.
3. Use the arrow keys to select Always continue boot.
4. Press Enter to confirm the selection.

   The node reboots and resolves the issue.

Blue screen appears in G03 and G04 GPU nodes

Resolution

1. Log in to the BMC console of the node.
2. Press the space bar immediately when the blue screen appears.
3. Use the arrow keys to select Always continue boot.
4. Press Enter to confirm the selection.

   The node reboots and resolves the issue.

mmaddnode command fails during node upsize on storage cluster

Resolution

After the initial failure of the mmaddnode command during node addition, run the following command to remove the files that do not get cleaned up properly:

oc exec -it compute-1-ru13 -c config -- /bin/sh
sh-5.1# rm /var/mmfs/gen/mmsdrfs

NVMe drives missing on XCC but seen on CoreOS

Resolution

1. Run resetsp to restart the BMC.
2. If the step 1 does not resolve the issue, run OS reboot command to reboot the node.
3. If both step 1 and step 2 do not resolve the issue, then contact IBM support to replace the faulty drive or faulty NVMe backplane.

Issues in node restart

Problem statement

Sometimes, after you restart a node, it may not change its state to ready and the ovs-configuration.service may fail.

Resolution

As a resolution, restart the node. For more information about Scale behavior during node restarts, see Scale behavior during node restarts.

GPU operator error

Certain Linux kernels that have KASLR enabled might experience a known issue with High Memory Mode initialization. This issue can cause CUDA initialization to fail. As a workaround, disable the High Memory Mode. For more information, see Disable High Memory Mode.

Management switch high availability is lost

When the management switch high availability is lost due to the management switch at RU18 being down, the cluster continues to function, but autodiscovery of nodes is not supported.

Incorrect port speed display for up Ports on configured and discovered nodes

Sometimes, the port speed for all up ports on configured and discovered nodes may be displayed as 0 Gbps instead of the actual speed.

GPU node capacity value shows 0

Problem statement

It is observed that sometimes users might see GPU node capacity as 0 and also that the NFD discovery instance has the nfd-worker-xxx in a constant crashloopback.

Resolution

This issue was observed in OpenShift Container Platform release 4.10.21. Upgrade the OpenShift Container Platform to 4.10.61, reinstall the NFD operator with the NFD Discovery CR, and install the NVIDIA operator and create the cluster policy that detects the GPUs.

Compute nodes in a hung state

Resolution

Steps to restart when the compute nodes get into a hung state:

As a prerequisite, you must have administrator access to Red Hat® OpenShift and OC (Red Hat OpenShift CLI) command access to the cluster.

1. Log in to OpenShift UI.
2. Go to Compute > Bare Metal Hosts.
3. Ensure to select openshift-machine-api project on the Bare Metal Hosts page.
4. From the ellipsis menu, click Power Off for the node.
5. Power it on again and check whether the issue is resolved.

Pod migration gets stuck in the Terminating or ContainerCreating state

Problem statement

A controller node goes down and the migration of the pod gets stuck in Terminating or ContainerCreating state.

Resolution

As a workaround, delete the container creating pod so that it can get scheduled in another available controller node.

Compute node can abruptly become unreachable

Problem statement

A compute node can abruptly become unreachable either due to a power outage or network disruption.

Cause

The pods that run on that specific compute node might have not got evacuated cleanly and they remain stuck in Terminating or Unknown state.

Resolution

To ensure a complete evacuation and migration of the pod that is stuck on the failed node, force delete the pod manually by using the following command:

oc delete pod <pod name> --grace-period=0 --force --namespace <namespace>

Add rack workflow from base rack fails in node validation

Problem statement

Add rack workflow from base rack fails in node validation for the nodes of auxiliary rack in case of high availability cluster.

Cause

The issue might occur because monitoring failed for the nodes of the auxiliary rack.

Resolution

1. Before performing the add rack operation, ensure that all the nodes on the auxiliary rack are healthy and that there are no warnings, critical events or any component failures on any of the nodes.
2. Restart the BMC on the failing nodes to clear the events on the compute node.

OpenShift Container Platform node status shows NotReady but IBM Fusion HCI user interface node status shows as Running

Problem statement

The Running state of the node on the IBM Fusion HCI user interface indicates that the Baremetal node is healthy and running. Whenever you see IBM Fusion HCI node status is Running and OpenShift Container Platform node status is Not ready, then follow the steps in the resolution.

Resolution

1. Log in to OpenShift user interface.
2. Go to Compute > Bare Metal.
3. From the ellipsis menu, click Power Off for the node.
4. Power it on again and check if the issue is resolved.

Node goes missing from the IBM Fusion HCI rack graphical view

Problem statement

OpenShift Container Platform node status shows Ready even though the IBM Fusion HCI user interface node status shows the node state as Error connecting to node

For more information about node status, see Monitoring hardware from IBM Fusion HCI user interface.

This issue might occur due to the power or network issues.

Resolution

The node can be viewed graphically in the user interface of the IBM Fusion HCI rack after it gets into a healthy state. For more information, see Monitoring hardware from IBM Fusion HCI user interface.

1. Get the name of the compute-operator pod name:

   oc get po -n ibm-spectrum-fusion-ns | grep isf-compute-operator
   

2. Copy the OneCli folder from the operator pod:

   oc rsync -c manager isf-compute-operator-controller-manager-<pod name>:/fw/onecli

   Note: For the subsequent commands, create a separate session.
3. Start a debug pod on the target node (BMC has error connecting to the node) that is in Ready state.

   oc debug node/<nodename>

4. Get the debug pod name:

   oc get po -A -o wide | grep <nodename> | grep debug

5. Copy the local copy of the OneCli folder to debug pod.

   oc rsync onecli pod/<debug pod>:/tmp

6. From the other session, run the following command to restart the node:

   
   cd /tmp/onecli
   ./onecli misc rebootbmc

7. Go to the Infrastructure > Nodes and wait for the following error message to go away:

   Error connecting to node

   It takes around three minutes for the error to disappear from the page.

Node in NotReady state due to certificate errors in OpenShift pods