Control nodes creation failed

Message: Failed to create the Red Hat® OpenShift® Container Platform cluster control nodes.

Diagnostics

Note: Run all commands as a kni user from provisioner node (also known as RU7 or compute-1-ru7).
  1. Check the .openshift_install.log for errors.
  2. If the logs do not provide the necessary information, then continue next steps in this procedure for further diagnostics.
  3. As a kni user, run oc get nodes on the provisioner (RU7 or compute-1-ru7 node).
  4. Check whether the following three ready nodes are not listed in the output:
    NAME                                        STATUS   ROLES                         AGE   VERSION
control-1-ru2.rackm03.rtp.raleigh.ibm.com   Ready    control-plane,master,worker   55m   v1.27.3+4aaeaec
control-1-ru3.rackm03.rtp.raleigh.ibm.com   Ready    control-plane,master,worker   56m   v1.27.3+4aaeaec
control-1-ru4.rackm03.rtp.raleigh.ibm.com   Ready    control-plane,master,worker   56m   v1.27.3+4aaeaec
  5. Identify the node that is not booted. If you do not see these three nodes, then note down the missing node(s) from the output.
  6. Identify the IMM of the missing node. Here is mapping of nodes to IMM:
    control-1-ru2 ==> imm_ru2
control-1-ru3 ==> imm_ru3
control-1-ru4 ==> imm_ru4
  7. For node(s) that did not show up in the step 4 output, then run the corresponding IMM command to connect to its IMM. For example, if node control-1-ru3 did not show up, then from RU7/provisioner (compute-1-ru7), run the imm_ru3 command as a kni user.
  8. Run resetsp command on the IMM prompt.
  9. Wait for 10 minutes and retry the installation.
  10. If issue persists, ping the control nodes by using the IP and hostname that are reserved for the node in DHCP.
  11. If ping is not working, then do the following steps to access the IMM user interface of the Baremetal node:
    1. Run the following command to find the IPv6 address of the node to debug:
      oc get bmh -A -o wide |grep control
      Output:
      openshift-machine-api   control-1-ru2   OK       externally provisioned   isf-rackae4-2bztv-master-0   ipmi://[fd8c:215d:178e:c0de:a94:efff:fefe:2f95]:623   unknown            true             3h28m
openshift-machine-api   control-1-ru3   OK       externally provisioned   isf-rackae4-2bztv-master-1   ipmi://[fd8c:215d:178e:c0de:a94:efff:fefd:cecd]:623   unknown            true             3h28m
openshift-machine-api   control-1-ru4   OK       externally provisioned   isf-rackae4-2bztv-master-2   ipmi://[fd8c:215d:178e:c0de:a94:efff:fefe:3031]:623   unknown            true             3h28m
    2. Note down the IPv6 address of the node you are checking.
    3. Go to /home/kni/isfconfig/ and open the kickstart-.json
    4. Find the password for the IMM user (USERID) of the node and search for the string OCPRole` with value equal to hostname (for example control-1-ru3) of node. Sample section:
      ```
  "ipv6ULA": "XXXXXXXXXXXXXXXXXXXXXX",
       "ipv6LLA": "XXXXXXXXXXXXX",
       "serialNum": "J1025PXX",
       "mtm": "7D2XCTO1WW",
       "ibmSerialNumber": "rackae402",
       "ibmMTM": "9155-C01",
       "type": "storage",
       "OCPRole": "control-1-ru3",
       "location": "RU2",
       "name": "IMM_RU2",
       "bootDevice": "/dev/sda",
       "users": [
          {
             "user": "CEUSER",
             "password": "XXXXX",
             "group": "Administrator",
             "number": 2
          },
          {
             "user": "ISFUSER",
             "password": "XXXXXX",
             "group": "Administrator",
             "number": 3
          },
          {
             "user": "USERID",
             "password": "XXXXXX",
             "group": "Administrator",
             "number": 1
          }

  ```
    5. Run the following command to create a tunnel to the IMM of the node:
      ```
ssh -N -f -L :1443:[IPv6 address of IMM]:443 -L :3900:[IPv6 address of IMM]:3900 root@localhost
```
    6. To access the IMM user interface using the provisioner node IP address, open your browser and access the following IP.
       ```
 https://<provisioner ip>:1443
 ```
 - user - Enter `USERID`.
 - password - Use the value obtained in the previous step.
    7. From the IMM user interface page, open the remote console and check whether the node is up and shows a valid hostname prompt.
      • If the console shows localhost, then review your DHCP/DNS settings to ensure whether correct reservation is made for the node.
      • If the node shows a Red Hat linux login prompt instead of Core OS, then it infers that the node Base management controller (BMC) is not responsive.
      Do the following steps to resolve the issue:
      1. Identify IMM of the missing node. Here is mapping of nodes to IMM:
        ```
control-1-ru2 ==> imm_ru2
control-1-ru3 ==> imm_ru3
control-1-ru4 ==> imm_ru4
```
      2. For node(s) that did not show up in the previous steps, run the following sample IMM command to connect to their IMMs: For example, if node control-1-ru3 did not show up, then from RU7/provisioner (compute-1-ru7), run the following command as a kni user: imm_ru3
      3. Wait for the successful connection to IMM.
      4. In the system prompt, run the resetsp on IMM prompt.
      5. Wait for 10 minutes.

Next actions

In the installation user interface, click the Retry to restart the installation.