Failed to install IBM Fusion operator

Failed to install the IBM Fusion operator. Check the log file /home/kni/logs/installoperator_playbook.log for more details.

Diagnosis

Run all commands as a kni user from the provisioner node.

Run the following command to check whether the catalogsource exists:
```
oc get catsrc -n openshift-marketplace
```

Verify whether the catalog source with name isf-catalog exists.

Sample output:

NAME                          DISPLAY                   TYPE   PUBLISHER   AGE
certified-operators           Certified Operators       grpc   Red Hat     3d9h
community-operators           Community Operators       grpc   Red Hat     3d9h
ibm-operator-catalog          IBM Operator Catalog      grpc   IBM         3d2h
isf-data-foundation-catalog   Data Foundation Catalog   grpc   IBM         3d5h
isf-catalog.                                            grpc               8h
redhat-marketplace            Red Hat Marketplace       grpc   Red Hat     3d9h
redhat-operators              Red Hat Operators         grpc   Red Hat     3d9h

Run the following command to check whether the catalogsource pod exists:
```
oc get pods -n openshift-marketplace
```

Check whether a pod that starts with isf-catalog- exists with the Status as Running and Ready as 1/1.

Example:

NAME                                  READY   STATUS             RESTARTS   AGE
isf-catalog-wh8zn                     1/1     Running            0          8h
marketplace-operator-dd4dff8f-jxwm4   1/1     Running            0          32h
redhat-marketplace-d9dl8              1/1     Running            0          32h
redhat-operators-gzk4q                1/1     Running            0          32h

If you find that the pod is not in running status, then run the following command to view the details:
```
oc descibe pod <POD NAME> -n openshift-marketplace
```

Scroll to the bottom and examine the Events section of the output.

Events:
  Type     Reason          Age   From               Message
  ----     ------          ----  ----               -------
  Normal   Scheduled       98m   default-scheduler  Error: ImagePullBackOff

If error from the describe command indicates authorisation or authentication error, then check whether the credentials you provided during the stage2 installation are correct.
- For online installation, it is the entitlement key that you provided. In this case, the user is always cp.
- For online installation, also check whether the specified entitlement key includes an IBM Fusion HCI entitlement.
- For installation from enterprise registry, check the provided credentials for your enterprise registry (for both multiple and single repositories).
- Try to pull the image manually on one OpenShift® Container Platform node by connecting to the node from provisioner by using oc debug node/<NODE-NAM>.
If the error from the previous describe command indicates manifest unknown error, then do the following checks:
- For installation from enterprise registry, check whether your enterprise registry is configured properly and reachable from provisoner (RU7).
- For installation from enterprise registry, make sure that the IBM Fusion HCI images are mirrored correctly.
- Make sure that the user mirror images with right digest to right path exist in the enterprise registry.
If isf-catalog pod is running successfully, then run the following command to check for IBM Fusion HCI namespace:
```
oc get namespace | grep ibm-spectrum-fusion-ns
```
Sample output:
```
ibm-spectrum-fusion-ns                             Active   3d9h
```
If the command oc get namespace | grep ibm-spectrum-fusion-ns gives a blank output, then it means that the IBM Fusion HCI namespace is not created. Check the log file /home/kni/logs/installoperator_playbook.log for the following message:
```
message":"etcdserver: request timed out","code":500}\n'", "reason": "Internal
      Server Error", "status": 500}
```
The failure indicates that this API issue is caused by the network bandwidth or any other latency. If the installation retries fail, contact IBM Support. Even if you do not find any API related issue in your logs, then contact IBM Support.
If the namespace is created, then run the following command to check for IBM Fusion operator group:
```
oc get og -n ibm-spectrum-fusion-ns
```
Sample output:
```
NAME     AGE
isf-og   3d9h
```
If the previous oc get og -n ibm-spectrum-fusion-ns command gives a blank output, then it means that the operator group is not created. Check the log file /home/kni/logs/installoperator_playbook.log for the following message:
```
message":"etcdserver: request timed out","code":500}\n'", "reason": "Internal
      Server Error", "status": 500}
```
The failure indicates that this API issue is caused by the network bandwidth or any other latency. If the installation retries fail, contact IBM Support. Even if you do not find any API related issue in your logs, then contact IBM Support.

If the operator group is created, then run the following command to check for IBM Fusion subscription:

oc get sub  -n ibm-spectrum-fusion-ns

Sample output:

NAME               PACKAGE        SOURCE            CHANNEL
isf-subscription   isf-operator   iso-catalog-new   v2.0

If the previous oc get og -n ibm-spectrum-fusion-ns command gives a blank output, then it means that the operator group is not created. Check the log file /home/kni/logs/installoperator_playbook.log for the following message:
```
message":"etcdserver: request timed out","code":500}\n'", "reason": "Internal
      Server Error", "status": 500}
```
The failure indicates that this API issue is caused by the network bandwidth or any other latency. If the installation retries fail, contact IBM Support. Even if you do not find any API related issue in your logs, then contact IBM Support.
If subscription is created, then run the following command to check for IBM Fusion install plan:
```
oc get ip -n ibm-spectrum-fusion-ns
```
Sample output:
```
NAME            CSV                            APPROVAL   APPROVED
install-bcqb4   isf-operator.v2.8.0-13321732   Manual     true
```
If the command oc get ip -n ibm-spectrum-fusion-ns gives a blank output, then it means that the install plan is not created.

If install plan is not found, then run the following commands to check jobs:

oc get job -n openshift-marketplace

Example output:

NAME                                                              COMPLETIONS   DURATION   AGE
0308d219fbaf8eec9372a4314df5af975f06f2ff3cd55c8fc33489e88deb346   1/1           9s         3d2h
08dd142dfe6be404a597f0c82c8bad5787997272aa8867842e1018e06b3c32f   1/1           8s         3d4h
0f59dc23d14241f8977ca45073457ddfaca24a68c7d0a6c8e14edf4d9bca337   1/1           7s         3d6h
19a4f4d1b807398820a95a3b292d4f595c442ae7861c49d5b27d09624c0e9b1   1/1           10s        3d
235cc047aad86ecb3c6b1d2d45ebf4bcbd54d89c352e386745edaa2b5e89b33   1/1           9s         3d2h
31c5fff269b3664c17b94430b42e652ab175ec2ebee85e6c7df01a7da6ff600   1/1           7s         3d3h
324ae09e681c9662339ae2dd9a38b2ad70c2430272c3229da527fd0f2b77fb8   1/1           12s        3d9h
56d513408bd5a601c1701a5ca18fa71ff4e8dc049c5f205ecfcde1897ee03bd   1/1           7s         3d6h
63d794fe8f947104f804d2db67793404dd4c95756e405bf30f5bd4ac59409b1   1/1           9s         3d
69cd2f81af46d28954bf54d01ba81582a3ee4059da62aba89fc4762f7c1d199   1/1           10s        3d6h
6f568dbc4626d4adcf92316d6d9e0a7250ae118b5cad7d97cac9525bf810ee3   1/1           9s         3d3h
7a16b39766293d24b1ea674bf2221c218603f148aa5b01def75c3c792dd6ee5   1/1           27s        3d6h
82067d1666f17dd7acd36ec050bfa065850240fcaaef5aba3ad11a91d6aa63f   1/1           9s         3d
963e64e4cb5615267ab3821d47a8acb6f04586013df3472949178b49f049c64   1/1           8s         3d6h
96eb43df54bdc1c822600b707e1078090ed2cfdd3587360a7545525097f52a6   1/1           9s         3d6h
c7906c42e8405161bde11a3e044dbc23c87bf31e17393e061c917b6074ff28a   1/1           12s        3d2h
c7f12c9dc97b5a771181f4fb58ecf7cd812f8d8a5756f810d40e5295f345f73   1/1           38s        3d
d0d80a9e2553e077d37d4a407fe3a58b4b44596df02efad7d5d962e0505680b   1/1           22s        3d
d68f3512c112308d77831d6c2765ee4c8e8d1960661a33dad59f0419e9fd732   1/1           9s         3d3h
f31ed4a99e495429a1d47697faef595eb7a0f59b40d43617f8f3e2018bd1842   1/1           7s         3d6h

If any job does not have COMPLETION as 1/1, then run the following command to check for errors:
```
oc get jobname -n openshift-marketplace
```

Next actions

Take corrective actions and click Retry in the installation user interface.