Unable to access Management UIs

If you are unable to access the UI login page, getting a blank page, an HTTP 404, or some other error.

When this happens it means that either the Management subsystem is down, or the access attempt is not reaching the Management subsystem due to a problem in the environment.

Note: If you are on VMware, some of the following checks involve running kubectl commands on your VMs. To access your VMs for running kubectl commands:

SSH to one of your subsystem VMs as the user 'apicadm', for example: ssh apicadm@mgmt1.example.com
Sudo to the root user: sudo -i

From here you can run kubectl commands, for example:

# kubectl get pods
NAME                                                              READY   STATUS      RESTARTS      AGE
abc-management-analytics-proxy-5f8bcd74c4-jbhch                   1/1     Running     1 (97d ago)   113d
...

API Connect pods run in the default namespace on VMware, so you do not need to specify the namespace with -n in the kubectl commands.

Check the following:

Is the access attempt actually reaching the Management subsystem? To confirm this, check what is in the logs of the juhu pods which run on the Management subsystem. The juhu pods are the entry point for all UI, REST, and CLI requests to the Management subsystem. For every access request there should be line logged by the juhupod, for example: When the user attempts to access the admin UI login page, the following is logged by the juhu pod:
```
kubectl logs abc-management-juhu-fc8f6bb5d-jnw2q --since=5m
...
9.20.85.205 - - [27/May/2022:10:35:49 +0000] [675fa59030a1a68028712ad2fdb2fdb1] "GET /api/cloud/admin/identity-providers HTTP/1.1" 200 301 "https://apimsvc20.hursley.ibm.com/auth/admin/sign-in/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/101.0.4951.64 Safari/537.36" "9.145.13.216" [127.0.0.1:2000] request_time=0.222 upstream_response_time=0.224
...
```
Note that the timestamp of these log records are not the time when the request arrived, but the time when the response to the request was sent back. To work out the arrival time you can subtract the request_time=0.222 that appears later in the log message. If there is nothing logged in the juhu pods at the time the UI access is attempted, then check if you have any network problems between your browser and the Management subsystem. If you are on OVA then the tcpdump utility is available which you can use to confirm if traffic is reaching your Management VM, for example tcpdump -i any src host 9.20.86.22 and dst port 443
Are the Management subsystem pods all up and running? Check this with kubectl get ManagementCluster -n <namespace>. If this does not show all pods running, then check which pods are failing: kubectl get pods -n <namespace>. If you are on OVA then gather postmortem logs and open a support case at this point. Otherwise it's worth checking the reasons why the pods are failing to start with kubectl describe as it might be something in your kubernetes environment that you can fix. Note that on the Management subsystem the postgres pods will need to be running first, so if these are not running then most other Management pods will not start.
Check the logs of the apim pods while attempting to access the UI. The juhu pods receive the requests from the UI, and send them on to the apim pods for processing. The logs of the apim pods may reveal some clues as to the cause of the problem. If nothing is logged in the apim pods at the time of the access attempt then it suggests a problem in the environment, the juhu pods unable to forward requests to apim.