UI fails to launch

A leadership election between the pods can sometimes cause the nominee to crash, or move and change its IP address.

Problem

The primary host's UI fails to launch. The following error message is displayed in both the logs and the browser:
Reached the max retries per request limit (which is 20). Refer to \"maxRetriesPerRequest\" option for details, "level": "fatal"

Cause

This is due to a timing issue in selecting a controller pod and is a known issue.

Resolution

See the following example to scale the Redis pods down and up again to bring the UI back.

To scale down:
kubectl scale statefulset -ibm-redis-server --replicas=0
To scale up:
kubectl scale statefulset -ibm-redis-server --replicas=<number of redis pods>