Heapster remains in an unhealthy state

After installation, Heapster might be in an unhealthy state and unable to recover.

Symptoms

Either, or both, of the following two symptoms might occur:

1. Running the command kubectl logs <heapster_pod_name> eventer reveals entries with messages such as Failed to config ElasticSearch or no Elasticsearch node available.
2. The Logs tab for a pod in the IBM® Cloud Private management console does not load. No log entries appear even after an extended period.

Causes

In some cases, Calico failed to correctly configure network isolation for the Elasticsearch cluster. Pods that are configured to contact the cluster on port 9200 might be rejected.

Resolving the problem

Delete and re-create the network policies for Elasticsearch and Kibana.

1. Delete the existing policies.

   kubectl delete netpol elasticsearch-client-policy elasticsearch-internal-policy kibana-policy -n kube-system
   

2. Connect remotely into the master node.

3. From your cluster installation directory, run this command:

   kubectl install -f cluster/cfc-components/elasticsearch/es-netpolicy.yaml
   

   This command resets the Calico settings on the Elasticsearch nodes. The next time that Kubernetes attempts to start the Heapster pod, it can access the Elasticsearch pod.

4. (Optional) Delete the Heapster pod to force Kubernetes to immediately restart it:

   kubectl delete pod <heapster_pod_name>