Empty Analytics display on Kubernetes

If your API Connect analytics data doesn't display in the Analytics dashboard, complete the following steps to determine the cause of the problem.

Procedure

Run the following command to check the logs of the mtls-gw pod:
```
kubectl logs analyt-mtls-gw-7665f5d445-4cx4n -n <APIC_namespace>
```
The response looks like the following example:
```
10.254.6.1 - - [22/Sep/2021:18:05:54 +0000] "POST /x2020/v1/apievent/_bulk HTTP/1.1" 200 2 "-""-" 2841 0.072 0.070
```
1. Check the response for any _bulk request coming from the gateway, as shown in the example.
  
  If there are no _bulk requests, then the problem is with the gateway and the wsm-agent, which is not sending bulk requests to the Analytics server.
2. If you see the _bulk request., then check the return code in the response.
  
  If _bulk requests are processed successfully you will see a return code 200 (as in the example), which indicates that the bulk request has been ingested.
3. If it is any other return code, check the ingestion pod logs for any errors, which will indicate any issues during ingestion:
```
oc logs analyt-ingestion-78d7cff7f7-6pktd
```

If everything with the data flow looks fine, and if you are unable to see analytics data, then check the health of Analytics by executing into one of the storage pods:

kubectl exec -it <storage_pod> — bash

       d.bash-4.4$ curl_es  _cluster/health?pretty=1

For example:

{
  "cluster_name" : "apic-analytics-cluster",
  "status": "yellow",
  "timed_out": false,
  "number_of_nodes": 3,
  "number_of_data_nodes": 1,
  "active_primary_shards": 258,
  "active_shards": 258,
  "relocating_shards": 0,
  "initializing_shards": 0,
  "unassigned_shards": 510,
  "delayed_unassigned_shards": 0,
  "number_of_pending_tasks": 0,
  "number_of_in_flight_fetch": 0,
  "task_max_waiting_in_queue_millis": 0,
  "active_shards_percent_as_number": 33.59375
}

Check the status of the cluster (red, yellow, green) in the response.
In the example, the status is set to yellow.
- Green indicates all indices are fine.
- Yellow indicates there are some unassigned shards, which might still be replicating.
- Red indicates there is a problem with one or more of the indices.
Run the following command to get a list of indices:
```
bash-4.4$ curl_es  _cat/indices?v
```
In the list, find the red indices.
Delete each red index by running the following command:
```
bash-4.4$ curl_es -XDELETE <index_name>
```