Empty Analytics display on Kubernetes

If your API Connect analytics data doesn't display in the Analytics dashboard, complete the following steps to determine the cause of the problem.

Procedure

Run the following command to check the logs of the mtls-gw pod:
```
kubectl logs analyt-mtls-gw-7665f5d445-4cx4n -n <APIC_namespace>
```
The response looks like the following example:
```
10.254.6.1 - - [22/Sep/2021:18:05:54 +0000] "POST /x2020/v1/apievent/_bulk HTTP/1.1" 200 2 "-""-" 2841 0.072 0.070
```
1. Check the response for any _bulk request coming from the gateway, as shown in the example.
  
  If there are no _bulk requests, then the problem is with the gateway and the wsm-agent, which is not sending bulk requests to the Analytics server.
2. If you see the _bulk request., then check the return code in the response.
  
  If _bulk requests are processed successfully you will see a return code 200 (as in the example), which indicates that the bulk request has been ingested.
3. If it is any other return code, check the ingestion pod logs for any errors, which will indicate any issues during ingestion:
```
kubectl logs analyt-ingestion-78d7cff7f7-6pktd
```
If everything with the data flow looks fine, and if you are unable to see analytics data, then check the health of Analytics using the CLI.
1. If you do not have the CLI installed see ../com.ibm.apic.apionprem.doc/analytics_use_cli_and_rest_api.html for the steps on installing and using.
2. Run the following CLI command:
```
apic --mode analytics clustermgmt:getHealth  --server <management server platform api endpoint> --analytics-service <analytics service name> --format json
```
where:
- <management server platform api endpoint> is the Management server admin endpoint.
- <analytics service name> is the name of Analytics service as configured in the Cloud Manager.
When you check the cluster's health, the response for a single node analytics deployment looks like the following example:
```
{
    "cluster_name": "apic-analytics-cluster",
    "status": "yellow",
    "timed_out": false,
    "number_of_nodes": 1,
    "number_of_data_nodes": 1,
    "discovered_master": true,
    "discovered_cluster_manager": true,
    "active_primary_shards": 8,
    "active_shards": 8,
    "relocating_shards": 0,
    "initializing_shards": 0,
    "unassigned_shards": 12,
    "delayed_unassigned_shards": 0,
    "number_of_pending_tasks": 0,
    "number_of_in_flight_fetch": 0,
    "task_max_waiting_in_queue_millis": 0,
    "active_shards_percent_as_number": 40
}
```
Note: Single node analytics clusters never show green in the health output. Green indicates that the data is replicated to at least one other node, and since there are no other nodes in a single node cluster it remains as yellow.
1. Check the status of the cluster (red, yellow, green) in the response.
  In the example, the status is set to yellow.
  - Green indicates all indices are fine.
  - Yellow indicates there are some unassigned shards, which might still be replicating.
  - Red indicates there is a problem with one or more of the indices.
2. Run the following command to get a list of indices:
```
apic --mode analytics clustermgmt:catIndices --server [SERVER] --analytics-service [ANALYTICS SERVICE]
```
  where:
  - SERVER is the Management server admin endpoint.
  - ANALYTICS SERVICE is the name of Analytics service as configured in the Cloud Manager.
3. From the list of indices returned, identify the those with status of 'red'.
4. Delete each red index by running the following command:
```
apic --mode analytics clustermgmt:deleteIndex --index <index_name> --server [SERVER] --analytics-service [ANALYTICS SERVICE] --format json
```
  where:
  - SERVER is the Management server admin endpoint.
  - ANALYTICS SERVICE is the name of Analytics service as configured in the Cloud Manager.