Logstash persistent queue

Configuring the Logstash persistent queue feature.

The Logstash persistent queue provides enhanced reliability in the data pipeline. The ingestion pod buffers incoming API event data to persistent storage so that no events are lost if the downstream local storage pods or offload targets are temporarily not available.

For example, when you restart the analytics storage pods they take a few seconds to be able to receive API event data. While the storage pods are restarting, incoming API events are held in the persistent queue until the storage pods are available again. If the storage pods are not available and the persistent queue is not enabled, then new API event data is lost during the storage pod outage.

If you find you are missing API event records, it is possible that the default persistent queue size is insufficient. This document provides the steps to increase your persistent queue size.

Note:

For OpenShift® users - The example steps in this document use the Kubernetes kubectl command. On OpenShift, use the equivalent oc command in its place.
If you are using a top-level CR, you must edit the APIConnectCluster CR (the top-level CR), instead of directly editing the subsystem CRs.
If you are using a top-level CR (APIConnectCluster), make sure that the spec.gateway section is included. If it is not included already, you must add the spec.gateway section before adding the required property within it.

Customizing the persistent queue size

By default the size of the persistent queue is 8 Gi. If you want to change this size, then follow these steps according to your platform.

Note: The configuration of the size of the persistent queues is supported only on API Connect 10.0.8.1 and later.

Calculate the storage space required based on the queue size you want to set.
- Local storage enabled and no offload is configured - 2 x queue size + 5 Gi.
- Local storage enabled and one offload target is configured - 3 x queue size + 5 Gi.
- Local storage enabled and two offload targets configured - 4 x queue size + 5 Gi.
- Local storage disabled and one offload target is configured - 2 x queue size + 5 Gi.
- Local storage disabled and two offload targets configured - 3 x queue size + 5 Gi.
Note: Disabling local storage and no offload configured require 1 * queue size + 5 Gi. However, this configuration is not required as it means that analytics data is not recorded anywhere.
Check whether the required storage exceeds the current value of ingestion.queue.volumeClaimTemplate.volumeSize. By default this is 50 Gi, so if you are increasing the queue size over 11 Gi, then you must increase storage.
On Kubernetes and Red Hat OpenShift you can check the storage by looking at the analytics CR.
```
kubectl -n <namespace> edit a7s
```
If you need more storage than you have defined in ingestion.queue.volumeClaimTemplate.volumeSize, then complete the following steps.
Note: Increasing the ingestion storage size causes analytics ingestion downtime. No incoming API event data is stored or offloaded while you complete these steps.
1. Scale down the ingestion pods to 0 replicas.
  Identify the name of the ingestion StatefulSet:
```
kubectl get sts | grep ingestion
```
  Set ingestion replicas to zero:
```
kubectl scale sts <ingestion statefulset> --replicas=0
```
2. Delete the ingestion pod PVCs.
  Identify the ingestion pod PVCs:
```
kubectl get pvc | grep ingestion
```
  Delete all ingestion PVCs:
```
kubectl delete pvc <name>-analytics-ingestion-<integer>
```
3. Update the analytics CR with the new defaultQueueSize and volumeSize:
```
    queue:
      type: persisted
      defaultQueueSize: <queue size>Gi
      volumeClaimTemplate:
        ...
        volumeSize: <volumeSize>Gi
```
4. Scale the ingestion pods back up:
```
kubectl scale sts <ingestion statefulset> --replicas=<original number of replicas>
```
5. Verify that ingestion pod and PVC are re-created:
```
kubectl get pods | grep ingestion
```
```
kubectl get pvc | grep ingestion
```
Make sure that you have sufficient storage that is defined in ingestion.queue.volumeClaimTemplate.volumeSize, then complete the following steps.
- Kubernetes, Openshift, and Cloud Pak for Integration - Edit your analytics CR by using the following command.
```
kubectl -n <namespace> edit a7s
```
  Add the spec.ingestion.queue.defaultQueueSize property to your analytics CR, and set the volumeClaimTemplate.volumeSize. For example,
```
    queue:
      type: persisted
      defaultQueueSize: <queue size>Gi
      volumeClaimTemplate:
        ...
        volumeSize: <volumeSize>Gi
```

Important points are as follows.

defaultQueueSize is optional. If not specified it defaults to 8 Gi.
defaultQueueSize must be specified in Gi, and must be an integer.
The minimum value for defaultQueueSize 8 Gi.
The maximum value for defaultQueueSize 100 Gi.
The ingestion.queue.volumeClaimTemplate.volumeSize must be at least 30 Gi, the recommended and default value is 50 Gi.