Analytics persistent queue

For enhanced reliability in your analytics data pipeline, enable the persistent queue feature.

About this task

The Logstash persistent queue is an optional feature that provides enhanced reliability in the data pipeline. With the persistent queue enabled, the ingestion pod buffers incoming API event data to persistent storage so that no events are lost if the downstream API Connect services become unavailable.

For example, if the analytics storage service is restarted, any incoming API events are kept in persistent storage and sent on to the storage service when it is available again. If the persistent queue is not enabled, then the ingestion pod rejects further API event data coming from the gateway.

The persistent queue feature does not replay previously sent API events (if you change offload settings for example).

Another benefit of this feature is better handling of unexpected traffic spikes. The only disadvantages are higher CPU and persistent storage requirements.

Procedure

  1. Open the analytics-extra-values.yaml file for editing.
    Remember: All additions to the file must appear after the spec: statement.
  2. Add the following section to the file: 
    ingestion:
  queue:
     type: persisted
     volumeClaimTemplate:
        storageClassName: local-storage
        volumeSize: 30Gi

    You can set volumeSize larger if required, but 30Gi is the minimum.

  3. Save the file.

What to do next

If you do not want to add any more configuration options, skip to the topic Deploying the Analytics subsystem.

Otherwise, continue adding configuration options to the file as explained in the following topics.