Performance tuning with ingestion pipeline properties

Fine tune your ingestion pipeline settings for optimal performance.

Table 1.
Field Type Default value Description Comments 
ingestion:
  pipelineWorkers
 4 <= integer <= 128 24 Number of pipeline workers. More workers mean more CPU and memory requirements, but gives you the benefit of extra threads to pull data off the ingestion HTTP interface and process it for storage. Increasing the number of pipeline workers excessively results in diminishing returns. pipelineWorkers should usually be a multiple of the number of cores of your hardware. Adjust with caution. 
ingestion:
  pipelineBatchSize
 10 <= integer <= 10000 250 Batch size of a pipeline worker. Pulling more or less data off the HTTP interface per worker can have benefits. Recommended values are 125, 250, and 500. Increasing the value requires more memory. 
ingestion:
  pipelineBatchDelay
 10 <= integer <= 5000 50 Batch delay of a pipeline worker. How long to wait for the batch size to fill before leaving the HTTP interface to process for storage. Having threads wait too long causes idle threads in your system; however, setting this value too low decreases performance. Adjust with caution. 
ingestion:
  configReloadAutomatic
 Boolean true Whether to automatically reload the pipeline on configuration changes. Typically, when the underlying files for the pipeline are modified, the pipeline process automatically reloads and picks up the changes. However, in Kubernetes environments where the pipeline is mapped to ConfigMaps, this is unreliable. Ingestion pod restarts are usually required. 
ingestion:
  pipelineUnsafeShutdown
 Boolean false Whether to allow unsafe shutdown of the pipeline. Typically, the pipeline attempts to empty all workers' batches before shutdown to prevent data loss. You might need to enable this setting if the pipeline is badly configured and refuses to respond to shut down requests (uncommon). Adjust with caution.