Centralized Logstash server

Effective 10.0.2604.2 June release, all Logstash servers of a service which is inside UtilityServiceGroup are merged into a single deployment.

Previously, you could create and manage each pipeline as a separate deployment. Now, all pipelines for a particular service run within one centralized Logstash server, making deployment simpler.

This architectural improvement simplifies management, reduces resource overhead, and provides better scalability for Logstash operations.

The following example shows how the logstash servers of a search service in the UtilityServiceGroup are configured in v1beta1 and v1.

Previous configuration - v1beta1

In earlier versions, each Logstash pipeline was configured as a separate server deployment.

searchService
 logstashServers:
  - active: true
    groupName: logstash-supply
    names:
      - logstash-supply
  - active: true
    groupName: logstash-demand
    names:
      - logstash-demand
  - active: true
    groupName: logstash-reservation
    names:
      - logstash-reservation
  - active: true
    groupName: logstash-inventorytag
    names:
      - logstash-inventorytag

New configuration - v1

With the centralized approach, all pipelines are consolidated into a single Logstash server deployment.

searchService
 logstashServers:
  - active: true
    groupName: logstash
    names:
      - logstash-supply
      - logstash-demand
      - logstash-reservation
      - logstash-inventorytag

For more information, see Versioning of Custom Resource Definitions.

Property consolidation behaviour

When migrating from multiple separate Logstash server configurations to a single consolidated deployment, understand how properties are handled in the following scenario.

Scenario: If you have multiple Logstash servers with different configurations such as resources, annotations, labels, HPA settings, and others, the consolidation process retains the properties from the first Logstash server in the list.

Example:

Before migration - v1beta1

logstashServers:
  - active: true
    affinityAndTolerations: <affinity_tolerations>
    horizontalPodAutoscaler: ApiSuppliesHPA
    logLevel: INFO
    pod:
      podAnnotations:
        lssupplyannkey1: lssupplyannvalue
      podLabels:
        logsupplylabkey1: logsupplylabvalue
    property:
      envVars: nn
      jvmArgs: mm
    resources:
      limits:
        cpu: '1'
        memory: 1536Mi
      requests:
        cpu: 250m
        memory: 512Mi
    names:
      - logstash-supply
  - active: true
    names:
      - logstash-demand
  - active: true
    affinityAndTolerations: different-setting
    horizontalPodAutoscaler: ApiReservationHPA
    logLevel: DEBUG
    pod:
      podAnnotations:
        lsreservannkey1: lsreservannvalue
      podLabels:
        logreservlabkey1: logreservlabvalue
    property:
      envVars: nnn
      jvmArgs: mmm
    resources:
      limits:
        cpu: '2'
        memory: 2048Mi
      requests:
        cpu: 500m
        memory: 1024Mi
    names:
      - logstash-reservation
  - active: true
    names:
      - logstash-inventorytag

After migration - v1

logstashServers:
  - active: true
    groupName: logstash
    affinityAndTolerations: <affinity_tolerations>  # From first server
    horizontalPodAutoscaler: ApiSuppliesHPA  # From first server
    logLevel: INFO  # From first server
    pod:
      podAnnotations:
        lssupplyannkey1: lssupplyannvalue  # From first server
      podLabels:
        logsupplylabkey1: logsupplylabvalue  # From first server
    property:
      envVars: nn  # From first server
      jvmArgs: mm  # From first server
    resources:
      limits:
        cpu: '1'  # From first server
        memory: 1536Mi  # From first server
      requests:
        cpu: 250m  # From first server
        memory: 512Mi  # From first server
    names:
      - logstash-supply
      - logstash-demand
      - logstash-reservation
      - logstash-inventorytag

Important considerations

  • Properties from subsequent servers are discarded: Any unique configurations defined in the second, third, or fourth Logstash servers will not be carried over to the consolidated deployment. If needed, manually merge required settings from other servers into the first server's configuration.
  • Unified behavior: After consolidation, all Logstash pipelines such as supply, demand, reservation, inventorytag will share the same resource limits, scaling policies, and runtime configurations. As these are all now in one single deployement.
  • If different Logstash pipelines previously had different resource requirements or configurations, evaluate whether the consolidated settings are appropriate for all pipelines. You may need to adjust the resource limits and HPA settings to accommodate the combined workload.
Tip: If different Logstash pipelines previously had different resource requirements or configurations, evaluate whether the consolidated settings are appropriate for all pipelines. You may need to adjust the resource limits and HPA settings to accommodate the combined workload.