Configuration

• Chart instances
  • Installing additional logging instances
  • Scaling out logging services after installation
• Custom certificate authority
• Data retention
  • Modifying the data retention policy for logging services
  • Manually removing log indices
• Data location
  • Enabling security for logging services
  • Managing resource allocation for logging services
  • Updating logging service data collection filters
  • Customizing Filebeat registry locations

Chart instances

You can deploy as many independent instances of logging as hardware capacity permits. The Helm chart used to deploy the logging service is included in the mgmt-charts repository. For more information about configuring multiple instances of logging for security and multi-tenancy, see Installing additional logging instances.

Custom certificate authority

The default configuration of the managed ELK stack uses your product certificate authority (CA). You can find the CA in the cluster-ca-cert secret in the kube-system namespace. The secret has two fields (tls.crt and tls.key) that contain the actual certificate and its private key. All later deployments of the ibm-icplogging Helm chart can use an existing certificate authority. Three requirements must be met:

1. The CA must be stored in a Kubernetes secret.
2. The secret must exist in the namespace to which the ELK stack is deployed.
3. The contents of the certificate and its secret key must be stored in separately named fields (or keys) within the Kubernetes secret.

For example, given a sample secret like the following code:

  apiVersion: v1
  kind: Secret
  metadata:
    name: my-ca-secret
  type: Opaque
  data:
    my_ca.crt: ...
    my_ca.key: ...

You must then configure the Helm chart with the following subset of values:

  security:
    ca:
      origin: external
      external:
        secretName: my-ca-secret
        certSecretKey: my_ca.crt
        keySecretKey: my_ca.key

Data retention

A container is deployed as a curator within each ELK stack. The curator removes indexes from Elasticsearch that are older than the configured maximum index age. Take care when you store logs for long periods of time. Each additional day of retained logs increases the memory and storage resources that Elasticsearch requires.

To modify default values for the managed ELK stack curator, add and customize the following lines in your config.yaml file.

logging:
  curator:
    name: log-curator
    image:
      repository: "ibmcom/indices-cleaner"
      tag: "2.0.0"
    # Runs at 23:30 UTC daily
    schedule: "30 23 * * *"
    # Application log retention
    app:
      unit: days
      count: 1
    # Elastcisearch cluster monitoring log retention
    monitoring:
      unit: days
      count: 1
    # X-Pack watcher plugin log retention
    watcher:
      unit: days
      count: 1

To change the settings after installation, see Modifying the data retention policy for logging services.

Timing

The curator is set to run on UTC time. Using a single time standard makes it easier to coordinate and anticipate curation across geographical regions.

The default launch time is set for half an hour before midnight UTC. The purpose is to avoid any risk that lag—perhaps due to congestion or system load—might start the curator after the midnight boundary and store more logs than expected.

Data location

Your product Elasticsearch deployment is configured to store documents in the /var/lib/icp/logging/elk-data directory of each management node to which it is deployed. You can change this path before installation by adding the following parameter to config.yaml. The new path must exist on all management nodes in the cluster.

elasticsearch_storage_dir: <your_path>