Configuration

Chart instances

You can deploy as many independent instances of logging as hardware capacity permits. The Helm chart used to deploy the logging service is included in the mgmt-charts repository. For more information about configuring multiple instances of logging for security and multi-tenancy, see Installing additional logging instances.

Custom certificate authority

The default configuration of the managed ELK stack uses the IBM Cloud Private certificate authority (CA). You can find the CA in the cluster-ca-cert secret in the kube-system namespace. The secret has two fields (tls.crt and tls.key) that contain the actual certificate and its private key. All later deployments of the ibm-icplogging Helm chart can use an existing certificate authority. Three requirements must be met:

  1. The CA must be stored in a Kubernetes secret.
  2. The secret must exist in the namespace to which the ELK stack is deployed.
  3. The contents of the certificate and its secret key must be stored in separately named fields (or keys) within the Kubernetes secret.

For example, given a sample secret like the following code:

  apiVersion: v1
  kind: Secret
  metadata:
    name: my-ca-secret
  type: Opaque
  data:
    my_ca.crt: ...
    my_ca.key: ...

You must then configure the Helm chart with the following subset of values:

  security:
    ca:
      origin: external
      external:
        secretName: my-ca-secret
        certSecretKey: my_ca.crt
        keySecretKey: my_ca.key

Data retention

A container is deployed as a curator within each ELK stack. The curator removes indexes from Elasticsearch that are older than the configured maximum index age. Take care when you store logs for long periods of time. Each additional day of retained logs increases the memory and storage resources that Elasticsearch requires.

To modify default values for the managed ELK stack curator, add and customize the following lines in your config.yaml file.

logging:
  curator:
    name: log-curator
    image:
      repository: "ibmcom/indices-cleaner"
      tag: "2.0.0"
    # Runs at 23:30 UTC daily
    schedule: "30 23 * * *"
    # Application log retention
    app:
      unit: days
      count: 1
    # Elastcisearch cluster monitoring log retention
    monitoring:
      unit: days
      count: 1
    # X-Pack watcher plugin log retention
    watcher:
      unit: days
      count: 1

To change the settings after installation, see Modifying the data retention policy for logging services.

Timing

The curator is set to run on UTC time. Using a single time standard makes it easier to coordinate and anticipate curation across geographical regions.

The default launch time is set for half an hour before midnight UTC. The purpose is to avoid any risk that lag—perhaps due to congestion or system load—might start the curator after the midnight boundary and store more logs than expected.

Streaming IBM Cloud Private platform logs off-site

Platform components are deployed into the IBM Cloud Private system namespace kube-system by default. Also by default, only platform components will deploy to nodes labeled master, management, or proxy.

In this scenario, it is possible to configure the managed ELK stack in the IBM Cloud Private system namespace to stream IBM Cloud Private platform logs to an off-platform collection service.

Complete the following steps to stream all IBM Cloud Private platform logs to an external service.

  1. Modify the Filebeat Daemonset definition for the IBM Cloud Private system namespace to specify node affinity only to nodes labeled master, management, or proxy.
  2. Modify the Logstash configuration for the stack that is deployed to the IBM Cloud Private system namespace to stream logs to an off-platform collection service. For more information, see the Logstash documentation.
  3. If no longer needed, delete the Elasticsearch and Kibana deployments and StatefulSets defined in the IBM Cloud Private system namespace.

Important notes about this configuration:

  1. The managed ELK stack will no longer collect any application logs.
  2. The configuration changes will not persist through an upgrade or rollback of the Logging release.
  3. Not every possible Logstash configuration change has been tested in the managed ELK stack. Depending on the changes made, it may be necessary to completely delete and recreate the Logging stack to return to the default state, losing previously collected logs and configuration.
  4. Some platform services may run on separate nodes, and would not have their logs captured. For example,
    • Vulnerability Advisor runs on nodes with a separate label and would not be captured.
    • Metering, and even Logging itself, utilize Daemonsets that run on worker nodes. Logs from the components running on worker nodes would not be captured.
  5. Additional logging stacks or log collection services can still capture the platform logs if they are configured to collect logs from the labeled cluster nodes.
  6. Kibana in the managed ELK stack may fail to load or may fail to have access to collected logs depending on configuration changes that are made.

Data location

IBM Cloud Private's Elasticsearch deployment is configured to store documents in the /var/lib/icp/logging/elk-data directory of each management node to which it is deployed. You can change this path before installation by adding the following parameter to config.yaml. The new path must exist on all management nodes in the cluster.

elasticsearch_storage_dir: <your_path>