Collecting Kubernetes and Red Hat OpenShift logs

OpenTelemetry (OTEL) log collection topic covers the requirements for collecting logs and sending them to Instana with the help of OpenTelemetry. Instana accepts OpenTelemetry (OTEL) logs through different mechanisms. For more information, see Sending OpenTelemetry data to the Instana agent.

OpenTelemetry Collector cluster deployment scenarios

OpenTelemetry log collection: Running in Kubernetes

OpenTelemetry Collector installation

You can deploy the OTEL Collector by using either the Helm chart or Operator, both are configured to use the community-driven open source OTEL Collector Contrib package that includes features that are necessary to collect and send logs to Instana.

If you want to communicate with the Instana agent by using TLS-encrypted methods, follow the steps to set up TLS encryption for the agent endpoint.

This document assumes a minimum version for the Helm chart v0.71.1 and Operator v0.110.0, both of which use OTEL Collector v0.110.0. Older versions can still work but require different configurations and might have bugs that are fixed in newer releases.

Configuring the log collection in Kubernetes clusters with Helm Install

When you install the OTEL Collector by using the Helm Chart installation method, note the following considerations:

The Helm chart's opentelemetry-collector/templates/_config.tpl file contains preset definitions for the filelog receiver and the k8sattributes processor. You can enable these definitions by using the logsCollection and kubernetesAttributes presets options that are documented in the opentelemetry-collector/values.yaml file. To modify the behavior of the filelog receiver or k8sattributes processor in the Helm chart, update the opentelemetry-collector/templates/_config.tpl file instead of adding your own configuration in the opentelemetry-collector/values.yaml file, as doing so can partially overwrite of the configuration and cause unexpected behavior.

The pre-configured filelog receiver is able to parse container logs from multiple CRI supporting runtimes, for example ,containerd and cri-o.

  • (Optional) You can enhance the provided filelog receiver configuration in the opentelemetry-collector/templates/_config.tpl file by adding the following example of Recombine operator configuration to handle multi-line logs. Alternatively, users can allow Instana's internal multi-line log algorithms to handle the merging. However, if you have custom multi-line log formats that require custom stitching logic, you can implement that custom logic.
  • See the following example where recombine operator is added after the container-parser operator that extracts logs based on predetermined container log formatting:
  receivers:

    ## [REQUIRED] The filelog receiver will collect logs written to file by Kubernetes.
    filelog:
    [...]

      ## [REQUIRED] Provides file location information optional operators below (i.e. recombine operator) and to Instana.
      include_file_name: true

      operators:
        - type: container
          id: container-parser

        ## [OPTIONAL] Example recombine operator config to handle multi-line log messages for stack-traces. Requires `include_file_path: true` above.
        - type: recombine
          combine_field: body
          is_first_entry: body matches "^[^\\s]"
          source_identifier: attributes["log.file.path"]

The pre-configured k8sattributes processor is able to collect many potentially important Kubernetes pod configuration/metadata fields.

  • You must add the container.id field to the k8sattributes processor configuration in the opentelemetry-collector/templates/_config.tpl file. This field is necessary to link any given log record to its origin container.
    k8sattributes:
      extract:
        metadata:
          - "k8s.namespace.name"
          - "k8s.deployment.name"
          - "k8s.statefulset.name"
          - "k8s.daemonset.name"
          [...]
          [...]
          - "k8s.pod.uid"   ## Allows logs to be correlated to specific k8s pods by Instana.
          - "container.id"  ## Allows logs to be correlated to specific containers by Instana.

To enable the pre-configured filelog receiver and k8sattributes processor components, edit the opentelemetry-collector/values.yaml file and set enabled to true for logsCollection and kubernetesAttributes. See the following example:

presets:
  ## [REQUIRED] Automatically adds a pre-configured `filelog` receiver to all logs pipelines and configures container runtime specific message parsers.
  logsCollection:
    enabled: true
  ## [REQUIRED] Automatically adds a pre-configured `k8sattributes` processor to all logs pipelines to provide `k8s.pod.uid` metrics.
  kubernetesAttributes:
    enabled: true

For filelog receiver, It is recommended to set the mode for the OTEL Collector to daemonset. Set the following setting at the beginning of the opentelemetry-collector/values.yaml file:

mode: "daemonset"

To make sure that the OTEL Collector can access the pod log files generated by Kubernetes, grant the necessary security privileges to the OTEL Collector pod. In the opentelemetry-collector/values.yaml file, add the following security privileges to the securityContext field:

securityContext: { privileged: true }

When you set both the presets.logsCollection.enabled and presets.kubernetesAttributes.enabled flags to true, the OTEL Collector loads the pre-configured filelog receiver and k8sattributes processor. However, you must add some additional configurations to the opentelemetry-collector/values.yaml file. Specifically, replace the existing OTEL Collector configuration under the config key with the following:

Make sure that you to back up your existing opentelemetry-collector/values.yaml file before making the following larger changes.

config:
  ## [REQUIRED] Unless you require additional log receivers, no need to specify the `filelog` receiver since the `logCollection.enabled: true` preset automatically configures the receiver.
  ## Note: If no additional receivers are to be configured, setting the value to `{}` prevents a `null value` error from occurring in the OTEL Collector.
  receivers: {}

  processors:
    ## [REQUIRED] Resource processor can be manually added to set the `container.runtime` field. Adding this field helps Instana correlate logs to the correct type of Container technology.
    ## Note: In this example, the K8s cluster is running Containerd containers, so the appropriate `container.runtime` attribute is added.
    resource/container_runtime:
      attributes:
        - key: container.runtime
          value: containerd
          action: insert

    ## [OPTIONAL] This is an example log severity parser that sets the **severity_text** field in the log payload, each runs in-order such that the highest matching severity is set.
    ## Note: If the OpenTelemetry Collector does not set log severity, the severity is set by Instana when analyzing the log message.
    transform/severity_parse:
      log_statements:
        - context: log
        statements:
          - set(severity_text, "Info") where IsMatch(body.string, ".*INFO.*")
          - set(severity_text, "Warn") where IsMatch(body.string, ".*WARN.*")
          - set(severity_text, "Error") where IsMatch(body.string, ".*ERROR.*")
          - set(severity_text, "Fatal") where IsMatch(body.string, ".*FATAL.*")

    ## [REQUIRED] Logs must be sent in batches for performance reasons.
    ## Note: No additional `batch` processor configuration is provided since configuration depends on the user scenario.
    batch: {}

  exporters:
    ## [REQUIRED] The Instana Agent supports GRPC payloads
    otlp/instanaAgent:
      ## Note: This configuration assumes the Instana Agent is also deployed in the same cluster.
      ## Note: The GRPC port will be 4317 (unless port-forwarding is used to change this).
      endpoint: 'instana-agent.instana-agent:4317'
      ## TLS encryption is disabled in this example.
      tls:
        insecure: true

  service:
    pipelines:
      ## [REQUIRED] Sample logs pipeline using the above configurations.
      logs:
        receivers: [filelog]
        processors: [resource/container_runtime, k8sattributes, transform/severity_parse, batch]
        exporters: [otlp/instanaAgent]

Configuring the log collection in Kubernetes clusters with Operator Install

Before you install the OTEL Collector by using the operator, you must first install the cert-manager.

To install the cert-manager, run the following command:

  • kubectl apply -f https://github.com/cert-manager/cert-manager/releases/download/v1.16.1/cert-manager.yaml

After you install the cert-manager, install the operator by running the latest Installation manifest for Kubernetes. Run the following command:

  • kubectl apply -f https://github.com/open-telemetry/opentelemetry-operator/releases/latest/download/opentelemetry-operator.yaml

After you install the operator, proceed to configure the OTEL Collector. Unlike with the Helm Chart installation method, the operator does not provide presets for configuring the filelog receiver and k8sattributes processor, so you need to configure them manually. Use the following example OTEL Collector manual configuration, copy it to a YAML file, for example otel-collector-config.yaml, and apply it to your Kubernetes cluster by running the following command:

  • kubectl apply -f otel-collector-config.yaml
apiVersion: opentelemetry.io/v1beta1

kind: OpenTelemetryCollector

metadata:
  ## Users should set this name as needed
  name: simple-otelcol

spec:
  ## [REQUIRED] Grants the OTEL Collector necessary privileges to access kubernetes log files.
  securityContext: { privileged: true }

  ## [REQUIRED] Best configuration for the `filelog` receiver.
  mode: daemonset

  ## [REQUIRED] We assume minimum OTEL Collector Contrib release v0.110.0 in this documentation.
  image: otel/opentelemetry-collector-contrib:0.110.0
  imagePullPolicy: IfNotPresent

  ## [REQUIRED] It is necessary to specify the mounts that provide the OTEL Collector with access to the pod log files.
  volumeMounts:
    - name: varlogpods
      mountPath: /var/log/pods
      readOnly: true
  volumes:
    - name: varlogpods
      hostPath:
        path: /var/log/pods

  config:
    receivers:
      ## [REQUIRED] The filelog receiver will collect logs written to file by Kubernetes.
      filelog:
        ## [REQUIRED] Standard location for cluster pod log files.
        include: [ /var/log/pods/*/*/*.log ]

        ## [OPTIONAL] Path (or regex) to the log files that must be ignored.
        ## Note: This could be the OTEL Collector pod log located at:
        ##      /var/log/pods/{"opentelemetry collector namespace"}_{"opentelemetry collector fullname"}*_*/{"opentelemetry collector lowercase_chartname"}/*.log
        exclude: [ "/path/to/log/files/to/ignore" ]

        ## [REQUIRED] Whether to include the file path in the logs
        include_file_path: true

        ## [OPTIONAL] Whether to include the file name in the logs
        include_file_name: true

        ## [OPTIONAL] Preserve the leading white spaces so that the example 'recombine' operator works as expected.
        preserve_leading_whitespaces: true

        operators:
          ## [REQUIRED] Container log parser.
          - type: container
            id: container-parser

          ## [OPTIONAL] Example recombine operator config to handle multi-line log messages for stack-traces. Requires `include_file_path: true` above.
          - type: recombine
            combine_field: body
            is_first_entry: body matches "^[^\\s]"
            source_identifier: attributes["log.file.path"]

    processors:
      ## [REQUIRED] The k8sattributes processor provides UIDs needed to correlate container/pod logs to their corresponding entities.
      ## Note: The following configures collection of the `container.id` and `k8s.pod.uid` fields in the `k8sattributes` processor for correlation.
      ## Note: In some cases, the `container.id` field cannot be provided by the OpenTelemetry SDK, which means Instana will fallback on the `k8s.pod.uid`.
      k8sattributes:
        passthrough: false
        pod_association:
        - sources:
          - from: resource_attribute
            name: container.id
        - sources:
          - from: resource_attribute
            name: k8s.pod.ip
        - sources:
          - from: resource_attribute
            name: k8s.pod.uid
        - sources:
          - from: connection
        extract:
          ## [REQUIRED] At minimum we need the following metadata fields extracted by the `k8sattributes` processor.
          metadata:
            - "k8s.pod.uid"   ## If container.id is unavailable, logs are correlated to a specific Kubernetes pod.
            - "container.id"  ## If this field is available, logs are correlated to specific containers in a Kubernetes pod.
          ## [OPTIONAL] Useful if you want to collect pod labels and send them to Instana as "custom-tags". See the "transform/copy-from-resourcelog" processor below.
          labels:
            - tag_name: $$1
              key_regex: (.*)
              from: pod
          ## [OPTIONAL] Useful if you want to collect pod annotations and send them to Instana as "custom-tags". See the "transform/copy-from-resourcelog" processor below.
          annotations:
            - tag_name: $$1
              key_regex: (.*)
              from: pod

      ## [OPTIONAL] Example OTEL Processor for extracting resource-log attributes collected from pod labels/annotations to be processed as "custom-tags" by Instana
      transform/copy-from-resourcelog:
        log_statements:
          - context: log
            statements:
              ## Example where a pod service name is collected by the k8sattributes processor. This field can be added as a pod label in the `pod.yaml` file.
              - set(attributes["service_name"], resource.attributes["service_name"])
              ## Example where a pod's start time is collected by the k8sattributes processor.
              - set(attributes["Pod Start Timestamp"], resource.attributes["k8s.pod.start_time"])
              ## Example where a specific field, in this case some `x-forwarder-ip` value is extracted from log messages, using a regex.
              ## Note: Due to the usage of the regex, the entire `set(...)` command is surrounded by single-quotes to avoid YAML parse errors.
              - 'set(attributes["x-forwarder-ip"], ExtractPatterns(body, "x-forwarder-ip: (?P<value>[0-9\\.]+)"))'

      ## [REQUIRED] Resource processor can be manually added to set the `container.runtime` field. Adding this field helps Instana correlate logs to the correct type of Container technology.
      ## Note: In this example, the K8s cluster is running Containerd containers, so the appropriate `container.runtime` attribute is added.
      resource/container_runtime:
        attributes:
        - key: container.runtime
          value: containerd
          action: insert

      ## [OPTIONAL] This is an example log severity parser that sets the **severity_text** field in the log payload, each runs in-order such that the highest matching severity is set.
      ## Note: If the OpenTelemetry Collector does not set log severity, then the severity is set by Instana when analyzing the log message.
      transform/severity_parse:
        log_statements:
        - context: log
          statements:
          - set(severity_text, "Info") where IsMatch(body.string, ".*INFO.*")
          - set(severity_text, "Warn") where IsMatch(body.string, ".*WARN.*")
          - set(severity_text, "Error") where IsMatch(body.string, ".*ERROR.*")
          - set(severity_text, "Fatal") where IsMatch(body.string, ".*FATAL.*")

      ## Logs must be sent in batches for performance reasons.
      ## Note: No additional `batch` processor configuration is provided since configuration depends on the user scenario.
      batch: {}

    exporters:
      ## [REQUIRED] The Instana Agent supports GRPC payloads
      otlp/instanaAgent:
        ## Note: This configuration assumes the Instana Agent is also deployed in the same cluster.
        ## Note: The GRPC port will be 4317 (unless port-forwarding is used to change this).
        endpoint: 'instana-agent.instana-agent:4317'
        ## TLS encryption is disabled in this example.
        tls:
          insecure: true

    service:
      pipelines:
        ## [REQUIRED] Sample logs pipeline using the above configurations.
        logs:
          receivers: [filelog]
          processors: [resource/container_runtime, k8sattributes, transform/severity_parse, batch]
          exporters: [otlp/instanaAgent]

OpenTelemetry log collection: Running in Red Hat OpenShift

You can set up Red Hat OpenShift similarly to the previously documented steps for generic Kubernetes deployments. Complete the following additional steps to deploy the OTEL Collector into Red Hat OpenShift clusters.

Setting up the Red Hat OpenShift CLI

In general, you can manage your Red Hat OpenShift cluster remotely with the help of the Red Hat OpenShift command line interface (CLI). While you might follow the rest of the steps in this document by SSHing to the INF (Infrastructure) node, use the Red Hat OpenShift CLI to manage your cluster.

Install the Red Hat OpenShift CLI oc on your workstation. For more information, see Getting started with the OpenShift CLI.

Configure your Red Hat OpenShift CLI to connect to your Red Hat OpenShift cluster by using the following command to remotely log in to the cluster:

$ oc login --web --server=https://<infrastructure node's hostname>:6443

Now, any oc commands that you run on your workstation are redirected to your Red Hat OpenShift cluster.

Security context

Run the OTEL Collector as a privileged container under Red Hat OpenShift to access log files in /var/log/pods on the underlying node. Define a security context for the OTEL Collector to set the required security access permissions.

Define the security access permissions that the OTEL Collector requires by creating a Security Context Constraint (SCC) file with the following contents.

In this example, name the file scc.yaml.

apiVersion: security.openshift.io/v1

kind: SecurityContextConstraints

metadata:
  name: otel-scc

allowPrivilegedContainer: true
allowHostDirVolumePlugin: true
allowHostPorts: true

runAsUser:
  type: RunAsAny

seLinuxContext:
  type: RunAsAny

fsGroup:
  type: RunAsAny

users:
## [REQUIRE] Replace `<namespace>:<otel-pod-name>` as needed for your OTEL Collector installation.
## Example: `system:serviceaccount:mynamespace:opentelemetry-collector`
- system:serviceaccount:<namespace>:<otel-pod-name>

Run the command oc apply -f scc.yaml to setup the security context.