Collecting Kubernetes and Red Hat OpenShift logs

You can deploy the OTEL Collector by using either the Helm chart or Operator, both are configured to use the community-driven open source OTEL Collector Contrib package that includes features that are necessary to collect and send logs to Instana.

If you want to communicate with the Instana agent by using TLS-encrypted methods, follow the steps to set up TLS encryption for the agent endpoint.

This document assumes a minimum version for the Helm chart v0.71.1 and Operator v0.110.0, both of which use OTEL Collector v0.110.0. Older versions can still work but require different configurations and might have bugs that are fixed in newer releases.

In either case, first install the Instana agent and have the OTEL collector route data through the agent. When you install the agent, you must also enable the OpenTelemetry plug-in by specifying the following configuration in the installation YAML file that contains the OpenTelemetry options. See the following example:

    configuration_yaml: |
      # You can leave this empty, or use this to configure your instana agent.
      # See https://docs.instana.io/setup_and_manage/host_agent/on/kubernetes/
      com.instana.plugin.opentelemetry:
        enabled: true
        http:
          enabled: true
        grpc:
          enabled: true

When you install the OTEL Collector by using the Helm Chart installation method, note the following considerations:

The Helm chart's opentelemetry-collector/templates/_config.tpl file contains preset definitions for the filelog receiver and the k8sattributes processor. You can enable these definitions by using the logsCollection and kubernetesAttributes presets options that are documented in the opentelemetry-collector/values.yaml file. To modify the behavior of the filelog receiver or k8sattributes processor in the Helm chart, update the opentelemetry-collector/templates/_config.tpl file instead of adding your own configuration in the opentelemetry-collector/values.yaml file, as doing so can partially overwrite of the configuration and cause unexpected behavior.

The preconfigured filelog receiver is able to parse container logs to extract Kubernetes (k8s) metadata.

You can enhance the provided filelog receiver configuration in the opentelemetry-collector/templates/_config.tpl file by adding the following example of Recombine operator configuration to handle multiline logs. Alternatively, you can rely on the Instana internal multiline log algorithms to handle the merging.

However, for custom multiline log formats that require custom stitching logic, you can implement that custom logic. See the following example where recombine operator is added after the container-parser operator that extracts logs based on predetermined container log formatting:

  receivers:

    ## [REQUIRED] The filelog receiver will collect logs written to file by Kubernetes.
    filelog:
    [...]

      ## [REQUIRED] Provides file location information optional operators below (i.e. recombine operator) and to Instana.
      include_file_name: true

      operators:
          ## [REQUIRED] Container log parser. Collects necessary information for correlation by k8sattributes processor.
        - type: container
          id: container-parser

        ## [OPTIONAL] Example recombine operator config to handle multi-line log messages for stack-traces. Requires `include_file_path: true` above.
        - type: recombine
          combine_field: body
          is_first_entry: body matches "^[^\\s]"
          source_identifier: attributes["log.file.path"]

    k8sattributes:
      extract:
        metadata:
          - "k8s.node.name"
          - "k8s.namespace.name"
          - "k8s.deployment.name"
          ## Here add any additional metadata you want to extract from the k8s environment. ## For the full list of supported attributes: https://github.com/open-telemetry/opentelemetry-collector-contrib/blob/main/processor/k8sattributesprocessor/README.md 
          - "service.name" ## Allows logs to be correlated to specific services by Instana.
          - "k8s.pod.uid"   ## Allows logs to be correlated to specific k8s pods by Instana.
          - "container.id"  ## Allows logs to be correlated to specific containers by Instana.

To enable the pre-configured filelog receiver and k8sattributes processor components, edit the opentelemetry-collector/values.yaml file and set enabled to true for logsCollection and kubernetesAttributes. See the following example:

presets:
  ## [REQUIRED] Automatically adds a pre-configured `filelog` receiver to all logs pipelines and configures container runtime specific message parsers.
  logsCollection:
    enabled: true
  ## [REQUIRED] Automatically adds a pre-configured `k8sattributes` processor to all logs pipelines to provide `k8s.pod.uid` metrics.
  kubernetesAttributes:
    enabled: true

For filelog receiver, set the mode for the OTEL Collector to daemonset. Include the following setting at the beginning of the opentelemetry-collector/values.yaml file:

mode: "daemonset"

To make sure that the OTEL Collector can access the pod log files that Kubernetes generates, grant the necessary security privileges to the OTEL Collector pod. In the opentelemetry-collector/values.yaml file, add the following security privileges to the securityContext field:

securityContext: { privileged: true }

When you set both the presets.logsCollection.enabled and presets.kubernetesAttributes.enabled flags to true, the OTEL Collector loads the pre-configured filelog receiver and k8sattributes processor. However, you must add some additional configurations to the opentelemetry-collector/values.yaml file. Specifically, replace the existing OTEL Collector configuration under the config key with the following configuration:

config:
  ## [REQUIRED] Unless you require additional log receivers, no need to specify the `filelog` receiver since the `logCollection.enabled: true` preset automatically configures the receiver.
  ## Note: If no additional receivers are to be configured, setting the value to `{}` prevents a `null value` error from occurring in the OTEL Collector.
  receivers: {}

  processors:
    ## [REQUIRED] Resource processor can be manually added to set the `container.runtime` field. Adding this field helps Instana correlate logs to the correct type of Container technology.
    ## Permitted values:
    ## - containerd
    ## - crio
    ## - docker
    ## - garden
    ## Note: In this example, the K8s cluster is running Containerd containers, so the appropriate `container.runtime` attribute is added.
    resource/container_runtime:
      attributes:
        - key: container.runtime
          value: containerd
          action: insert

    ## [OPTIONAL] This is an example log severity parser that sets the **severity_text** field in the log payload, each runs in-order such that the highest matching severity is set.
    ## Note: If the OpenTelemetry Collector does not set log severity, the severity is set by Instana when analyzing the log message.
    transform/set_log_severity:
      log_statements:
        - context: log
        statements:
          - set(severity_text, "Info") where IsMatch(body.string, ".*INFO.*")
          - set(severity_text, "Warn") where IsMatch(body.string, ".*WARN.*")
          - set(severity_text, "Error") where IsMatch(body.string, ".*ERROR.*")
          - set(severity_text, "Fatal") where IsMatch(body.string, ".*FATAL.*")

    ## [REQUIRED] Logs must be sent in batches for performance reasons.
    ## Note: No additional `batch` processor configuration is provided since configuration depends on the user scenario.
    batch: {}
    ## See the page on best practices for Instana OpenTelemetry logging for more information.

  exporters:
    ## [REQUIRED] The Instana Agent supports GRPC payloads
    otlp/instanaAgent:
      ## Note: This configuration assumes the Instana Agent is also deployed in the same cluster.
      ## Note: The GRPC port will be 4317 (unless port-forwarding is used to change this).
      endpoint: instana-agent.instana-agent:4317
      ## TLS encryption is disabled in this example.
      tls:
        insecure: true

  service:
    pipelines:
      ## [REQUIRED] Sample logs pipeline using the above configurations.
      logs:
        receivers: [filelog]
        processors: [resource/container_runtime, k8sattributes, transform/set_log_severity, batch]
        exporters: [otlp/instanaAgent]

Before you install the OTEL Collector by using the operator, you must first install the cert-manager.

To install the cert-manager, run the following command:

• kubectl apply -f https://github.com/cert-manager/cert-manager/releases/download/v1.16.1/cert-manager.yaml

Then, create a new project to hold the OTEL Collector:

oc new-project otel-collector

Run the OTEL Collector as a privileged container under Red Hat OpenShift to access log files in /var/log/pods on the underlying node. Define a security context for the OTEL Collector to set the required security access permissions.

Define the security access permissions that the OTEL Collector requires by creating a Security Context Constraint (SCC) file with the following contents.

In this example, name the file scc.yaml.

apiVersion: security.openshift.io/v1

kind: SecurityContextConstraints

metadata:
  name: otel-scc

allowPrivilegedContainer: true
allowHostDirVolumePlugin: true
allowHostPorts: true

runAsUser:
  type: RunAsAny

seLinuxContext:
  type: RunAsAny

fsGroup:
  type: RunAsAny

users:
## [REQUIRE] The format of the next line is
##  `system:serviceaccount:<namespace>:<otel-pod-name>`
## The provided example matches the defaults used in this documentation.
## If you did not use the default values, then you should change this
## line to reflect that.
- system:serviceaccount:otel-collector:simple-otelcol-collector

Run the command oc apply -f scc.yaml to set up the security context.

After you install the security context, install the operator by running the latest Installation manifest for Kubernetes. Run the following command:

• kubectl apply -f https://github.com/open-telemetry/opentelemetry-operator/releases/latest/download/opentelemetry-operator.yaml

After you install the operator, proceed to configure the OTEL Collector. Unlike with the Helm Chart installation method, the operator does not provide presets for configuring the filelog receiver and k8sattributes processor. Therefore, you must configure them manually. Use one of the following example OTEL Collector manual configurations, copy it to a YAML file (for example otel-collector-config.yaml), and apply it to your Kubernetes cluster by running the following command:

• kubectl apply -f otel-collector-config.yaml

The first example is a basic configuration, which contains only the minimum settings that is required to start the OTEL Collector and monitor the logs of all detected pods in the Red Hat OpenShift environment. This configuration does no particular processing of the data, and instead, it forwards the data to the agent without any modifications. This configuration is a good starting point to test the working of the Collector. You can add the required customizations later.

apiVersion: opentelemetry.io/v1beta1

kind: OpenTelemetryCollector

metadata:
  ## Users should set this name as needed
  name: simple-otelcol

spec:
  ## [REQUIRED] Grants the OTEL Collector necessary privileges to access kubernetes log files.
  securityContext: { privileged: true }

  ## [REQUIRED] Best configuration for the `filelog` receiver.
  mode: daemonset

  ## [REQUIRED] We assume minimum OTEL Collector Contrib release v0.110.0 in this documentation.
  image: otel/opentelemetry-collector-contrib:0.110.0
  imagePullPolicy: IfNotPresent

  ## [REQUIRED] It is necessary to specify the mounts that provide the OTEL Collector with access to the pod log files.
  volumeMounts:
    - name: varlogpods
      mountPath: /var/log/pods
      readOnly: true
  volumes:
    - name: varlogpods
      hostPath:
        path: /var/log/pods

  config:
    receivers:
      ## [REQUIRED] The filelog receiver will collect logs written to file by Kubernetes.
      filelog:
        ## [REQUIRED] Standard location for cluster pod log files. Update if different.
        include: [ /var/log/pods/*/*/*.log ]

        ## [REQUIRED] Whether to include the file path in the logs.
        include_file_path: true

        operators:
          ## [REQUIRED] Container log parser. Collects necessary information for correlation by k8sattributes processor.
          - type: container
            id: container-parser

    processors:
      ## [REQUIRED] The k8sattributes processor provides UIDs needed to correlate container/pod logs to their corresponding entities.
      ## Note: The following configures collection of the `container.id` and `k8s.pod.uid` fields in the `k8sattributes` processor for correlation.
      ## Note: In some cases, the `container.id` field cannot be provided by the OpenTelemetry SDK, which means Instana will fallback on the `k8s.pod.uid`.
      k8sattributes:
        passthrough: false
        pod_association:
        - sources:
          - from: resource_attribute
            name: container.id
        - sources:
          - from: resource_attribute
            name: k8s.pod.ip
        - sources:
          - from: resource_attribute
            name: k8s.pod.uid
        - sources:
          - from: connection
        extract:
          ## [REQUIRED] At minimum we need the following metadata fields extracted by the `k8sattributes` processor.
          metadata:
            - "k8s.node.name" 
            - "k8s.namespace.name" 
            - "k8s.deployment.name" 
            ## Here add any additional metadata you want to extract from the k8s environment. 
            ## For the full list of supported attributes: https://github.com/open-telemetry/opentelemetry-collector-contrib/blob/main/processor/k8sattributesprocessor/README.md 
            - "service.name" ## Allows logs to be correlated to specific services by Instana.
            - "k8s.pod.uid"   ## If container.id is unavailable, logs are correlated to a specific Kubernetes pod.
            - "container.id"  ## If this field is available, logs are correlated to specific containers in a Kubernetes pod.

      ## [REQUIRED] Resource processor can be manually added to set the `container.runtime` field. Adding this field helps Instana correlate logs to the correct type of Container technology.
      ## Note: In this example, the K8s cluster is running Containerd containers, so the appropriate `container.runtime` attribute is added.
      resource/container_runtime:
        attributes:
        - key: container.runtime
          value: containerd
          action: insert

      ## Logs must be sent in batches for performance reasons.
      ## Note: No additional `batch` processor configuration is provided since configuration depends on the user scenario.
      batch: {}
      ## See the page on best practices for Instana OpenTelemetry logging for more information.

    exporters:
      ## [REQUIRED] The Instana Agent supports GRPC payloads
      otlp/instanaAgent:
        ## Note: This configuration assumes the Instana Agent is also deployed in the same cluster.
        ## Note: The GRPC port will be 4317 (unless port-forwarding is used to change this).
        endpoint: 'http://instana-agent.instana-agent:4317'
        ## TLS encryption is disabled in this example.
        tls:
          insecure: true

    service:
      pipelines:
        ## [REQUIRED] Sample logs pipeline using the above configurations.
        logs:
          receivers: [filelog]
          processors: [resource/container_runtime, k8sattributes, batch]
          exporters: [otlp/instanaAgent]

The next configuration contains some optional fields to show some additional capabilities that are possible with the OTEL Collector. These fields are marked with [OPTIONAL] in a comment before the field. This configuration does not contain a comprehensive list of everything that is possible with the OTEL Collector, and instead, it only provides examples of some commonly used options.

apiVersion: opentelemetry.io/v1beta1

kind: OpenTelemetryCollector

metadata:
  ## Users should set this name as needed
  name: simple-otelcol

spec:
  ## [REQUIRED] Grants the OTEL Collector necessary privileges to access kubernetes log files.
  securityContext: { privileged: true }

  ## [REQUIRED] Best configuration for the `filelog` receiver.
  mode: daemonset

  ## [REQUIRED] We assume minimum OTEL Collector Contrib release v0.110.0 in this documentation.
  image: otel/opentelemetry-collector-contrib:0.110.0
  imagePullPolicy: IfNotPresent

  ## [REQUIRED] It is necessary to specify the mounts that provide the OTEL Collector with access to the pod log files.
  volumeMounts:
    - name: varlogpods
      mountPath: /var/log/pods
      readOnly: true
  volumes:
    - name: varlogpods
      hostPath:
        path: /var/log/pods

  config:
    receivers:
      ## [REQUIRED] The filelog receiver will collect logs written to file by Kubernetes.
      filelog:
        ## [REQUIRED] Standard location for cluster pod log files. Update if different.
        include: [ /var/log/pods/*/*/*.log ]

        ## [OPTIONAL] Path (or regex) to the log files that must be ignored.
        ## Note: This could be the OTEL Collector pod log located at:
        ##      /var/log/pods/{"opentelemetry collector namespace"}_{"opentelemetry collector fullname"}*_*/{"opentelemetry collector lowercase_chartname"}/*.log
        exclude: [ "/path/to/log/files/to/ignore" ]

        ## [REQUIRED] Whether to include the file path in the logs
        include_file_path: true

        ## [OPTIONAL] Whether to include the file name in the logs
        include_file_name: true

        ## [OPTIONAL] Preserve the leading white spaces so that the example 'recombine' operator works as expected.
        preserve_leading_whitespaces: true

        operators:
          ## [REQUIRED] Container log parser. Collects necessary information for correlation by k8sattributes processor.
          - type: container
            id: container-parser

          ## [OPTIONAL] Example recombine operator config to handle multi-line log messages for stack-traces. Requires `include_file_path: true` above.
          - type: recombine
            combine_field: body
            is_first_entry: body matches "^[^\\s]"
            source_identifier: attributes["log.file.path"]

    processors:
      ## [REQUIRED] The k8sattributes processor provides UIDs needed to correlate container/pod logs to their corresponding entities.
      ## Note: The following configures collection of the `container.id` and `k8s.pod.uid` fields in the `k8sattributes` processor for correlation.
      ## Note: In some cases, the `container.id` field cannot be provided by the OpenTelemetry SDK, which means Instana will fallback on the `k8s.pod.uid`.
      k8sattributes:
        passthrough: false
        pod_association:
        - sources:
          - from: resource_attribute
            name: container.id
        - sources:
          - from: resource_attribute
            name: k8s.pod.ip
        - sources:
          - from: resource_attribute
            name: k8s.pod.uid
        - sources:
          - from: connection
        extract:
          ## [REQUIRED] At minimum we need the following metadata fields extracted by the `k8sattributes` processor.
          metadata:
            - "k8s.node.name" 
            - "k8s.namespace.name" 
            - "k8s.deployment.name" 
            ## Here add any additional metadata you want to extract from the k8s environment. 
            ## For the full list of supported attributes: https://github.com/open-telemetry/opentelemetry-collector-contrib/blob/main/processor/k8sattributesprocessor/README.md 
            - "service.name" ## Allows logs to be correlated to specific services by Instana.
            - "k8s.pod.uid"   ## If container.id is unavailable, logs are correlated to a specific Kubernetes pod.
            - "container.id"  ## If this field is available, logs are correlated to specific containers in a Kubernetes pod.
          ## [OPTIONAL] Useful if you want to collect pod labels and send them to Instana as "custom-tags". See the "transform/copy-from-resourcelog" processor below.
          labels:
            - tag_name: $$1
              key_regex: (.*)
              from: pod
          ## [OPTIONAL] Useful if you want to collect pod annotations and send them to Instana as "custom-tags". See the "transform/copy-from-resourcelog" processor below.
          annotations:
            - tag_name: $$1
              key_regex: (.*)
              from: pod

      ## [OPTIONAL] Example OTEL Processor for extracting resource-log attributes collected from pod labels/annotations to be processed as "custom-tags" by Instana
      transform/copy-from-resourcelog:
        log_statements:
          - context: log
            statements:
              ## Example where a pod's start time is collected by the k8sattributes processor.
              - set(attributes["Pod Start Timestamp"], resource.attributes["k8s.pod.start_time"])
              ## Example where a specific field, in this case some `x-forwarder-ip` value is extracted from log messages, using a regex.
              ## Note: Due to the usage of the regex, the entire `set(...)` command is surrounded by single-quotes to avoid YAML parse errors.
              - 'set(attributes["x-forwarder-ip"], ExtractPatterns(body, "x-forwarder-ip: (?P<value>[0-9\\.]+)"))'

      ## [REQUIRED] Resource processor can be manually added to set the `container.runtime` field. Adding this field helps Instana correlate logs to the correct type of Container technology.
      ## Note: In this example, the K8s cluster is running Containerd containers, so the appropriate `container.runtime` attribute is added.
      resource/container_runtime:
        attributes:
        - key: container.runtime
          value: containerd
          action: insert

      ## [OPTIONAL] This is an example log severity parser that sets the **severity_text** field in the log payload, each runs in-order such that the highest matching severity is set.
      ## Note: If the OpenTelemetry Collector does not set log severity, then the severity is set by Instana when analyzing the log message.
      transform/set_log_severity:
        log_statements:
        - context: log
          statements:
          - set(severity_text, "Info") where IsMatch(body.string, ".*INFO.*")
          - set(severity_text, "Warn") where IsMatch(body.string, ".*WARN.*")
          - set(severity_text, "Error") where IsMatch(body.string, ".*ERROR.*")
          - set(severity_text, "Fatal") where IsMatch(body.string, ".*FATAL.*")

      ## Logs must be sent in batches for performance reasons.
      ## Note: No additional `batch` processor configuration is provided since configuration depends on the user scenario.
      batch: {}
      ## See the page on best practices for Instana OpenTelemetry logging for more information.

    exporters:
      ## [REQUIRED] The Instana Agent supports GRPC payloads
      otlp/instanaAgent:
        ## Note: This configuration assumes the Instana Agent is also deployed in the same cluster.
        ## Note: The GRPC port will be 4317 (unless port-forwarding is used to change this).
        endpoint: instana-agent.instana-agent:4317
        ## TLS encryption is disabled in this example.
        tls:
          insecure: true

    service:
      pipelines:
        ## [REQUIRED] Sample logs pipeline using the above configurations.
        logs:
          receivers: [filelog]
          processors: [resource/container_runtime, k8sattributes, transform/set_log_severity, batch]
          exporters: [otlp/instanaAgent]

In general, you can manage your Red Hat OpenShift cluster remotely with the help of the Red Hat OpenShift command line interface (CLI). While you might follow the rest of the steps in this document by logging in to the INF (Infrastructure) node, use the Red Hat OpenShift CLI to manage your cluster.

Install the Red Hat OpenShift CLI oc on your workstation. For more information, see Getting started with the Red Hat OpenShift CLI.

Configure your Red Hat OpenShift CLI to connect to your Red Hat OpenShift cluster by using the following command to remotely log in to the cluster:

$ oc login --web --server=https://<infrastructure node's hostname>:6443

Now, any oc commands that you run on your workstation are redirected to your Red Hat OpenShift cluster.