Collecting Kubernetes and Red Hat OpenShift logs
OpenTelemetry (OTEL) log collection topic covers the requirements for collecting logs and sending them to Instana with the help of OpenTelemetry. Instana accepts OpenTelemetry (OTEL) logs through different mechanisms. For more information, see Sending OpenTelemetry data to the Instana agent.
OpenTelemetry Collector cluster deployment scenarios
OpenTelemetry log collection: Running in Kubernetes
OpenTelemetry Collector installation
You can deploy the OTEL Collector by using either the Helm chart or Operator, both are configured to use the community-driven open source OTEL Collector Contrib
package that includes features
that are necessary to collect and send logs to Instana.
If you want to communicate with the Instana agent by using TLS-encrypted methods, follow the steps to set up TLS encryption for the agent endpoint.
This document assumes a minimum version for the Helm chart v0.71.1 and Operator v0.110.0, both of which use OTEL Collector v0.110.0. Older versions can still work but require different configurations and might have bugs that are fixed in newer releases.
Configuring the log collection in Kubernetes clusters with Helm Install
When you install the OTEL Collector by using the Helm Chart installation method, note the following considerations:
The Helm chart's opentelemetry-collector/templates/_config.tpl
file contains preset definitions for the filelog
receiver and the k8sattributes
processor. You can enable these definitions by using the
logsCollection
and kubernetesAttributes
presets options that are documented in the opentelemetry-collector/values.yaml
file. To modify the behavior of the filelog
receiver or k8sattributes
processor in the Helm chart, update the opentelemetry-collector/templates/_config.tpl
file instead of adding your own configuration in the opentelemetry-collector/values.yaml
file, as doing so can partially overwrite
of the configuration and cause unexpected behavior.
The pre-configured filelog
receiver is able to parse container logs from multiple CRI supporting runtimes, for example ,containerd
and cri-o
.
- (Optional) You can enhance the provided
filelog
receiver configuration in theopentelemetry-collector/templates/_config.tpl
file by adding the following example of Recombine operator configuration to handle multi-line logs. Alternatively, users can allow Instana's internal multi-line log algorithms to handle the merging. However, if you have custom multi-line log formats that require custom stitching logic, you can implement that custom logic. - See the following example where
recombine operator
is added after thecontainer-parser
operator that extracts logs based on predetermined container log formatting:
receivers:
## [REQUIRED] The filelog receiver will collect logs written to file by Kubernetes.
filelog:
[...]
## [REQUIRED] Provides file location information optional operators below (i.e. recombine operator) and to Instana.
include_file_name: true
operators:
- type: container
id: container-parser
## [OPTIONAL] Example recombine operator config to handle multi-line log messages for stack-traces. Requires `include_file_path: true` above.
- type: recombine
combine_field: body
is_first_entry: body matches "^[^\\s]"
source_identifier: attributes["log.file.path"]
The pre-configured k8sattributes
processor is able to collect many potentially important Kubernetes pod configuration/metadata fields.
- You must add the
container.id
field to thek8sattributes
processor configuration in theopentelemetry-collector/templates/_config.tpl
file. This field is necessary to link any given log record to its origin container.
k8sattributes:
extract:
metadata:
- "k8s.namespace.name"
- "k8s.deployment.name"
- "k8s.statefulset.name"
- "k8s.daemonset.name"
[...]
[...]
- "k8s.pod.uid" ## Allows logs to be correlated to specific k8s pods by Instana.
- "container.id" ## Allows logs to be correlated to specific containers by Instana.
To enable the pre-configured filelog
receiver and k8sattributes
processor components, edit the opentelemetry-collector/values.yaml
file and set enabled
to true
for logsCollection
and kubernetesAttributes
. See the following example:
presets:
## [REQUIRED] Automatically adds a pre-configured `filelog` receiver to all logs pipelines and configures container runtime specific message parsers.
logsCollection:
enabled: true
## [REQUIRED] Automatically adds a pre-configured `k8sattributes` processor to all logs pipelines to provide `k8s.pod.uid` metrics.
kubernetesAttributes:
enabled: true
For filelog
receiver, It is recommended to set the mode for the OTEL Collector to daemonset
. Set the following setting at the beginning of the opentelemetry-collector/values.yaml
file:
mode: "daemonset"
To make sure that the OTEL Collector can access the pod log files generated by Kubernetes, grant the necessary security privileges to the OTEL Collector pod. In the opentelemetry-collector/values.yaml
file, add the following
security privileges to the securityContext
field:
securityContext: { privileged: true }
When you set both the presets.logsCollection.enabled
and presets.kubernetesAttributes.enabled
flags to true
, the OTEL Collector loads the pre-configured filelog
receiver and k8sattributes
processor. However, you must add some additional configurations to the opentelemetry-collector/values.yaml
file. Specifically, replace the existing OTEL Collector configuration under the config
key with the following:
Make sure that you to back up your existing opentelemetry-collector/values.yaml
file before making the following larger changes.
config:
## [REQUIRED] Unless you require additional log receivers, no need to specify the `filelog` receiver since the `logCollection.enabled: true` preset automatically configures the receiver.
## Note: If no additional receivers are to be configured, setting the value to `{}` prevents a `null value` error from occurring in the OTEL Collector.
receivers: {}
processors:
## [REQUIRED] Resource processor can be manually added to set the `container.runtime` field. Adding this field helps Instana correlate logs to the correct type of Container technology.
## Note: In this example, the K8s cluster is running Containerd containers, so the appropriate `container.runtime` attribute is added.
resource/container_runtime:
attributes:
- key: container.runtime
value: containerd
action: insert
## [OPTIONAL] This is an example log severity parser that sets the **severity_text** field in the log payload, each runs in-order such that the highest matching severity is set.
## Note: If the OpenTelemetry Collector does not set log severity, the severity is set by Instana when analyzing the log message.
transform/severity_parse:
log_statements:
- context: log
statements:
- set(severity_text, "Info") where IsMatch(body.string, ".*INFO.*")
- set(severity_text, "Warn") where IsMatch(body.string, ".*WARN.*")
- set(severity_text, "Error") where IsMatch(body.string, ".*ERROR.*")
- set(severity_text, "Fatal") where IsMatch(body.string, ".*FATAL.*")
## [REQUIRED] Logs must be sent in batches for performance reasons.
## Note: No additional `batch` processor configuration is provided since configuration depends on the user scenario.
batch: {}
exporters:
## [REQUIRED] The Instana Agent supports GRPC payloads
otlp/instanaAgent:
## Note: This configuration assumes the Instana Agent is also deployed in the same cluster.
## Note: The GRPC port will be 4317 (unless port-forwarding is used to change this).
endpoint: 'instana-agent.instana-agent:4317'
## TLS encryption is disabled in this example.
tls:
insecure: true
service:
pipelines:
## [REQUIRED] Sample logs pipeline using the above configurations.
logs:
receivers: [filelog]
processors: [resource/container_runtime, k8sattributes, transform/severity_parse, batch]
exporters: [otlp/instanaAgent]
Configuring the log collection in Kubernetes clusters with Operator Install
Before you install the OTEL Collector by using the operator, you must first install the cert-manager
.
To install the cert-manager
, run the following command:
kubectl apply -f https://github.com/cert-manager/cert-manager/releases/download/v1.16.1/cert-manager.yaml
After you install the cert-manager
, install the operator
by running the latest Installation manifest for Kubernetes. Run the following
command:
kubectl apply -f https://github.com/open-telemetry/opentelemetry-operator/releases/latest/download/opentelemetry-operator.yaml
After you install the operator
, proceed to configure the OTEL Collector. Unlike with the Helm Chart installation method, the operator does not provide presets for configuring the filelog
receiver and k8sattributes
processor, so you need to configure them manually. Use the following example OTEL Collector manual configuration, copy it to a YAML file, for example otel-collector-config.yaml
, and apply it to your Kubernetes cluster by running
the following command:
kubectl apply -f otel-collector-config.yaml
apiVersion: opentelemetry.io/v1beta1
kind: OpenTelemetryCollector
metadata:
## Users should set this name as needed
name: simple-otelcol
spec:
## [REQUIRED] Grants the OTEL Collector necessary privileges to access kubernetes log files.
securityContext: { privileged: true }
## [REQUIRED] Best configuration for the `filelog` receiver.
mode: daemonset
## [REQUIRED] We assume minimum OTEL Collector Contrib release v0.110.0 in this documentation.
image: otel/opentelemetry-collector-contrib:0.110.0
imagePullPolicy: IfNotPresent
## [REQUIRED] It is necessary to specify the mounts that provide the OTEL Collector with access to the pod log files.
volumeMounts:
- name: varlogpods
mountPath: /var/log/pods
readOnly: true
volumes:
- name: varlogpods
hostPath:
path: /var/log/pods
config:
receivers:
## [REQUIRED] The filelog receiver will collect logs written to file by Kubernetes.
filelog:
## [REQUIRED] Standard location for cluster pod log files.
include: [ /var/log/pods/*/*/*.log ]
## [OPTIONAL] Path (or regex) to the log files that must be ignored.
## Note: This could be the OTEL Collector pod log located at:
## /var/log/pods/{"opentelemetry collector namespace"}_{"opentelemetry collector fullname"}*_*/{"opentelemetry collector lowercase_chartname"}/*.log
exclude: [ "/path/to/log/files/to/ignore" ]
## [REQUIRED] Whether to include the file path in the logs
include_file_path: true
## [OPTIONAL] Whether to include the file name in the logs
include_file_name: true
## [OPTIONAL] Preserve the leading white spaces so that the example 'recombine' operator works as expected.
preserve_leading_whitespaces: true
operators:
## [REQUIRED] Container log parser.
- type: container
id: container-parser
## [OPTIONAL] Example recombine operator config to handle multi-line log messages for stack-traces. Requires `include_file_path: true` above.
- type: recombine
combine_field: body
is_first_entry: body matches "^[^\\s]"
source_identifier: attributes["log.file.path"]
processors:
## [REQUIRED] The k8sattributes processor provides UIDs needed to correlate container/pod logs to their corresponding entities.
## Note: The following configures collection of the `container.id` and `k8s.pod.uid` fields in the `k8sattributes` processor for correlation.
## Note: In some cases, the `container.id` field cannot be provided by the OpenTelemetry SDK, which means Instana will fallback on the `k8s.pod.uid`.
k8sattributes:
passthrough: false
pod_association:
- sources:
- from: resource_attribute
name: container.id
- sources:
- from: resource_attribute
name: k8s.pod.ip
- sources:
- from: resource_attribute
name: k8s.pod.uid
- sources:
- from: connection
extract:
## [REQUIRED] At minimum we need the following metadata fields extracted by the `k8sattributes` processor.
metadata:
- "k8s.pod.uid" ## If container.id is unavailable, logs are correlated to a specific Kubernetes pod.
- "container.id" ## If this field is available, logs are correlated to specific containers in a Kubernetes pod.
## [OPTIONAL] Useful if you want to collect pod labels and send them to Instana as "custom-tags". See the "transform/copy-from-resourcelog" processor below.
labels:
- tag_name: $$1
key_regex: (.*)
from: pod
## [OPTIONAL] Useful if you want to collect pod annotations and send them to Instana as "custom-tags". See the "transform/copy-from-resourcelog" processor below.
annotations:
- tag_name: $$1
key_regex: (.*)
from: pod
## [OPTIONAL] Example OTEL Processor for extracting resource-log attributes collected from pod labels/annotations to be processed as "custom-tags" by Instana
transform/copy-from-resourcelog:
log_statements:
- context: log
statements:
## Example where a pod service name is collected by the k8sattributes processor. This field can be added as a pod label in the `pod.yaml` file.
- set(attributes["service_name"], resource.attributes["service_name"])
## Example where a pod's start time is collected by the k8sattributes processor.
- set(attributes["Pod Start Timestamp"], resource.attributes["k8s.pod.start_time"])
## Example where a specific field, in this case some `x-forwarder-ip` value is extracted from log messages, using a regex.
## Note: Due to the usage of the regex, the entire `set(...)` command is surrounded by single-quotes to avoid YAML parse errors.
- 'set(attributes["x-forwarder-ip"], ExtractPatterns(body, "x-forwarder-ip: (?P<value>[0-9\\.]+)"))'
## [REQUIRED] Resource processor can be manually added to set the `container.runtime` field. Adding this field helps Instana correlate logs to the correct type of Container technology.
## Note: In this example, the K8s cluster is running Containerd containers, so the appropriate `container.runtime` attribute is added.
resource/container_runtime:
attributes:
- key: container.runtime
value: containerd
action: insert
## [OPTIONAL] This is an example log severity parser that sets the **severity_text** field in the log payload, each runs in-order such that the highest matching severity is set.
## Note: If the OpenTelemetry Collector does not set log severity, then the severity is set by Instana when analyzing the log message.
transform/severity_parse:
log_statements:
- context: log
statements:
- set(severity_text, "Info") where IsMatch(body.string, ".*INFO.*")
- set(severity_text, "Warn") where IsMatch(body.string, ".*WARN.*")
- set(severity_text, "Error") where IsMatch(body.string, ".*ERROR.*")
- set(severity_text, "Fatal") where IsMatch(body.string, ".*FATAL.*")
## Logs must be sent in batches for performance reasons.
## Note: No additional `batch` processor configuration is provided since configuration depends on the user scenario.
batch: {}
exporters:
## [REQUIRED] The Instana Agent supports GRPC payloads
otlp/instanaAgent:
## Note: This configuration assumes the Instana Agent is also deployed in the same cluster.
## Note: The GRPC port will be 4317 (unless port-forwarding is used to change this).
endpoint: 'instana-agent.instana-agent:4317'
## TLS encryption is disabled in this example.
tls:
insecure: true
service:
pipelines:
## [REQUIRED] Sample logs pipeline using the above configurations.
logs:
receivers: [filelog]
processors: [resource/container_runtime, k8sattributes, transform/severity_parse, batch]
exporters: [otlp/instanaAgent]
OpenTelemetry log collection: Running in Red Hat OpenShift
You can set up Red Hat OpenShift similarly to the previously documented steps for generic Kubernetes deployments. Complete the following additional steps to deploy the OTEL Collector into Red Hat OpenShift clusters.
Setting up the Red Hat OpenShift CLI
In general, you can manage your Red Hat OpenShift cluster remotely with the help of the Red Hat OpenShift command line interface (CLI). While you might follow the rest of the steps in this document by SSHing to the INF (Infrastructure) node, use the Red Hat OpenShift CLI to manage your cluster.
Install the Red Hat OpenShift CLI oc
on your workstation. For more information, see Getting started with the OpenShift CLI.
Configure your Red Hat OpenShift CLI to connect to your Red Hat OpenShift cluster by using the following command to remotely log in to the cluster:
$ oc login --web --server=https://<infrastructure node's hostname>:6443
Now, any oc
commands that you run on your workstation are redirected to your Red Hat OpenShift cluster.
Security context
Run the OTEL Collector as a privileged container under Red Hat OpenShift to access log files in /var/log/pods
on the underlying node. Define a security context for the OTEL Collector to set the required security access permissions.
Define the security access permissions that the OTEL Collector requires by creating a Security Context Constraint (SCC) file with the following contents.
In this example, name the file scc.yaml
.
apiVersion: security.openshift.io/v1
kind: SecurityContextConstraints
metadata:
name: otel-scc
allowPrivilegedContainer: true
allowHostDirVolumePlugin: true
allowHostPorts: true
runAsUser:
type: RunAsAny
seLinuxContext:
type: RunAsAny
fsGroup:
type: RunAsAny
users:
## [REQUIRE] Replace `<namespace>:<otel-pod-name>` as needed for your OTEL Collector installation.
## Example: `system:serviceaccount:mynamespace:opentelemetry-collector`
- system:serviceaccount:<namespace>:<otel-pod-name>
Run the command oc apply -f scc.yaml
to setup the security context.