Collecting Linux system logs with OpenTelemetry
You can collect logs from Linux-based operating systems and send them to Instana by using the OpenTelemetry (OTEL) Collector. Instana can receive OTEL logs through different mechanisms. For more information, see Configuring OpenTelemetry data ingestion.
Collecting normal logs
The Filelog Receiver is capable of collecting logs from regular log files.
Download the executable binary file from the Linux releases according to your Linux server configuration. The OTEL Collector should be configured to gather log data and forward it to the Instana Agent OTLP Endpoint. To configure the OTEL Collector, use the following example as a starting point.
If you want to communicate by using TLS-encrypted methods with the Instana Agent, complete the steps to set up TLS encryption for the agent endpoint.
Configuring the OTEL collector filelog
receiver
receivers:
filelog:
## Path to the log files that must be read.
include: [ "/path/to/log/files/to/read" ]
## Path to the log files that must be ignored.
exclude: [ "/path/to/log/files/to/ignore" ]
include_file_path: true
include_file_name: true
## Preserve the leading white spaces so that the example 'recombine' operator works as expected.
preserve_leading_whitespaces: true
operators:
# Example use of the 'recombine' operator to handle multi-line log statements.
- type: recombine
combine_field: body
is_first_entry: body matches "^[^\\s]"
source_identifier: attributes["log.file.path"]
processors:
## This is a sample severity parser that sets the **severity_text** field in the resource log payload.
## Note: If the OpenTelemetry Collector does not set log severity, then the severity is set by Instana when analyzing the log message.
transform/severity_parse:
log_statements:
- context: log
statements:
- set(severity_text, "Fatal") where IsMatch(body.string, ".*FATAL.*")
- set(severity_text, "Error") where IsMatch(body.string, ".*ERROR.*")
- set(severity_text, "Warn") where IsMatch(body.string, ".*WARN.*")
- set(severity_text, "Info") where IsMatch(body.string, ".*INFO.*")
## Logs must be sent in batches for performance reasons.
## Note: No default configuration is provided for the `batch` processor since configuration depends on the scenario.
batch: {}
exporters:
## The Instana Agent supports GRPC payloads
otlp/instanaAgent:
## Be sure to set the appropriate HOST:PORT combination.
## Note: The GRPC port will be 4317 (unless port-forwarding is used to change this).
endpoint: "INSTANA_AGENT_HOST:INSTANA_AGENT_GRPC_PORT"
## TLS encryption is disabled in this example. To enable TLS encryption, see [Sending OpenTelemetry data to the Instana agent](https://www.ibm.com/docs/en/instana-observability/current?topic=opentelemetry-sending-data-instana-agent#enabling-tls-encryption-for-opentelemetry-ingestion).
tls:
insecure: true
service:
pipelines:
## Sample logs pipeline using the above configurations.
logs:
receivers: [filelog]
processors: [transform/severity_parse, batch]
exporters: [otlp/instanaAgent]
Collecting syslog
logs
Use the filelog
receiver to locally collect the /var/log/syslog
contents, since Instana links the received logs with the host machine's information.
Collecting journald
logs
Unlike regular log files, journald
logs are stored in a binary format, which the Journald Receiver can ingest. For
more information about journald
logs, see How To Use Journalctl to View and Manipulate Systemd Logs.
Configuring the OTEL collector journald
receiver
Adding the journald
receiver requires that you provide the location of the journald
logs as shown in the following sample.
receivers:
journald:
# Note: The journald logs can be located in other locations, depending on the setup (that is, `/run/log/journal` or `/run/journal`).
directory: /var/log/journal
exporters:
## The Instana Agent supports GRPC payloads
otlp/instanaAgent:
## Be sure to set the appropriate HOST:PORT combination.
## Note: The GRPC port will be 4317 (unless port-forwarding is used to change this).
endpoint: "INSTANA_AGENT_HOST:INSTANA_AGENT_GRPC_PORT"
## TLS encryption is disabled in this example. To enable TLS encryption, see [Sending OpenTelemetry data to the Instana agent](https://www.ibm.com/docs/en/instana-observability/current?topic=opentelemetry-sending-data-instana-agent#enabling-tls-encryption-for-opentelemetry-ingestion).
tls:
insecure: true
processors:
## This is a sample severity parser that sets the **severity_text** field in the resource log payload.
## Note: If the OpenTelemetry Collector does not set log severity, then the severity is set by Instana when analyzing the log message.
transform/severity_parse:
log_statements:
- context: log
statements:
- set(severity_text, "Fatal") where IsMatch(body.string, ".*FATAL.*")
- set(severity_text, "Error") where IsMatch(body.string, ".*ERROR.*")
- set(severity_text, "Warn") where IsMatch(body.string, ".*WARN.*")
- set(severity_text, "Info") where IsMatch(body.string, ".*INFO.*")
## Logs must be sent in batches for performance reasons.
## Note: No default configuration is provided for the `batch` processor since configuration depends on the scenario.
batch: {}
service:
pipelines:
logs:
receivers: [journald]
processors: [transform/severity_parse, batch]
exporters: [otlp/instanaAgent]