Collecting Linux system logs with OpenTelemetry

You can collect logs from Linux-based operating systems and send them to Instana by using the OpenTelemetry (OTEL) Collector. Instana can receive OTEL logs through different mechanisms. For more information, see Configuring OpenTelemetry data ingestion.

Pre-requisite for Ingesting Logs through OpenTelemetry

To ingest logs into Instana through OpenTelemetry, you need an add-on to the Instana license. Contact your IBM Sales representative to purchase the add-on.

Collecting normal logs

The Filelog Receiver is capable of collecting logs from regular log files.

Download the otelcol-contrib executable binary file version 0.110.0 or later, from the Linux releases according to your Linux server configuration. The OTEL Collector Contrib package includes community-driven open source features that are necessary to collect and send logs to Instana. Configure the OTEL Collector to collect log data and forward it to the Instana agent OTLP Endpoint. To configure the OTEL Collector, use the following example as a reference.

If you want to communicate by using TLS-encrypted methods with the Instana Agent, complete the steps to set up TLS encryption for the agent endpoint.

Configuring the OTEL collector filelog receiver

The following is a complete example of an OTEL Collector configuration. Save this configuration in a otel-config.yaml file and run it using the following command:

  • ./otelcol-contrib --config=./otel-config.yaml
receivers:
  ## [REQUIRED] The filelog receiver will collect logs written to file by a process
  filelog:
    ## [REQUIRED] Path (or regex) to the log files that must be read.
    include: [ "/path/to/log/files/to/read" ]

    ## [OPTIONAL] Path (or regex) to the log files that must be ignored.
    exclude: [ "/path/to/log/files/to/ignore" ]

    ## [REQUIRED] Whether to include the file path in the logs
    include_file_path: true

    ## [OPTIONAL] Whether to include the file name in the logs
    include_file_name: true

    ## [OPTIONAL] Preserve the leading white spaces so that the example 'recombine' operator works as expected.
    preserve_leading_whitespaces: true

    operators:
      ## [OPTIONAL] Example recombine operator config to handle multi-line log messages for stack-traces. Requires `include_file_path: true` above.
      - type: recombine
        combine_field: body
        is_first_entry: body matches "^[^\\s]"
        source_identifier: attributes["log.file.path"]


processors:
  ## [OPTIONAL] This is an example log severity parser that sets the **severity_text** field in the log payload, each runs in-order such that the highest matching severity is set.
  ## Note: If the OpenTelemetry Collector does not set log severity, then the severity is set by Instana when analyzing the log message.
  transform/severity_parse:
    log_statements:
    - context: log
      statements:
      - set(severity_text, "Info") where IsMatch(body.string, ".*INFO.*")
      - set(severity_text, "Warn") where IsMatch(body.string, ".*WARN.*")
      - set(severity_text, "Error") where IsMatch(body.string, ".*ERROR.*")
      - set(severity_text, "Fatal") where IsMatch(body.string, ".*FATAL.*")

  ## Logs must be sent in batches for performance reasons.
  ## Note: No additional `batch` processor configuration is provided since configuration depends on the user scenario.
  batch: {}

exporters:
  ## [REQUIRED] The Instana Agent supports GRPC payloads
  otlp/instanaAgent:
    ## The GRPC port will be 4317 (unless port-forwarding is used to change this).
    ## Note: Be sure to set Instana Agent's OTLP endpoint HOST:PORT combination.
    endpoint: "INSTANA_AGENT_HOST:INSTANA_AGENT_GRPC_PORT"
    ## TLS encryption is disabled in this example.
    tls:
      insecure: true

service:
  pipelines:
    ## Sample logs pipeline using the above configurations.
    logs:
      receivers: [filelog]
      processors: [transform/severity_parse, batch]
      exporters: [otlp/instanaAgent]

Collecting syslog logs

Instana links the received logs with the host machine's information, so use the filelog receiver to locally collect the /var/log/syslog contents.

Collecting journald logs

Unlike regular log files that the filelog receiver can read, journald logs are stored in a binary format. The Journald Receiver can ingest these binary logs. For more information about journald logs, see How To Use Journalctl to View and Manipulate Systemd Logs.

Configuring the OTEL collector journald receiver

To add the journald receiver, provide the location of the journald logs in the configuration. Use the following configuration example. Copy the following configuration example into the otel-config.yaml file and and run it with the following command:

  • ./otelcol-contrib --config=./otel-config.yaml
receivers:
  # [REQUIRED] The journald logs can be located in other locations, depending on the setup (that is, `/run/log/journal` or `/run/journal`).
  journald:
    directory: /var/log/journal

exporters:
  ## [REQUIRED] The Instana Agent supports GRPC payloads
  otlp/instanaAgent:
    ## Be sure to set the appropriate HOST:PORT combination.
    ## Note: The GRPC port will be 4317 (unless port-forwarding is used to change this).
    endpoint: "INSTANA_AGENT_HOST:INSTANA_AGENT_GRPC_PORT"
    ## TLS encryption is disabled in this example.
    tls:
      insecure: true

processors:
  ## [OPTIONAL] This is an example log severity parser that sets the **severity_text** field in the log payload, each runs in-order such that the highest matching severity is set.
  ## Note: If the OpenTelemetry Collector does not set log severity, then the severity is set by Instana when analyzing the log message.
  transform/severity_parse:
    log_statements:
      - context: log
      statements:
        - set(severity_text, "Info") where IsMatch(body.string, ".*INFO.*")
        - set(severity_text, "Warn") where IsMatch(body.string, ".*WARN.*")
        - set(severity_text, "Error") where IsMatch(body.string, ".*ERROR.*")
        - set(severity_text, "Fatal") where IsMatch(body.string, ".*FATAL.*")

  ## [REQUIRED] Logs must be sent in batches for performance reasons.
  ## Note: No additional `batch` processor configuration is provided since configuration depends on the user scenario.
  batch: {}

service:
  pipelines:
    ## [REQUIRED] Sample logs pipeline using the above configurations.
    logs:
      receivers: [journald]
      processors: [transform/severity_parse, batch]
      exporters: [otlp/instanaAgent]