Collecting Linux system logs with OpenTelemetry

You can collect logs from Linux-based operating systems and send them to Instana by using the OpenTelemetry (OTEL) Collector. Instana can receive OTEL logs through different mechanisms. For more information, see Configuring OpenTelemetry data ingestion.

Collecting normal logs

The Filelog Receiver is capable of collecting logs from regular log files.

Download the executable binary file from the Linux releases according to your Linux server configuration. The OTEL Collector should be configured to gather log data and forward it to the Instana Agent OTLP Endpoint. To configure the OTEL Collector, use the following example as a starting point.

If you want to communicate by using TLS-encrypted methods with the Instana Agent, complete the steps to set up TLS encryption for the agent endpoint.

Configuring the OTEL collector filelog receiver

receivers:
  filelog:
    ## Path to the log files that must be read.
    include: [ "/path/to/log/files/to/read" ]
    ## Path to the log files that must be ignored.
    exclude: [ "/path/to/log/files/to/ignore" ]
    include_file_path: true
    include_file_name: true
    ## Preserve the leading white spaces so that the example 'recombine' operator works as expected.
    preserve_leading_whitespaces: true
    operators:
      # Example use of the 'recombine' operator to handle multi-line log statements.
      - type: recombine
        combine_field: body
        is_first_entry: body matches "^[^\\s]"
        source_identifier: attributes["log.file.path"]

processors:
  ## This is a sample severity parser that sets the **severity_text** field in the resource log payload.
  ## Note: If the OpenTelemetry Collector does not set log severity, then the severity is set by Instana when analyzing the log message.
  transform/severity_parse:
    log_statements:
    - context: log
      statements:
      - set(severity_text, "Fatal") where IsMatch(body.string, ".*FATAL.*")
      - set(severity_text, "Error") where IsMatch(body.string, ".*ERROR.*")
      - set(severity_text, "Warn") where IsMatch(body.string, ".*WARN.*")
      - set(severity_text, "Info") where IsMatch(body.string, ".*INFO.*")

  ## Logs must be sent in batches for performance reasons.
  ## Note: No default configuration is provided for the `batch` processor since configuration depends on the scenario.
  batch: {}

exporters:
  ## The Instana Agent supports GRPC payloads
  otlp/instanaAgent:
    ## Be sure to set the appropriate HOST:PORT combination.
    ## Note: The GRPC port will be 4317 (unless port-forwarding is used to change this).
    endpoint: "INSTANA_AGENT_HOST:INSTANA_AGENT_GRPC_PORT"
    ## TLS encryption is disabled in this example. To enable TLS encryption, see [Sending OpenTelemetry data to the Instana agent](https://www.ibm.com/docs/en/instana-observability/current?topic=opentelemetry-sending-data-instana-agent#enabling-tls-encryption-for-opentelemetry-ingestion).
    tls:
      insecure: true

service:
  pipelines:
    ## Sample logs pipeline using the above configurations.
    logs:
      receivers: [filelog]
      processors: [transform/severity_parse, batch]
      exporters: [otlp/instanaAgent]

Collecting syslog logs

Use the filelog receiver to locally collect the /var/log/syslog contents, since Instana links the received logs with the host machine's information.

Collecting journald logs

Unlike regular log files, journald logs are stored in a binary format, which the Journald Receiver can ingest. For more information about journald logs, see How To Use Journalctl to View and Manipulate Systemd Logs.

Configuring the OTEL collector journald receiver

Adding the journald receiver requires that you provide the location of the journald logs as shown in the following sample.

receivers:
  journald:
    # Note: The journald logs can be located in other locations, depending on the setup (that is, `/run/log/journal` or `/run/journal`).
    directory: /var/log/journal

exporters:
  ## The Instana Agent supports GRPC payloads
  otlp/instanaAgent:
    ## Be sure to set the appropriate HOST:PORT combination.
    ## Note: The GRPC port will be 4317 (unless port-forwarding is used to change this).
    endpoint: "INSTANA_AGENT_HOST:INSTANA_AGENT_GRPC_PORT"
    ## TLS encryption is disabled in this example. To enable TLS encryption, see [Sending OpenTelemetry data to the Instana agent](https://www.ibm.com/docs/en/instana-observability/current?topic=opentelemetry-sending-data-instana-agent#enabling-tls-encryption-for-opentelemetry-ingestion).
    tls:
      insecure: true

processors:
  ## This is a sample severity parser that sets the **severity_text** field in the resource log payload.
  ## Note: If the OpenTelemetry Collector does not set log severity, then the severity is set by Instana when analyzing the log message.
  transform/severity_parse:
    log_statements:
    - context: log
      statements:
      - set(severity_text, "Fatal") where IsMatch(body.string, ".*FATAL.*")
      - set(severity_text, "Error") where IsMatch(body.string, ".*ERROR.*")
      - set(severity_text, "Warn") where IsMatch(body.string, ".*WARN.*")
      - set(severity_text, "Info") where IsMatch(body.string, ".*INFO.*")

  ## Logs must be sent in batches for performance reasons.
  ## Note: No default configuration is provided for the `batch` processor since configuration depends on the scenario.
  batch: {}

service:
  pipelines:
    logs:
      receivers: [journald]
      processors: [transform/severity_parse, batch]
      exporters: [otlp/instanaAgent]