Configuring remote logging for a VMware deployment

Logging collection is required for IBM Support to assist with troubleshooting. You can configure fluent-bit to collect and forward logs to a remote Syslog server.

Before you begin

The fluent-bit service is already installed on the appliances and is configured to forward container logs to /var/log/syslog on the local machine.

Important: This article refers to third-party software that IBM does not control. As such, the software may change and this information might become outdated. In addition, the fluent-bit service was tested only for simple container log routing to local and remote Syslog servers as configured in the instructions that follow. Any customization of other fluent-bit settings (for example, to set up parsers or filters) is at your own risk and should be performed with caution by referring to the fluent-bit documentation.

About this task

Remote logs created with fluent-bit use the following format:

Mar 1 00:38:44 apicdev1147 calico-node-gqbv_42428a8ade169ad3 2023-03-01 00:38:44.684 [INFO][208] felix/int_dataplane.go 1245: Applying dataplane updates
Where:
  • Mar 1 00:38:44 is the time the log arrived on the remote server.
  • apicdev1147 is the hostname of the log source.
  • calico-node-gqbv_42428a8ade169ad3 is a portion of the pod name/container_id where the log is coming from.
  • 2023-03-01 00:38:44.684 is the timestamp at which the logged event actually took place within the container.
  • The rest of the log is the log message itself.

Logs posted to your local server look different, as in this example:

Feb 28 23:47:03 apicdev1147 fluent-bit[2332273]: [2] kube.var.log.containers.kqn2-management-portal-proxy-647b96b89f-pkz5w_default_portal-proxy-74a1723512574fdc23fb9c014b80c7e83aff225b9d44d995736251edd18c49b4.log: [1677628023.503190040, {"_p"=>"F", "log"=>"Tue, 28 Feb 2023 23:47:03 GMT express:router expressInit  : /healthz", "tag"=>"kqn2-management-_74a1723512574fdc", "time"=>"2023-02-28T23:47:03.503190005Z", "stream"=>"stderr"}]

On the local log, the whole fluent-bit JSON object is logged, which provides slightly more information (for example, the full file name of the log file) than in the remote log.

To configure remote logging, use the appropriate instructions for your deployed version of API Connect:

Configuring remote logging

About this task

The fluent-bit service is already installed on the appliance and is configured to forward container logs to /var/log/syslog on the local machine. If you want to change the output destination to a remote Syslog server, complete the following steps:

Procedure

  1. Create the /etc/fluent-bit/append-tag.lua file.
    1. Paste the following code into the new file:
      function append_tag(tag, timestamp, record)
          new_record = record
          local pod_name = string.sub(string.match(tag, "^kube%.var%.log%.containers%.(.+)$"), 1, 16)
          local container_id = string.sub(string.match(tag, "-([^-]+)%.log$"), 1, 16)
      
          new_record["tag"] = pod_name .. "_" .. container_id
          return 1, timestamp, new_record
      end
    2. Save and close the file.
  2. Create the /etc/fluent-bit/fluent-bit-override.conf file.
    1. Add the following statements to the new file::

      Replace the variables in the [OUTPUT] section with values for your deployment:

      [SERVICE]
              Daemon Off
              Flush 1
              Log_Level info
              Parsers_File parsers.conf
              HTTP_Server On
              HTTP_Listen 0.0.0.0
              HTTP_Port 2021
              Health_Check On
      
      [INPUT]
              Name tail
              Path /var/log/containers/*.log
              multiline.parser docker, cri
              Tag kube.*
              Mem_Buf_Limit 5MB
              Skip_Long_Lines On
      
      [INPUT]
              Name systemd
              Tag host.*
              Systemd_Filter _SYSTEMD_UNIT=kubelet.service
              Read_From_Tail On
      
      [FILTER]
              Name kubernetes
              Match kube.*
              Merge_Log On
              Keep_Log Off
              K8S-Logging.Parser On
              K8S-Logging.Exclude On
      
      [FILTER]
              Name lua
              Match kube.*
              Script /etc/fluent-bit/append-tag.lua
              call append_tag
      
      [OUTPUT]
              Name syslog
              Match *
              Host <host>
              port 514
              mode udp
              syslog_format rfc5424
              syslog_maxsize 2048
              syslog_hostname_key hostname
              syslog_hostname_preset <hostname of local machine/log source>
              syslog_appname_key tag
              syslog_message_key log
      Note: The default values shown in the [OUTPUT] section can be modified as explained in the fluent-bit documentation.
    2. Save and close the file.
  3. Run the following command to restart the fluent-bit service so it picks up the configuration changes:
    systemctl restart appliance-manager && systemctl restart fluent-bit

Updating the remote logging configuration

To update the fluent-bit configuration, complete the following steps:

Procedure

  1. Run the following command to edit the configuration file:
    vim /etc/fluent-bit/fluent-bit-override.conf
  2. Complete your configuration changes, then save and close the file.

  3. Run the following command to restart the appliance-manager:
    systemctl restart appliance-manager