Configuring remote logging for a VMware deployment

Logging collection is required for IBM Support to assist with troubleshooting. You can configure fluent-bit to collect and forward logs to a remote Syslog server.

Before you begin

The fluent-bit service is already installed on the appliances and is configured to forward container logs to /var/log/syslog on the local machine.

Important: This article refers to third-party software that IBM does not control. As such, the software may change and this information might become outdated. In addition, the fluent-bit service was tested only for simple container log routing to local and remote Syslog servers as configured in the instructions that follow. Any customization of other fluent-bit settings (for example, to set up parsers or filters) is at your own risk and should be performed with caution by referring to the fluent-bit documentation.

About this task

Remote logs created with fluent-bit use the following format:

Mar 1 00:38:44 apicdev1147 calico-node-gqbv_42428a8ade169ad3 2023-03-01 00:38:44.684 [INFO][208] felix/int_dataplane.go 1245: Applying dataplane updates
Where:
  • Mar 1 00:38:44 is the time the log arrived on the remote server.
  • apicdev1147 is the hostname of the log source.
  • calico-node-gqbv_42428a8ade169ad3 is a portion of the pod name/container_id where the log is coming from.
  • 2023-03-01 00:38:44.684 is the timestamp at which the logged event actually took place within the container.
  • The rest of the log is the log message itself.

Logs posted to your local server look different, as in this example:

Feb 28 23:47:03 apicdev1147 fluent-bit[2332273]: [2] kube.var.log.containers.kqn2-management-portal-proxy-647b96b89f-pkz5w_default_portal-proxy-74a1723512574fdc23fb9c014b80c7e83aff225b9d44d995736251edd18c49b4.log: [1677628023.503190040, {"_p"=>"F", "log"=>"Tue, 28 Feb 2023 23:47:03 GMT express:router expressInit  : /healthz", "tag"=>"kqn2-management-_74a1723512574fdc", "time"=>"2023-02-28T23:47:03.503190005Z", "stream"=>"stderr"}]

On the local log, the whole fluent-bit JSON object is logged, which provides slightly more information (for example, the full file name of the log file) than in the remote log.

To configure remote logging, use the appropriate instructions for your deployed version of API Connect:

Configuring remote logging

About this task

The fluent-bit service is already installed on the appliance and is configured to forward container logs to /var/log/syslog on the local machine. If you want to change the output destination to a remote Syslog server, complete the following steps:

Procedure

  1. Create the /etc/fluent-bit/append-tag.lua file.
    1. Paste the following code into the new file: 
      function append_tag(tag, timestamp, record)
    new_record = record
    local pod_name = string.sub(string.match(tag, "^kube%.var%.log%.containers%.(.+)$"), 1, 16)
    local container_id = string.sub(string.match(tag, "-([^-]+)%.log$"), 1, 16)

    new_record["tag"] = pod_name .. "_" .. container_id
    return 1, timestamp, new_record
end
    2. Save and close the file.
  2. Create the /etc/fluent-bit/fluent-bit-override.conf file.
    1. Add the following statements to the new file::

      Replace the variables in the [OUTPUT] section with values for your deployment:

      [SERVICE]
        Daemon Off
        Flush 1
        Log_Level info
        Parsers_File parsers.conf
        HTTP_Server On
        HTTP_Listen 0.0.0.0
        HTTP_Port 2021
        Health_Check On

[INPUT]
        Name tail
        Path /var/log/containers/*.log
        multiline.parser docker, cri
        Tag kube.*
        Mem_Buf_Limit 5MB
        Skip_Long_Lines On

[INPUT]
        Name systemd
        Tag host.*
        Systemd_Filter _SYSTEMD_UNIT=kubelet.service
        Read_From_Tail On

[FILTER]
        Name kubernetes
        Match kube.*
        Merge_Log On
        Keep_Log Off
        K8S-Logging.Parser On
        K8S-Logging.Exclude On

[FILTER]
        Name lua
        Match kube.*
        Script /etc/fluent-bit/append-tag.lua
        call append_tag

[OUTPUT]
        Name syslog
        Match *
        Host <host>
        port 514
        mode udp
        syslog_format rfc5424
        syslog_maxsize 2048
        syslog_hostname_key hostname
        syslog_hostname_preset <hostname of local machine/log source>
        syslog_appname_key tag
        syslog_message_key log
      Note: The default values shown in the [OUTPUT] section can be modified as explained in the fluent-bit documentation.
    2. Save and close the file.
  3. Run the following command to restart the fluent-bit service so it picks up the configuration changes: 
    systemctl restart appliance-manager && systemctl restart fluent-bit

Updating the remote logging configuration

To update the fluent-bit configuration, complete the following steps:

Procedure

  1. Run the following command to edit the configuration file: 
    vim /etc/fluent-bit/fluent-bit-override.conf
  2. Complete your configuration changes, then save and close the file.

  3. Run the following command to restart the appliance-manager: 
    systemctl restart appliance-manager