Configuring remote logging for a VMware deployment

Logging collection is required for IBM Support to assist with troubleshooting. You can configure fluent-bit to collect and forward logs to a remote Syslog server.

Before you begin

Beginning in version 10.0.5.1, API Connect uses fluent-bit to collect log files for appliance-based deployments using OVA and forward the logs to a remote Syslog server.

In API Connect 10.0.5.2 or later, the fluent-bit service is already installed on the appliances and is configured to forward container logs to /var/log/syslog on the local machine. For best results, upgrade to API Connect 10.0.5.2 or later so you can take advantage of the included fluent-bit service.

Important: This article refers to third-party software that IBM does not control. As such, the software may change and this information might become outdated. In addition, the fluent-bit service was tested only for simple container log routing to local and remote Syslog servers as configured in the instructions that follow. Any customization of other fluent-bit settings (for example, to set up parsers or filters) is at your own risk and should be performed with caution by referring to the fluent-bit documentation.

About this task

Remote logs created with fluent-bit use the following format:

Mar 1 00:38:44 apicdev1147 calico-node-gqbv_42428a8ade169ad3 2023-03-01 00:38:44.684 [INFO][208] felix/int_dataplane.go 1245: Applying dataplane updates
Where:
  • Mar 1 00:38:44 is the time the log arrived on the remote server.
  • apicdev1147 is the hostname of the log source.
  • calico-node-gqbv_42428a8ade169ad3 is a portion of the pod name/container_id where the log is coming from.
  • 2023-03-01 00:38:44.684 is the timestamp at which the logged event actually took place within the container.
  • The rest of the log is the log message itself.

Logs posted to your local server look different, as in this example:

Feb 28 23:47:03 apicdev1147 fluent-bit[2332273]: [2] kube.var.log.containers.kqn2-management-portal-proxy-647b96b89f-pkz5w_default_portal-proxy-74a1723512574fdc23fb9c014b80c7e83aff225b9d44d995736251edd18c49b4.log: [1677628023.503190040, {"_p"=>"F", "log"=>"Tue, 28 Feb 2023 23:47:03 GMT express:router expressInit  : /healthz", "tag"=>"kqn2-management-_74a1723512574fdc", "time"=>"2023-02-28T23:47:03.503190005Z", "stream"=>"stderr"}]

On the local log, the whole fluent-bit JSON object is logged, which provides slightly more information (for example, the full file name of the log file) than in the remote log.

To configure remote logging, use the appropriate instructions for your deployed version of API Connect:

API Connect 10.0.5.2 and later: Configuring remote logging

About this task

The fluent-bit service is already installed on the appliance and is configured to forward container logs to /var/log/syslog on the local machine. If you want to change the output destination to a remote Syslog server, complete the following steps:

Procedure

  1. Create the /etc/fluent-bit/append-tag.lua file.
    1. Paste the following code into the new file: 
      function append_tag(tag, timestamp, record)
    new_record = record
    local pod_name = string.sub(string.match(tag, "^kube%.var%.log%.containers%.(.+)$"), 1, 16)
    local container_id = string.sub(string.match(tag, "-([^-]+)%.log$"), 1, 16)

    new_record["tag"] = pod_name .. "_" .. container_id
    return 1, timestamp, new_record
end
    2. Save and close the file.
  2. Create the /etc/fluent-bit/fluent-bit-override.conf file.
    1. Add the following statements to the new file::

      Replace the variables in the [OUTPUT] section with values for your deployment:

      [SERVICE]
        Daemon Off
        Flush 1
        Log_Level info
        Parsers_File parsers.conf
        HTTP_Server On
        HTTP_Listen 0.0.0.0
        HTTP_Port 2021
        Health_Check On

[INPUT]
        Name tail
        Path /var/log/containers/*.log
        multiline.parser docker, cri
        Tag kube.*
        Mem_Buf_Limit 5MB
        Skip_Long_Lines On

[INPUT]
        Name systemd
        Tag host.*
        Systemd_Filter _SYSTEMD_UNIT=kubelet.service
        Read_From_Tail On

[FILTER]
        Name kubernetes
        Match kube.*
        Merge_Log On
        Keep_Log Off
        K8S-Logging.Parser On
        K8S-Logging.Exclude On

[FILTER]
        Name lua
        Match kube.*
        Script /etc/fluent-bit/append-tag.lua
        call append_tag

[OUTPUT]
        Name syslog
        Match *
        Host <host>
        port 514
        mode udp
        syslog_format rfc5424
        syslog_maxsize 2048
        syslog_hostname_key hostname
        syslog_hostname_preset <hostname of local machine/log source>
        syslog_appname_key tag
        syslog_message_key log
      Note: The default values shown in the [OUTPUT] section can be modified as explained in the fluent-bit documentation.
    2. Save and close the file.
  3. Run the following command to restart the fluent-bit service so it picks up the configuration changes: 
    systemctl restart appliance-manager && systemctl restart fluent-bit

API Connect 10.0.5.2 and later: Update the remote logging configuration

To update the fluent-bit configuration, complete the following steps:

Procedure

  1. Run the following command to edit the configuration file: 
    vim /etc/fluent-bit/fluent-bit-override.conf
  2. Complete your configuration changes, then save and close the file.

  3. Run the following command to restart the appliance-manager: 
    systemctl restart appliance-manager

API Connect 10.0.5.1: Installing and configuring remote logging

About this task

Download Debian packages from IBM Fix Central to install fluent-bit by completing the following steps:

Procedure

  1. If you previously performed a Helm installation of fluent-bit, uninstall it by running the following command:
    helm uninstall fluent-bit
  2. Download the following files from IBM Fix Central:
  3. Install libpq5 by running the following command: 
    apt install <path-to-libpq5_12.13-0ubuntu0.20.04.1_amd64.deb>
  4. Install fluent-bit by running the following command:

    Be sure to install libpq5 first because it is a prerequisite for fluent-bit.

    apt install <path-to-fluent-bit_2.0.6_amd64.deb>
  5. Create the /etc/fluent-bit/append-tag.lua file.
    1. Paste the following code into the new file: 
      function append_tag(tag, timestamp, record)
    new_record = record
    local pod_name = string.sub(string.match(tag, "^kube%.var%.log%.containers%.(.+)$"), 1, 16)
    local container_id = string.sub(string.match(tag, "-([^-]+)%.log$"), 1, 16)

    new_record["tag"] = pod_name .. "_" .. container_id
    return 1, timestamp, new_record
end
    2. Save and close the file.
  6. Configure the input and output settings for the configuration file, so that fluent-bit forwards logs to Syslog:
    1. Run the following command to edit the configuration: 
      vim /etc/fluent-bit/fluent-bit.conf
    2. Replace the contents of the file with the following information:
      • For a local Syslog:
        [SERVICE]
	flush        1
	daemon       Off
	log_level    info
	parsers_file parsers.conf
	plugins_file plugins.conf
	http_server  Off
	http_listen  0.0.0.0
	http_port    2020
	storage.metrics on

[INPUT]
	Name tail
	Path /var/log/containers/*.log
	Parser cri
	Mem_Buf_Limit 5MB
	Skip_Long_Lines On

[OUTPUT]
	name  stdout
	match *
      • For a remote Syslog:

        Replace the variables in the [OUTPUT] section with values for your deployment:

        [SERVICE]
        Daemon Off
        Flush 1
        Log_Level info
        Parsers_File parsers.conf
        HTTP_Server On
        HTTP_Listen 0.0.0.0
        HTTP_Port 2021
        Health_Check On

[INPUT]
        Name tail
        Path /var/log/containers/*.log
        multiline.parser docker, cri
        Tag kube.*
        Mem_Buf_Limit 5MB
        Skip_Long_Lines On

[INPUT]
        Name systemd
        Tag host.*
        Systemd_Filter _SYSTEMD_UNIT=kubelet.service
        Read_From_Tail On

[FILTER]
        Name kubernetes
        Match kube.*
        Merge_Log On
        Keep_Log Off
        K8S-Logging.Parser On
        K8S-Logging.Exclude On

[FILTER]
    Name lua
    Match *
    Script /etc/fluent-bit/append-tag.lua
    call append_tag
 
[OUTPUT]
        Name syslog
        Match *
        Host <host>
        port 514
        mode udp
        syslog_format rfc5424
        syslog_maxsize 2048
        syslog_hostname_key hostname
        syslog_hostname_preset <hostname of local machine/log source>
        syslog_appname_key tag
        syslog_message_key log
      Note: The default values shown in the [OUTPUT] section can be modified as explained in the fluent-bit documentation.
    3. Save and close the file.
  7. Run the following commands to start the fluent-bit service:
    1. Enable fluent-bit to start after a system restart:
      systemctl enable fluent-bit
    2. Start fluent-bit:
      systemctl start fluent-bit

API Connect 10.0.5.1: Update the remote logging configuration

To update the fluent-bit configuration, complete the following steps:

Procedure

  1. Run the following command to edit the configuration file: 
    vim /etc/fluent-bit/fluent-bit.conf
  2. Complete your configuration changes, then save and close the file.

  3. Run the following command to restart the fluent-bit service: 
    systemctl restart fluent-bit