Configuring remote logging for a VMware deployment

Appliance-based (OVA) deployments use Syslog for collecting logs. Logs must be collected for both running and terminated containers and processes. Logging collection is required for IBM Support to assist with troubleshooting.

About this task

For appliance-based deployments using OVA, rsyslog is used to collect log files.

There are three options for collecting logs:

Set the rsyslog configuration at boot up (this is the best practice in order to preserve settings when rebooting)
Set the rsyslog configuration post-boot up
Gather logs from a running system

For more information about rsyslog, see:https://www.rsyslog.com/doc/v8-stable/configuration/index.html

Procedure

Log collection from the Syslog collectors is accomplished using the rsyslog module included in the cloudinit application. Cloud-init is an open source package used for injecting configurations during boot up. It is included with the appliance-based (OVA) installation. Cloud-init uses a configuration file, for example, config_file.yaml, to configure logging and other customizations. The complete list of configuration customizations that can be made with cloud-init is documented in https://cloudinit.readthedocs.io/en/latest/topics/modules.html. Follow these steps to configure logging at boot up:

Enter the rsyslog configuration in the config_file.yaml file.
1. Following is an example of the options for rsyslog in config_file.yaml:
```
rsyslog:
    remotes:
        syslog_server1: "192.168.1.1:514"
        syslog_server2: "10.0.4.1:601"
   
```
  where:
  - remotes are key pairs specifying the remote Syslog collectors from which you want to collect logs. Each key is the name for an rsyslog remote entry. Each value holds the remote IP address and port for the Syslog collector.
  - syslog_serverX are Syslog collectors (514 = TCP, 601=UDP)
  Note:
  A number of Appliance containers such as the kube-apiserver log by default to stderr, which is interpreted by rsyslog as a high severity message. To avoid having non-error log entries be presented as errors in rsyslog, it is recommend to configure a format that defaults the severity of the message to info level, and rely on further parsing of the log messages to highlight higher severity messages. Example configuration:
```
rsyslog:
    configs:
    - "*.*  @192.168.1.187;myformat"
    - filename: 00-myformat.conf
      content: |
        template(name="myformat" type="string"
        string="<%$.myprio%> %TIMESTAMP:::date-rfc3339% %HOSTNAME% %syslogtag% %msg:::sp-if-no-1st-sp%%msg%")

        set $.myseverity = 6;
        set $.myprio = 20*8+$.myseverity;
```
  where the logs would be forwarded to the server 192.168.1.187 using the format myformat. Further scanning of the log lines could be defined in the logging infrastructure to highlight the higher severity log entries by, for example, scanning for occurrences of Error/Err/ERROR.
2. Enter the following command to configure log collection at boot up before starting installation:
```
apicup subsys set mgmt additional-cloud-init-file config_file.yaml
```
  Refer to the rsyslog documentation for more information. See https://www.rsyslog.com/doc/v8-stable/configuration/index.html
To collect logs post boot up, follow these steps:
1. ssh into each VM and change directories to etc/rsyslog.d.
2. Create a new .conf file, for example, <new>-cloud-config.conf.
3. Restart rsyslog with the following command: systemctl restart rsyslog
To gather logs from a running system, enter the following command:
```
sudo apic logs
```
The apic logs command collects the following information for the current VM:
- Logs for all containers (including terminated ones)
- Everything from journalctl
- All system service logs
The information is stored locally on the current VM. Entries are culled based upon age for a maximum size of 2 GB (compressed).