Logs flooding due to using systemd as cgroup driver

When using systemd cgroups driver in kubelet and docker, message flooding will happen.

Symptoms

• Log messages are flooding as the following status:

    systemd[1]: libcontainer-29519-systemd-test-default-dependencies.scope: Scope has no PIDs. Refusing.
    systemd[1]: libcontainer-29519-systemd-test-default-dependencies.scope: Scope has no PIDs. Refusing.
    systemd[1]: Created slice libcontainer_29519_systemd_test_default.slice.
    kubelet[6547]: W1209 03:04:49.649981    6547 container.go:422] Failed to get RecentStats("/libcontainer_29519_systemd_test_default.slice") while determining the next housekeeping: unable to find data in memory cache
    systemd[1]: Removed slice libcontainer_29519_systemd_test_default.slice.
    systemd[1]: libcontainer-29554-systemd-test-default-dependencies.scope: Scope has no PIDs. Refusing.
    systemd[1]: libcontainer-29554-systemd-test-default-dependencies.scope: Scope has no PIDs. Refusing.
    systemd[1]: Created slice libcontainer_29554_systemd_test_default.slice.
    systemd[1]: Removed slice libcontainer_29554_systemd_test_default.slice.
    systemd[1]: libcontainer-29561-systemd-test-default-dependencies.scope: Scope has no PIDs. Refusing.
    systemd[1]: libcontainer-29561-systemd-test-default-dependencies.scope: Scope has no PIDs. Refusing.
    systemd[1]: Created slice libcontainer_29561_systemd_test_default.slice.
    kubelet[6547]: W1209 03:04:50.591527    6547 container.go:523] Failed to update stats for container "/libcontainer_29561_systemd_test_default.slice": open /sys/fs/cgroup/memory/libcontainer_29561_systemd_test_default.slice/memory.use_hierarchy: no such file or directory, continuing to push stats
  

Causes

• From the community's discussion, This issue caused by an empty cgroup is created by runc for checking systemd capability.

• For more details, please check:

https://github.com/opencontainers/runc/blob/master/libcontainer/cgroups/systemd/apply_systemd.go#L123

Resolving the problem

Because this issue does not affect the container metrics, we can just configure rsyslog to ignore such flooding logs.

The configure file of rsyslog locate in /etc/rsyslog.conf or /etc/rsyslog.d/*.conf file.

In order to ignore the previous error message,

:rawmsg, contains, "libcontainer" ~

should be added before the message collecting command, for example,

*.info;mail.none;authpriv.none;cron.none /var/log/messages .

Then restart rsyslog service, this flooding message issue can be ignored.