LI82403: DISK FILL UP DUE TO SYSLOG.1 GROWING LARGE

APAR status

• Closed as program error.

Error description

• Customer's disk fill up 100% due to syslog file rotation not
  working properly and growing the syslog.1 file excessively large
  :
  
  root@ya6836:~# df -h
  
  Filesystem         Size Used Avail Use% Mounted on
  
  udev             16G   0  16G  0% /dev
  
  tmpfs            3.2G 1.1G 2.2G 32% /run
  
  /dev/sda1          97G  97G   0 100% /
  
  tmpfs            16G   0  16G  0% /dev/shm
  
  tmpfs            5.0M   0 5.0M  0% /run/lock
  
  tmpfs            16G   0  16G  0% /sys/fs/cgroup
  
  /dev/loop0          70M  70M   0 100% /snap/lxd/19188
  
  /dev/loop1          33M  33M   0 100% /snap/snapd/11107
  
  /dev/sda15         105M 7.9M  97M  8% /boot/efi
  
  /dev/loop2          56M  56M   0 100% /snap/core18/1988
  
  tmpfs            16G 1.4M  16G  1% /var/lib/cloud/instances
  
  /dev/mapper/apicSecureDisk 492G  23G 444G  5% /data/secure
  
  overlay           492G  23G 444G  5% /var/lib/docker/overlay2/4
  e02dfc97fc50add42255dec4bab0bf07a0ce9eee80de0e5c0b335ad7a66ab43
  /merged
  
  shm             64M   0  64M  0% /var/lib/docker/containers/507
  175aa6ef700ec52a8a5d75817846f541998edcc1910297932b1c1d13e62df/m
  ounts/shm
  
  overlay           492G  23G 444G  5% /var/lib/docker/overlay2/8
  4f4c363fedc5499e17ec4fb174ad1c131148e501e9ee8a82a6a566e55a57161
  /merged
  
  overlay           492G  23G 444G  5% /var/lib/docker/overlay2/8
  82ec20f50e1090d903ee1fffb83123603edfabb1b95e4b37d03bfc6be584676
  /merged
  
  shm             64M   0  64M  0% /var/lib/docker/containers/fb3
  433eafe9ec273c7d853389c894c8a2d74e581f6a38b77716ad9884136672a/m
  ounts/shm
  
  overlay           492G  23G 444G  5% /var/lib/docker/overlay2/7
  c2d04ddc2a48e481f2bc20a8c086d46f6fd00853e17fa20366d767b74640ea0
  /merged
  
  tmpfs            3.2G   0 3.2G  0% /run/user/1000
  
  
  
  This causes the appliance manager to run properly and all the
  pods keeps failing to come up.
  
  root@ya6836:/var/log# du -csh * | grep G
  
  1.1G    journal
  
  87G     syslog.1
  
  89G     total
  
  
  
  The syslog.1 contains the following messages :
  
  Sep 30 20:26:35 ya6836 kubelet[1513]: E0930 20:26:35.549963
  1513 pod_workers.go:190] "Error syncing pod, skipping"
  err="failed to \"StartContainer\" for \"calico-node\" with
  CrashLoopBackOff: \"back-off 5m0s restarting failed
  container=calico-node pod=calico-node-kdts2_kube-system(7bb71a2
  6-2e80-4c58-94c1-75401aa26d80)\""pod="kube-system/calico-node-kd
  podUID=7bb71a26-2e80-4c58-94c1-75401aa26d80
  
  Sep 30 20:26:35 ya6836 kernel: [7749577.868133]
  systemd-journald[467]: Failed to open system journal: No space
  left on device
  
  Sep 30 20:26:35 ya6836 kernel: [7749577.868173]
  systemd-journald[467]: Failed to open system journal: No space
  left on device
  
  Sep 30 20:26:35 ya6836 kernel: [7749577.868510]
  systemd-journald[467]: Failed to open system journal: No space
  left on device
  
  Sep 30 20:26:35 ya6836 kernel: [7749577.868545]
  systemd-journald[467]: Failed to open system journal: No space
  left on device
  
  Sep 30 20:26:35 ya6836 1063647e1c14[1345]: [1] 2021/09/30
  20:26:35.633295 [ERR] Error
  

Local fix

Problem summary

• <span style="background-color:rgb(255, 255, 255);color:rgb(36,
  41, 46);font-size:14px;font-family:-apple-system,system-ui,segoe
  ui,helvetica,arial,sans-serif,apple color emoji,segoe ui
  emoji">Customer's disk could fill up due to syslog file rotation
  not working properly</span>
  

Problem conclusion

• <span style="color:rgb(34, 34,
  34);font-size:14px;font-family:lato,helvetica
  neue,helvetica,arial,sans-serif">The fix is targeted for
  inclusion in IBM API Connect v10.0.1.6 and v10.0.</span><span
  style="color:rgb(34, 34,
  34);font-size:14px;font-family:lato,helvetica
  neue,helvetica,arial,sans-serif">4</span>
  

Temporary fix

Comments

APAR Information

• APAR is sysrouted FROM one or more of the following:

• APAR is sysrouted TO one or more of the following:

Fix information

• Fixed component name

  API CONNECT ENT

• Fixed component ID

  5725Z2201

Applicable component levels