SystemMemoryExceedsReservation alerts

Edit online

Resolve repeated SystemMemoryExceedsReservation alerts that occur when system components consume nearly all allocated reserved memory in IBM® Sovereign Core.

Affected versions

Sovereign Core Release: 1.0 and 1.1

Symptoms

The following symptoms indicate this issue:

Repeated SystemMemoryExceedsReservation alerts appear in the cluster
System components consume nearly all allocated systemReserved memory
System memory usage of approximately 1.2–1.4 GiB exceeds 95% of the reserved memory

Resolution

Configure automatic calculation of system reserved resources based on node capacity using a MachineConfig. This configuration can be applied proactively to prevent the issue or reactively to resolve existing alerts.

Create a MachineConfig file with the following configuration:

apiVersion: machineconfiguration.openshift.io/v1
kind: MachineConfig
metadata:
  labels:
    machineconfiguration.openshift.io/role: master
  name: 99-master-dynamic-system-reserved
spec:
  config:
    ignition:
      version: 3.2.0
    systemd:
      units:
      - name: dynamic-system-reserved.service
        enabled: true
        contents: |
          [Unit]
          Description=Calculate Dynamic System Reserved Resources
          Before=kubelet.service
          
          [Service]
          Type=oneshot
          ExecStart=/usr/local/sbin/dynamic-system-reserved-calc.sh true
          RemainAfterExit=yes
          
          [Install]
          WantedBy=multi-user.target

Apply the MachineConfig:

oc apply -f machineconfig-dynamic-reserved.yaml

Monitor the machine config pool status:
```
oc get mcp master -w
```

Verification

After applying the workaround, verify that the configuration has been applied correctly:

Access the node in debug mode:
```
oc debug node/<node-name>
```
Change root to the host:
```
chroot /host
```
Check the kubelet process for system-reserved parameters:
```
ps aux | grep kubelet | grep system-reserved
```
Verify the output shows the system-reserved configuration. Expected output example:
```
--system-reserved=cpu=1000m,memory=11Gi,ephemeral-storage=10Gi
```

References

OCP 4.20 Dynamic System Reserved Calculation Script
SystemMemoryExceedsReservation alert received in OCP 4
Which amount of CPU and memory are recommended to reserve for the system in OpenShift 4 nodes?