IBM Support

System boot ends in Grub Out of Memory (OOM)

Flashes (Alerts)


Abstract

In the Secure boot environment, GRUB2 currently negotiates with PowerVM to allocate a 512 MB memory region, known as the Real Mode Area (RMA), for boot memory. The RMA must be large enough to hold the Linux kernel, initramfs, Partition Firmware (PFW), GRUB2, and working memory for early Linux boot. The region is divided among the boot components and, if any component exceeds its allocation, out-of-memory failures occur.

Currently, PFW is approximately 130 MB, leaving approximately 382 MB for the other boot components. The Linux kernel and GRUB2 each claim one-fourths of the 382 MB area, providing each with approximately 95 MB. If either the kernel image or the initramfs requires more than 95 MB, then GRUB2 might fail with an out-of-memory error when it attempts to load them.

Generally, the default installed initramfs and vmlinux are well within the limits to avoid such issues, but if Firmware Assisted Dump (FADump) is enabled in the system, the default initramfs size might increase and go beyond 95 MB. In that case, every system reboot ends up in a GRUB OOM issue.

To avoid this issue, do not use secure boot and FADump together. Otherwise, you can use one of the fixes available in the Workaround section.

Content

Linux Releases Affected
Red Hat Enterprise Linux (RHEL) 8.6
Red Hat Enterprise Linux 8.7
Red Hat Enterprise Linux 9.0
Red Hat Enterprise Linux 9.1

 
IBM Systems Affected
Power10
Symptoms
Post the FADump configuration, the system fails to boot and prints the below error message on the screen:
error: ../../grub-core/kern/mm.c:376:out of memory.
Press any key to continue...
If you press a key to continue, the kernel boots until it fails to mount the root partition and initiates a system reboot.
Workaround
You can use any of the following methods to recover the system:
Note: The system is recoverable only if the default initramfs size has increased due to the FADump configuration. If the default initramfs size is greater than 95 MB or it has increased due to any system configuration other than FADump, the system might not recover.
A. Recover by disabling vTPM and secure boot and continue to use FADump:
  1. Disable secure boot in HMC.
  2. Disable vTPM in HMC.
  3. Restart the system from HMC.
  4. Edit the default grub boot entry in the boot screen.
  5. Edit the entry for initramfs to point to .initramfs-$kver.img.default instead of initramfs-$kver.img.
  6. Press Ctrl+x to boot the system.
  7. Revert grub core.elf in boot partition to the non-secure boot one:
    $ dd if=/boot/grub2/powerpc-ieee1275/core.elf of=/dev/mapper/<PReP-boot-Partition-Name>
  8. Restart the system from HMC.
B. Recover by disabling FADump and switching back to KDump:
  1. Restart system from HMC.
  2. Edit the default grub boot entry in the boot screen.
  3. Edit the entry for initramfs to point to .initramfs-$kver.img.default instead of initramfs-$kver.img.
  4. Remove "fadump=on" boot parameter in grub boot entry.
  5. Press Ctrl+x to boot the system.
  6. Use grubby to remove "fadump=on" parameter and update crashkernel parameter for kdump.
  7. Restart the kdump service.
  8. Reboot the system.
 
Fix Outlook
Multiple enhancements are in discussion to ensure the limitation is addressed in such a way that FADump can be used without issues in secure boot mode.
  1. An enhancement to GRUB2 is anticipated that will allocate more memory for the kernel and initramfs.
    https://bugzilla.redhat.com/show_bug.cgi?id=2141401
    
  2. Improve kexec scripts to avoid drastic increment in initramfs size when FADump is enabled.
    https://bugzilla.redhat.com/show_bug.cgi?id=2139000

[{"Type":"MASTER","Line of Business":{"code":"LOB26","label":"Storage"},"Business Unit":{"code":"BU058","label":"IBM Infrastructure w\/TPS"},"Product":{"code":"SGMV157","label":"IBM Support for Red Hat Enterprise Linux Server"},"ARM Category":[{"code":"a8m0z000000Gnl7AAC","label":"Red Hat Enterprise Linux"}],"Platform":[{"code":"PF025","label":"Platform Independent"}],"Version":"All Versions"}]

Document Information

Modified date:
12 December 2022

UID

ibm16846531