IBM Support

System boot ends in Grub Out of Memory (OOM)

Flashes (Alerts)


Abstract

Real mode area (RMA) is a part of RAM which is used by GRUB2 for its operations and to load all the required boot components (Mods, drivers, initrd etc) for successful booting. The size of the RMA is negotiated from GRUB2 as well as from Linux kernel. The current size of the RMA is 512 MB. In most scenarios, the current RMA size is large enough to hold the Linux kernel, initramfs, Partition Firmware (PFW), GRUB2, and working memory for early Linux boot. However, there are some scenarios where GRUB2 memory usage might over run the 512 MB limit, and the logical partition might fail to boot while throwing an out of memory error.

If either the kernel image or the initramfs requires more than 95 MB, then GRUB2 might fail with an out of memory error when it attempts to load them. Generally, the default installed initramfs and vmlinux are well within the limits, but if Firmware Assisted Dump (FADump) is enabled in the system, the default initramfs size might increase and go beyond 95 MB and machine boot might end up in a GRUB memory issue.

A similar out of memory issue in GRUB2 might occur while doing an OS installation via DVD or vDVD. The OS installation via DVD or vDVD require more memory in RMA due to a larger initramfs size. This requirement of more memory in RMA leads to the out of memory issue. In addition, vTPM and Secure Boot when enabled also use memory in the RMA region, which results in higher memory usage in the RMA and in some scenarios, can lead to out of memory in GRUB2.

In the following scenarios the GRUB2 might run out of memory:

1. Secure boot and FADump are enabled together while booting.
2. DVD or vDVD is used for OS installation.
3. vTPM, secure boot, or both are enabled while doing OS installation.

Content

Linux Releases Affected
Red Hat Enterprise Linux (RHEL) 8.6
Red Hat Enterprise Linux 8.7
Red Hat Enterprise Linux 9.0
Red Hat Enterprise Linux 9.5
Red Hat Enterprise Linux 10
IBM Systems Affected
POWER9
Power10
Symptoms
Post the FADump configuration or while doing an OS installation with DVD or vDVD, the system fails to
boot and prints the below error message on the screen:
error: ../../grub-core/kern/mm.c:376:out of memory.
Press any key to continue...
If you press a key to continue, the kernel boots until it fails to mount the root partition and initiates a system reboot.
Workaround
You can use any of the following methods to recover the system or for successful OS installation:
Note: The system is recoverable only if the default initramfs size has increased due to the FADump configuration. If the default initramfs size is greater than 95 MB or it has increased due to any system configuration other than FADump, the system might not recover.
A. Recover by disabling vTPM and secure boot and continue to use FADump:
  1. Disable secure boot in HMC.
  2. Disable vTPM in HMC.
  3. Restart the system from HMC.
  4. Edit the default grub boot entry in the boot screen.
  5. Edit the entry for initramfs to point to .initramfs-$kver.img.default instead of initramfs-$kver.img.
  6. Press Ctrl+x to boot the system.
  7. Revert grub core.elf in boot partition to the non-secure boot one: 
    $ dd if=/boot/grub2/powerpc-ieee1275/core.elf of=/dev/mapper/<PReP-boot-Partition-Name>
  8. Restart the system from HMC.
B. Recover by disabling FADump and switching back to KDump:
  1. Restart system from HMC.
  2. Edit the default grub boot entry in the boot screen.
  3. Edit the entry for initramfs to point to .initramfs-$kver.img.default instead of initramfs-$kver.img.
  4. Remove "fadump=on" boot parameter in grub boot entry.
  5. Press Ctrl+x to boot the system.
  6. Use grubby to remove "fadump=on" parameter and update crashkernel parameter for kdump.
  7. Restart the kdump service.
  8. Reboot the system.
C. To resolve the issue seen while doing OS installation with DVD or vDVD:
  1. Disable vTPM and secure boot.
    1. Power off the system from HMC.
    2. Disable vTPM in HMC.
    3. Disable secure boot in HMC.
    4. Reboot the system from HMC and restart the OS installation.
  2. Increase the LMB size to 1 GB or more (applicable only on Power10 systems).
    1. Login to HMC.
    2. Go to systems.
    3. Choose the machine.
    4. Click on the tab "Processor, Memory, and I/O".
    5. Expand the memory tab.
    6. For the option "Memory Region Size:" select 1 GB or more.
    7. Reboot the machine.
      Note: This option requires a machine reboot as it might impact the LPAR migration. In case of LPAR
      migration, the source and destination machines must have the same LMB size for successful
      migration.
If above steps does not resolves the issue, use network install or any medium other than DVD or vDVD for OS installation.
Fix Outlook
Multiple enhancements are in discussion to ensure the limitation is addressed in such a way that FADump can be used without issues in secure boot mode.
  1. An enhancement to GRUB2 is anticipated that will allocate more memory for the kernel and initramfs. 
    

    https://issues.redhat.com/browse/RHEL-54551
  2. Improve kexec scripts to avoid drastic increment in initramfs size when FADump is enabled. 
    https://bugzilla.redhat.com/show_bug.cgi?id=2139000

[{"Type":"MASTER","Line of Business":{"code":"LOB26","label":"Storage"},"Business Unit":{"code":"BU058","label":"IBM Infrastructure w\/TPS"},"Product":{"code":"SGMV157","label":"IBM Support for Red Hat Enterprise Linux Server"},"ARM Category":[{"code":"a8m0z000000Gnl7AAC","label":"Red Hat Enterprise Linux"}],"Platform":[{"code":"PF025","label":"Platform Independent"}],"Version":"All Versions"}]

Document Information

Modified date:
18 November 2024

UID

ibm16846531

Page Feedback