Memory Consumption in GRUB

Analyze RMA consumption

RMA usage can be monitored during system startup by capturing the Available RMA and Available Heap values at key stages.

To track the RMA consumption during boot, the Available RMA and Available Heap are calculated at each of the following stages:

• Before and after heap initialization.
• Before and after verifying the integrity of the vmlinuz.
• Before and after loading the vmlinuz.
• Before and after loading the initramfs.
• Before GRUB hands control over to Linux.

Computing available RMA

The grub_machine_mmap_iterate function helps in computing the Available RMA. This function reads the firmware-provided memory map, iterates through the available memory regions (/memory/available) and calls the provided callback function for each of these regions. Instead of calling a callback function, the code can be tweaked to accumulate the available memory across all regions to compute the total Available RMA.

grub_uint64_t
get_available_rma (void)
{
  grub_ieee1275_phandle_t root;
  grub_ieee1275_phandle_t memory;
  grub_uint32_t available[128];
  grub_ssize_t available_size;
  grub_uint32_t address_cells = 1;
  grub_uint32_t size_cells = 1;
  int i;
  grub_uint64_t total_size;

  /* Determine the format of each entry in `available'.  */
  grub_ieee1275_finddevice ("/", &root);
.
.
.
  /* Decode each entry and calculate total available RMA.  */
  i = 0;
  total_size = 0;
  available_size /= sizeof (grub_uint32_t);
  while (i < available_size)
    {
      grub_uint64_t address;
      grub_uint64_t size;

      address = available[i++];
      if (address_cells == 2)
       address = (address << 32) | available[i++];

      size = available[i++];
      if (size_cells == 2)
       size = (size << 32) | available[i++];

      total_size += size;
    }

  if(total_size > 0)
         return total_size;

  return grub_errno;
}

Computing available heap

The grub_mm_dump function helps in computing the Available Heap. This function displays information about all the memory regions in the heap, covering both allocated and free blocks of memory by iterating over individual memory blocks within each region. The grub_mm_base is a pointer to the head of the linked list linking these memory regions. Free blocks are detected, and their sizes are summed to determine the heap space available within a given region.

void
display_heap_stats (void)
{
  grub_mm_region_t r;
  grub_size_t total_heap_F = 0;
.
.
.
  for (r = grub_mm_base; r; r = r->next)
    {
      grub_mm_header_t p;
      grub_size_t heap_F = 0;

      grub_dprintf ("benchmark_rma", "Region %p\n\n", r);
      for (p = (grub_mm_header_t) ALIGN_UP ((grub_addr_t) (r + 1),
                                            GRUB_MM_ALIGN);
           (grub_addr_t) p < (grub_addr_t) (r+1) + r->size;
           p++)
        {
          switch (p->magic)
            {
            case GRUB_MM_FREE_MAGIC:
	      grub_size_t current_free_block = (p->size << GRUB_MM_ALIGN_LOG2);
	      heap_F += current_free_block;
.
.
.
            }
        }
        total_heap_F += heap_F;
        grub_dprintf ("benchmark_rma", "Free Heap region %p: %" PRIuGRUB_SIZE " bytes (%" PRIuGRUB_SIZE " MB)\n", r, heap_F, heap_F >> 20);
    }

The computation and display of Available RMA and the Available Heap are done at the following points within the GRUB execution flow:

• grub_ieee1275_mm_add_region
• grub_claim_heap
• grub_mm_init_region
• grub_memalign
• grub_linux_boot
• grub_linux_load64
• grub_cmd_linux
• grub_cmd_initrd

System configuration

The RMA consumption data is captured during the normal startup sequence for Linux kernel v5.14 on RHEL 9.6. The size of the vmlinuz is 47 MB and the size of the initramfs is 59 MB.

Out of memory scenarios

On a PowerVM LPAR, GRUB might run out of memory while loading the initramfs image. The following scenarios outline specific conditions that trigger this issue:

RHEL DVD installations

The out of memory issue affects RHEL DVD installations from version 9.4 through version 9.6, where the version of GRUB being used was 2.06. While investigating this issue, an observation revealed the circumstances under which GRUB exhausts available memory during boot.

A dual reservation of memory is observed for the vmlinuz and initramfs images, once in the heap and once in the available RMA. In regular scenarios, GRUB reserves memory for both vmlinuz and initramfs outside the heap, within the available RMA. GRUB then loads these images into the reserved space. However, in affected scenarios, GRUB additionally reserves memory for the same images inside the heap, before making reservations in the available RMA, resulting in duplicate memory reservations.

This dual reservation occurs even when Secure Boot is disabled or the Trusted Platform Module (TPM) device is inactive, conditions during which GRUB should reserve memory only once. As a result, the heap depletes prematurely, leading to an out of memory error.

A comparison between DVD-based installations and standard startup sequences highlights a key difference: the DVD-based installation flow registers the TPM verifier, while the standard boot path does not register the TPM verifier. The additional component appears to trigger the redundant memory reservation.

Analyzing the TPM verifier behavior

The TPM verifier reads files such as vmlinuz and initramfs into heap memory, measures them, and uses the verified content as a backing buffer for file access. Before GRUB version 2.12, loading the TPM module also results in registering the TPM verifier even when no TPM is present or when the TPM is disabled. This behavior causes unnecessary heap memory allocation and triggers heap expansion to accommodate the measurement process.

Because the system lacks access to the TPM in these cases, GRUB discards the measurements. However, the heap still expands, consuming space from the available RMA. This unnecessary memory usage contributes to boot-time memory exhaustion.

In GRUB 2.12 and later, a fix has been included to prevent the TPM verifier from registering when the TPM is absent or disabled. This change eliminates redundant memory reservations during the loading of vmlinuz and initramfs. As a result, RHEL 10, which includes GRUB 2.12, does not encounter the out of memory issue during DVD-based installations on PowerVM LPARs.

Impact of initramfs size growth

Another key factor contributing to the out of memory issue during DVD installation is the increase in the size of the initramfs image, from 88 MB in RHEL 9.3 to 133 MB in RHEL 9.4. Although dual memory reservation still occurs in RHEL 9.3, the smaller initramfs size does not trigger memory exhaustion. The larger image in RHEL 9.4, combined with redundant reservations, pushes the system beyond the available memory limits.

More RMA usage with secure boot and trusted boot

Enabling Secure Boot or Trusted Boot significantly increases GRUB's RMA consumption. GRUB allocates heap memory to hash the vmlinuz and initramfs images for integrity verification. Although GRUB frees this memory after verification, the heap might become fragmented.

Fragmentation reduces the availability of large, contiguous memory blocks that are required for subsequent operations, such as further hashing or final image loading. When both Secure Boot and Trusted Boot are enabled, GRUB might read the same images up to three times: for integrity verification; for boot-time measurements; and for actual loading into memory.

Due to fragmentation, GRUB might fail to allocate a sufficiently large contiguous block for the second or third read, increasing the risk of memory exhaustion.

Mitigation through increased RMA reservation

A fix has been included in GRUB to increase the RMA reservation for PowerVM LPARs from 512 MB to 768 MB. This change helps ensure that more real memory is available during startup, reducing the risk of memory exhaustion.

The updated reservation accommodates the increased memory requirement that is caused by larger initramfs images and complex boot scenarios involving Secure Boot, TPM, FADump, and DVD-based installations. The master branch of GRUB includes this fix, and is scheduled for availability in the next major release.

Red Hat and SUSE have already integrated this fix into their current releases, which includes RHEL 9.6, RHEL 10, SLES 15 SP7, and SLES 16, effectively resolving the out of memory issue in these environments.