SLES will not boot with intel_iommu=on kernel parameter - IBM Servers

Resolving The Problem

Source

RETAIN tip: H213410

Symptom

If the intel_iommu=on kernel parameter is added in order to utilize TBOOT, KVM PCIe passthrough, or SR-IOV in SUSE Linux Enterprise Server 11 (SLES11) SP3 or SLES 12, then the system will hang on boot showing messages similar to the following:

Additionally, the boot log will show a DMAR fault:

(where TBOOT = Trusted Boot, KVM = Kernel-based Virtual Machine, PCIe = Peripheral Component Interconnect Express, SR-IOV = Single Root I/O Virtualization)

Affected configurations

The system may be any of the following IBM servers:

• Flex System x280 X6 Compute Node, type 4259, any model
• Flex System x280 X6 Compute Node, type 7903 , any model
• Flex System x480 X6 Compute Node, type 4259, any model
• Flex System x480 X6 Compute Node, type 7903 , any model
• Flex System x880 X6 Compute Node, type 4259, any model
• Flex System x880 X6 Compute Node, type 7903 , any model
• System x3650 M4 HD, type 5460, any model
• System x3750 M4, type 8752, any model
• System x3850 X6, type 3837 (4-socket, 3-year warranty), any model
• System x3850 X6, type 3839 (4-socket, 4-year warranty), any model
• System x3950 X6, type 3837 (8-socket, 3-year warranty), any model
• System x3950 X6, type 3839 (8-socket, 4-year warranty), any model

The system is configured with at least one of the following:

• SUSE Linux Enterprise Server 11, Service Pack 3

The system is configured with one or more of the following IBM Options:

• ServeRAID M1215 SAS/SATA Controller for IBM System x, Option part number 46C9114, any model
• ServeRAID M5210 SAS/SATA Controller for IBM System x, Option part number 46C9110, any model
• ServeRAID M5210e SAS/SATA Controller for IBM System x, Option part number 46C9117CTO, any model

The < 24.2.1-0045 firmware for the ServeRAID M1215 and M5200 series adapters is affected.

Note: This does not imply that the network operating system will work under all combinations of hardware and software.

Please see the compatibility page for more information:

Solution

In SLES 11 SP3, the fix for this issue is to update the Unified Extensible Firmware Interface (UEFI) and ServeRAID firmware to the latest available levels and the issue will disappear.

In SLES 12, updating the UEFI and ServeRAID firmware is necessary, but will not fix the issue alone. This behavior will be corrected in a future release of SLES 12 kernel.

UEFI firmware should be updated to the following build versions or later:

ServeRAID firmware on all affected systems should be updated to version 24.2.1-0045 or later.

The target date for this release is scheduled for fourth quarter 2014.

The file is or will be available by selecting the appropriate Product Group, type of System, Product name, Product machine type, and Operating system on IBM Support's Fix Central web page, at the following URL:

Workaround

Either update the firmware/kernel to the latest levels or do not boot with the intel_iommu=on parameter. This includes disabling TBOOT for the time being.

Additional information

In IBM's UEFI firmware, there is a mechanism for creating Reserved Memory Region Reporting (RMRR) entries so that the operating system can reserve memory for various devices. In this case, when a Transportable Memory Module (TMM) is not installed in the system, the ServeRAID controller utilizes system memory as its cache and an RMRR entry in the Direct Memory Access Remapping (DMAR) table is required. In this RMRR entry, a complete PCI path up to the device in question is defined. However, in IBM's UEFI/ServeRAID firmware, the complete path is not defined. Instead, only the end point path is specified, which is not what the Operating System (OS) is expecting.

SUSE has created a fix in SLES 12 that works around this issue while IBM/Lenovo work to provide a fix for their firmware. SUSE's fix is expected to be released at by the end of November 2014.