Multi-node OpenIPMI device names in wrong order - Servers

Resolving The Problem

Source

RETAIN tip: H193198

Symptom

The Baseboard Management Controllers (BMC) found in many servers are accessed programmatically at the Linux Kernel level by the OpenIPMI device driver. The OpenIPMI driver creates a "device node" special file /dev/ipmi<N> for each BMC found at driver start-up which will allow the ipmi driver to access each BMC. On single-node (standalone) systems, there is normally only one BMC and only one of these "device node" files will be created /dev/ipmi0. On multi-node systems, however, each node will contain a BMC and a separate device node file will be created for each by the OpenIPMI driver (/dev/ipmi0, /dev/ipmi1, etc.).

An issue was discovered in several Linux distributions in which the device nodes on multi-node systems were named in reverse order of how the BMCs are physically located. In these cases, commands sent to /dev/ipmi0 will actually execute on the BMC physically located last on the system. Commands sent to /dev/ipmi1 will be sent to the second-to-last BMC.

There are multiple ways to determine if this issue is present on a given multi-node Linux system. Using the multi-node addressing capability of the ipmitool v1.8.9 (or later) utility, the MAC and IP addresses of each BMC can be checked to determine if the device nodes are named in the expected order.

Additionally, ipmitool v1.8.9 commands can be sent to each node that will flash the LED front panel identify light, therefore, allowing system-node-to-IPMI-device-name cross-identification.

The ipmitool v1.8.9 utility is available in recent Linux distributions or may be downloaded from the internet.

Instructions on how to download and build ipmitool may be found at the following URL:

• Download latest Linux Open Source ipmitool utility - Servers

Affected configurations

The system may be any of the following IBM servers:

• System x3950 E, type 8874, any model
• System x3950 E, type 8879, any model
• System x3950 M2, type 7141, any model
• System x3950, type 8872, any model
• System x3950, type 8878, any model
• xSeries 460, type 8872, any model
• xSeries MXE-460, type 8874, any model

This tip is not option specific.

The BIOS firmware for the Baseboard Management Controller is affected.

The OpenIPMI device driver for the Baseboard Management Controller is affected.

The system is configured with at least one of the following:

• Red Hat Enterprise Linux 5 Update 1, Update 2
• SUSE Linux Enterprise Server 10, Service Pack 1
• SUSE Linux Enterprise Server 10 for AMD64/EM64T, Service Pack 1
• SUSE Linux Enterprise Desktop 10, Service Pack 1
• SUSE Linux Enterprise Desktop 10 for AMD64/EM64T, Service Pack 1

Note: This does not imply that the network operating system will work under all combinations of hardware and software.

Please see the compatibility page for more information:

• IBM ServerProven

The ipmitool utility is affected.

Solution

This issue is fixed in updates to Red Hat Enterprise Linux 5 and SUSE Linux Enterprise Server 10:

Workaround

The Linux udev rules device naming facility may be used to work around this issue.

A. Determine The Current IPMI Device Node Naming Order

First, determine the order in which the OpenIPMI device nodes are currently named. If they are named in the order in which they physically appear, then the rest of the workaround information in this RETAIN tip should be ignored.

There are two different methods that may be used to determine the current device-name-to-system-node mapping using ipmitool v1.8.9 (or later).

Option 1

The known BMC MAC and BMC IP addresses can be matched up against what is reported for the different nodes by the ipmitool commands.

Repeat for each system node, changing the N in the -d <N> option to the system node number - 1.

Examine the MAC and IP addresses reported for the BMCs using each particular ipmi device node and match it up against the known MAC/IP addresses for each. If these addresses are not known, they may be found in the System BIOS menus at boot time.

If the addresses indicate that the ipmi device names are reversed or out of order, the udev rules file workaround described herein should be implemented. If the ipmi device names do not appear to be reversed or out of order, take no further action because the system does not have the issue described in this RETAIN tip.

Option 2

The second method of determining the current device-name-to-system-node mapping involves briefly turning on the front-panel identify LED of each node in turn and requires physical proximity to the multi-node system. This visual identification method may be accomplished using ipmitool v1.8.9 (or later) with the ipmitool command.

This command will cause the front panel LED of the system node addressed by /dev/ipmi0 to be illuminated for 15 seconds. Observe which node's LED becomes illuminated. Wait until the light turns off or use the ipmitool -d 0 chassis identify 0 command to turn off the light immediately.

This command will cause the front panel LED of the system node addressed by /dev/ipmi1 to be illuminated for 15 seconds. Observe which node's LED becomes illuminated.

Again, wait 15 seconds for the light to turn off automatically or proactively turn off the LED immediately as described earlier.

Repeat, in turn, for each system node, changing the N in the -d <N> option to the system node number - 1 and noting in each case, which system node's LED becomes lit.

If, based on the information gathered, it appears that the ipmi device names are reversed or out of order, the udev rules file workaround should be implemented. If the ipmi device names do *not* appear to be reversed or out of order, that is, if /dev/ipmi0 addresses system node 1, /dev/ipmi1 addresses system node 2 and on, take no further action because the system does not have the issue described in this RETAIN tip.

B. Implement UDEV Rules File Workaround

Udev rules provide a method for persistent naming of devices on a Linux system. The rules are created by matching device information found in sysfs (the /sys directory) against device naming rules written by the user and stored in the udev rules directory /etc/udev/rules.d.

Writing a udev rules file for ipmi devices in Linux involves creating a file in /etc/udev/rules.d specifically for ipmi device naming. A good choice for an ipmi udev rules file name might be /etc/udev/rules.d/10-local-ipmi.rules. Each ipmi device (for example, BMC) on the system will require one line to be added to the ipmi udev rules file to map the ipmi device to a particular ipmi device node name.

The first step in creating an ipmi udev rules file is to make sure the ipmi udev rules file does not already exist. If there is already an ipmi udev rules file on the system, the user may choose to edit it or instead to move it to a *rules.prev file and create a new ipmi udev rules file.

Search for an existing ipmi udev rules file using the following command:

If found, the user should check to see what it is currently doing to determine whether the user should move it aside (to a <filename>.prev file for instance), or add the device naming rules described below.

To create a new udev rules file use the following command:

Add a comment and rules lines to the file:

Add lines similar to the examples below that will support the stated number of nodes for the multi-node system.

Note that the rules lines the user will need to include in the ipmi rules file will be the exact ones listed below for the system size with the exception of the number '253'. This number will be different on different systems. To find out what number the user should replace the '253' with to support the particular system, use the following command:

Change the 253 in the SYSFS dev =="253:0" field on each line in the new ipmi udev rules file to be the number in that same field that is returned by this udevinfo command.

All other text on each line should be identical to the example given.

Note: The user must replace the '253' on each line with the correct number for the system.

Restart the OpenIPMI driver with the following command to enable it to use the new ipmi device node name mappings:

Check that the ipmi device node names are now mapped as defined in the new ipmi udev rules file using one of the ipmitool -d <N> lan print or ipmitool -d <N> chassis identify methods described earlier in this RETAIN tip. The ipmi device nodes should now be named in the same order in which they appear on the system. This new naming will persist across reboots unless the ipmi udev rules file is removed from the system.