Dead storage paths with CN4022 on VMware with SAN Volume Controller (SVC) - IBM Flex Systems

Resolving The Problem

Source

RETAIN tip: H212748

Symptom

If storage paths are lost for any reason in a configuration with VMware, and IBM Flex System CN4022 Dual-Port 10 Gb Converged Adapter, and the SAN Volume Controller (SVC), the paths might not recover. Paths might also recover and then fail again in about five to 45 minutes.

Dead paths can be located by issuing the following command: esxcfg-mpath -L | grep dead.

Affected configurations

The system can be any of the following IBM servers:

• Flex System x220 Compute Node, type 2585, any model
• Flex System x240 Compute Node, type 7863, any model
• Flex System x440 Compute Node, type 2584, any model

The system is configured with at least one of the following:

• VMware ESXi 5.1, any update
• VMware ESXi 5.5, any model, any update

The system is configured with one or more of the following IBM options:

• IBM Flex System CN4022 Dual-Port 10 Gb Converged Adapter, option part number 88Y5921, any model
  

Note: This does not imply that the network operating system will work under all combinations of hardware and software.

Please see the compatibility page for more information:

Solution

Update CN4022 firmware to 7.12b.4.2e and driver to 2.710.70.v55 or later.

The file is or will be available by selecting the appropriate Product Group, type of System, Product name, Product machine type, and Operating system on IBM Support's Fix Central web page, at the following URL:

     http://www.ibm.com/support/fixcentral/

Workaround

The paths can be recovered by issuing the command esxcli storage core adapter rescan -a. Most installations are deployed with redundant paths, so the chances of taking both paths down are reduced.

Users could run this command periodically, when they expect that they have disrupted a path, or by means of some kind of cron-like job. It could be run as frequently as every five minutes, and then conditionally call rescan when dead paths are located. The objective of this workaround is to recognize dead paths and recover them. That way, if the currently functional path is disrupted, the pseudo-failed paths have already been re-established as available to maintain connectivity.