Question & Answer
Question
I’ve received questions, similar to the one below, on more than one occasion, so I thought I’d post something about this for future reference. The question is in relation to setting the maximum value for num_cmd_elems for a Fibre Channel adapter on AIX (and VIOS).
“I have a customer that wants to set their num_cmd_ele on the VIO's as 4096, but every time they set it, they still show 3200. Any ideas?”
This is not a new issue and is most likely because their adapter type doesn't support 4096. But AIX allows you to set it to a higher value and then adjusts it back down to the max of 3200 (usually on boot up).
Cause
The following information (from APAR Technote IV88990) summarises the problem and the expected behaviour for AIX.
Error description
In some instances the maximum allowed value of num_cmd_elems
AIX Development found that the num_cmd_elems value supported by the adapter firmware varied somewhat between different adapter models from the same vendor.
The supported value could also vary for other reasons, such as how many ports and PCI functions are configurable on the adapter, changes across firmware updates, etc. The maximum value of 4096 in the PdAt ODM object class is the maximum supported by most of our PCIe2 fibre adapters, but the 'actual' supported value may vary at times depending on what the adapter firmware reports at the time the adapter is configured. In those cases, the 'true' maximum supported value will be set after the user attempts a chdev operation to change num_cmd_elems to 4096.
This is a limitation of the adapter firmware, and not a limitation within the AIX adapter driver. There are critical reasons NOT to indicate a failure for the chdev operation. So, rather than failing the chdev, the driver simply sets the 'true' maximum allowed num_cmd_elems, and the user will see it after reboot or reconfiguration of the adapter.
Local fix
N/A
Problem summary
Documentation does not have information on
Answer
Essentially this is an adapter hardware/firmware resource limitation. It is not a fabric issue. From my understanding this is how some FC adapters parcel out resources in their ASIC between different ports (PCI functions). The only new adapter that reliably allows more than 3200 num_cmd_elems is the 32Gb FC adapter, which allows 6144.
For more information, please refer to the following links:
IV88990: SOME FIBRE ADAPTERS NUM_CMD_ELEMS CANNOT BE SET TO MAX ODM VALUE
http://www-01.ibm.com/support/docview.wss?uid=isg1IV88990
IJ10890: RULES SETTING FOR NUM_CMD_ELEMS ON 16G FC ADAPTER INCORRECT
https://www-01.ibm.com/support/docview.wss?uid=isg1IJ10890
Related Information
Was this topic helpful?
Document Information
Modified date:
19 December 2019
UID
ibm11107759