Performance improvements for IBM AIX FC and FCoE device driver stacks
Multiple I/O queue support
This article describes performance improvements for IBM® AIX® Fibre Channel (FC) / Fibre Channel over Ethernet (FCoE) device drivers stack for 16 Gb FC (Feature Code: EN0A) and10 Gb FCoE (Feature Code: EN0H) HBAs. The AIX FC driver stack includes an initiator mode Small Computer System Interface (SCSI) protocol driver and an adapter driver. The existing AIX FC adapter driver maintains a SCSI I/O queue for submitting all I/O requests to the FC HBA. Scaling issues have been observed with the existing FC stack with extreme I/O transactions per second (IOPS) and small I/O sizes. This is due to I/O serialization through a single I/O queue.
To improve the number of IOPS on smaller block size I/O requests, the multiqueue functionality is introduced with the 16 Gb FC or 10 Gb FCoE adapter driver starting from:
- AIX releases – AIX 7.2 TL01 SP1, AIX 7.1 TL04 SP3 and AIX 6.1 TL09 SP8
- VIOS release – VIOS 18.104.22.168 and VIOS 22.214.171.124
Figure 1 depicts how I/O is parallelized on multiple I/O queues with the improvements discussed in this article.
Figure 1. A traditional FC driver stack versus an improved FC driver stack
The following configuration is used for the performance analysis of random read operations with a block size of 4 KB
- IBM Power® System E870 server with 64 processors at a frequency of 4.350 GHz
- IBM FlashSystem® 900 with eight storage FC ports:
- With FC
- Brocade 16 Gb FC Switch: 2498-B24
- PCIe2 two-port 16 Gb FC adapter (Feature Code: EN0A)
- With FCoE:
- PCIe2 10 Gb four-port FCoE adapter (Feature Code: EN0H)
- Brocade 10 Gb FCoE Switch
- With Native (stand-alone) AIX configuration:
- Operating system: AIX 7.2 TL01 SP1
- Number of processors: 32
- With N_ Port ID Virtualization (NPIV) configuration:
- VIOS release: 126.96.36.199
- NPIV client OS: AIX 7.2 TL01 SP1
- Number of processors on VIOS host: 32
- Number of processors on each NPIV client: 4
To support multiple I/O queue feature, a new Object Data Manager (ODM) attribute,
num_io_queues, is introduced for FC/FCoE devices (fcs) to indicate the number
of I/O queues configured in the FC adapter driver. Each I/O queue is associated with a
hardware work queue in the FC HBA. All the I/O requests issued to a particular hdisk will be
mapped to the same SCSI I/O queue. Each SCSI I/O queue can service multiple hdisks, However,
I/O request to a given hdisk cannot be distributed to multiple SCSI I/O queues.
Example: For a 16 Gb FC HBA, the ODM stanza for
num_io_queues attribute is
# lsdev | grep fcs fcs0 Available 00-00 PCIe2 2-Port 16Gb FC Adapter (df1000e21410f103) fcs1 Available 00-01 PCIe2 2-Port 16Gb FC Adapter (df1000e21410f103) # odmget -q name=fcs0 CuDv CuDv: name = "fcs0" status = 1 chgstatus = 0 ddins = "pci/emfcdd" location = "00-00" parent = "pci0" connwhere = "0" PdDvLn = "adapter/pciex/df1000e21410f10" # # odmget -q uniquetype="adapter/pciex/df1000e21410f10" PdAt | grep -p num_io_queues PdAt: uniquetype = "adapter/pciex/df1000e21410f10" attribute = "num_io_queues" deflt = "8" values = "1-16,1" width = "" type = "R" generic = "DU" rep = "nr" nls_index = 67 #
The value of this attribute can be changed using the
chdev command or the
System Management Interface Tool (SMIT) interface. The possible values are:
# lsattr -l fcs0 -a num_io_queues -R 1...16 (+1)
To enable multiple I/O queues, the HBA's direct memory access (DMA) resources should
be sufficient to distribute I/O requests across multiple queues. The existing ODM attribute,
io_dma, controls the amount of I/O DMA region that the adapter driver
requests while configuring the HBA.
Default ODM attribute values
This section provides the default values for the ODM attributes related to the multiple I/O queue feature.
For AIX 7.2 TL01 SP1 and VIOS 2.2.5 releases
The default value of the
num_io_queues attribute is set to 8 and to have
sufficient DMA resources, the default value of the
io_dma attribute is
increased to 256, starting with AIX 7.2 TL01 SP1 and VIOS 2.2.5 releases.
# lsattr -El fcs0 | grep -e num_io_queues -e io_dma io_dma 256 IO_DMA True num_io_queues 8 Desired number of IO queues True
For AIX 7.1 TL04 SP3, AIX 6.1 TL09 SP8 and VIOS 188.8.131.52 releases
The default value of the
num_io_queues attribute is set to 1 and
io_dma is set to 64 for the AIX 7.1 TL04 SP3, AIX 6.1 TL09 SP8 and VIOS
# lsattr -El fcs0 | grep -e num_io_queues -e io_dma io_dma 64 IO_DMA True num_io_queues 1 Desired number of IO queues True
As mentioned earlier in this article, there should be sufficient DMA resources to enable
support for multiple I/O queues. Therefore, the value of the
should be increased from 64 to 256. In case the user changes only the
num_io_queues' value without increasing the
io_dma value to
256, the adapter instance will be configured with a single SCSI I/O queue and the following
informational error will be logged with the AIX error log.
# errpt | grep fcs0 29FA8C20 0629173616 I O fcs0 Additional FC Adapter Information
Steps for tuning the num_io_queues attributes using the chdev command
You need to perform the following steps to tune the
chdev command for 16 Gb FC HBAs:
- Unconfigure the device instance.
#rmdev -Rl fcs0
- Change the attribute to the required value (say 16).
# chdev -l fcs0 -a num_io_queues=16 fcs0 changed
- Configure the device instance.
# cfgmgr -l fcs0
- Verify if the attribute is set to the required value.
# lsdev | grep fcs0 fcs0 Available 00-00 PCIe2 2-Port 16Gb FC Adapter (df1000e21410f103) # lsattr -El fcs0 | grep num_io_queues num_io_queues 16 Desired number of IO queues True
Performance results – random read operations with a block size of 4 KB
The following results are for AIX native [that is, physical HBAs owned by the logical
partition (LPAR)] case. I/O requests are running in parallel on the FlashSystem 900 storage
targets using the default
shortest_queue algorithm on the hdisk devices.
Figure 2. AIX (native) results for 16 Gb FC HBA
Figure 3. AIX (native) results for 10 Gb FCoE HBA
The following results are for the NPIV (that is, VIOS owning physical HBAs and
num_io_queues tuned on VIOS) case. I/O requests are running in parallel on
all the I/O paths for a given disk on each NPIV client using the default
shortest_queue algorithm on the Flash System 900 storage disks.
Figure 4. NPIV results for 16 Gb FC and 10 Gb FCoE HBA
In native configuration, the number of IOPS for random read operations with a block size of 4KB, for a single FC HBA port case increased by approximately 2.5 times with the improved FC stack, which is a significant improvement. The achieved IOPS count of 390,000 is very close to the line speed for a single FC HBA port.
The IOPS gain for random read operations with a block size of 4 KB in the NPIV configuration is almost equivalent to that of the native configuration when the number of clients is six or more.