NPIV Multiple-Queue support

Learn about the modernization of N_Port ID Virtualization (NPIV) by enabling multiple-queues, which is commonly known as NPIV multiple-queue (MQ).

Currently, Fibre Channel (FC) adapters with high bandwidth, such as 16 GB or 32 GB FC adapters support multiple-queue pairs for storage I/O communication. multiple-queue pairs in the physical FC stack significantly improve the input/output requests per second (IOPS) due to the ability to drive the I/Os in parallel through the FC adapter. The objective of the NPIV multiple-queue is to add similar multiple-queue support to all components such as the client operating system (OS), POWER® Hypervisor (PHYP), and the Virtual I/O Server (VIOS). The NPIV VIOS stack and the PHYP are updated to allow client LPARs to access multiple-queues. The NPIV multiple-queue feature is supported only on an AIX® or an IBM® i logical partitions, and on VIOS Version 3.1.2, or later.

NPIV scaling improvements through multiple-queue provides the following benefits:

  • Efficient utilization of available multiple-queue FC adapters bandwidth when mapped to a single or multiple LPARs.
  • Enable and drive multiple logical units (LUN) level I/O traffic in parallel through FC adapter queues.
  • Storage I/O performance improvement due to increased input/output requests per second (IOPS).

The following figure shows a managed system that is configured to use NPIV multiple-queues:

An illustration of a managed system configured to use NPIV multiple-queues.

Hardware support and requirements to enable the multiple-queue feature for NPIV

Table 1. Multiple-queue for NPIV
Operating system/PFW Supported versions
Hardware POWER9™ processor-based systems
AIX Version 7.2 Technology Level 05, or later
VIOS Version 3.1.2, or later
POWER firmware Version 940, or later
Fibre Channel (FC) adapter Emulex FC 16 or 32 Gb FC adapters or any high-bandwidth Fibre Channel adapters that support multiple-queue feature.
IBM i IBM i 7.4 Technology Refresh 6 and IBM i 7.3 Technology Refresh 12
Linux® Enterprise Server (SUSE, Red Hat®) Not supported

Performance benefits

NPIV multiple-queue enablement provides improved storage I/O performance for different types of workloads.

LPAR mobility in a multiple-queue supported environment

NPIV multiple-queue enablement for all components requires support from the client operating system, hypervisor, and VIOS. During the LPM operation, if either hypervisor or the VIOS of the destination system does not support multiple-queue, multiple-queue is not enabled after the LPM operation.

LPAR mobility in a multiple-queue supported environment section is described based on the following perspectives:

  • The multiple-queue feature is supported only on an AIX or an IBM i logical partitions, and on VIOS Version 3.1.2, or later.
  • LPM from a VIOS perspective, considering the potential implementations of other PowerVM® clients.
  • LPM and multiple-queue from a firmware perspective.
  • Considerations for NPIV configuration and LPM validation
    • During the initial configuration, when you connect the NPIV client to the VIOS, the VIOS reports whether the multiple-queue feature is supported. If the feature is supported, VIOS reports whether it can migrate from an environment where it has established multiple queues to the destination where fewer queues can be established. The VIOS also reports whether it can continue to perform I/O operations in a single-queue environment (systems with VIOS version earlier than 3.1.2).
    • Power® firmware supports the multiple-queue feature through the implementation of a construct called Subordinate Command Response Queues (sub-CRQs). The NPIV sub-CRQ construct is supported on POWER9, or later systems. The sub-CRQ construct is lost if a client is moved from a POWER9 system to an earlier model POWER system, or if a client is moved to systems with older firmware levels than the current system.
    • During the initial configuration, the VIOS provides information about the firmware and adapters so that the NPIV client can determine whether to maintain NPIV sub-CRQ construct that support the multiple-queue feature. During the LPM operation, if the firmware moves the sub-CRQ construct from the source managed system to the destination managed system, the NPIV client can store the queue resources and use it later when the LPM operation is performed on an environment where all the resources are available.
  • LPM scenarios and multiple-queue behavior in an AIX client
    • During the initial configuration of NPIV client, the AIX NPIV client LPAR exchanges capabilities with the VFC host such as multiple-queue, migration, and firmware levels and then performs the configuration. These capabilities are exchanged again during the LPM operation at the destination managed system. The multiple-queue feature is enabled or deprecated based on these capabilities.
    • When the AIX LPAR is migrated from the source system with the NPIV multiple-queue support setup to the destination system with NPIV multiple-queue support setup, the NPIV stack continues to run in the multiple-queue environment:
      • The performance might remain the same until the NPIV client can create the same number of queues and has similar FC adapter bandwidth that is available at the destination system when compared to the source managed system.
      • While exchanging the initial capabilities during the LPM operation, if the VFC host at the destination managed system reports less queues as compared to the number of queues that are configured on the source managed system, the NPIV client configures and continues sending I/O requests through these available queues.
      • The performance might be impacted if either of queues on the source managed system or the destination managed system is less, or if the storage bandwidth at the destination managed system is less when compared to the source managed system.
      • While exchanging the initial capabilities during the LPM operation, if the VFC host at the destination managed system reports more queues, the NPIV client uses the same number of queues when compared to the number of queues that are configured on the source managed system.

        Examples:

        • If the number of queues that are configured at the source managed system is 8 and if the VFC host at the destination managed system reports 4 queues, only 4 queues are configured at the destination managed system. If the same LPAR is migrated back or to another destination system where the VFC host reports 8 queues, the NPIV client is configured with 8 queues.
        • If the number of queues that are configured at the source managed system is 8 and if the VFC host at the destination managed system reports 16 queues, the VFC client continues to run with 8 queues.
    • When the AIX LPAR is migrated from the NPIV multiple-queue environment to a managed system with an older firmware level, the multiple-queue resources are lost and the performance might reduce regardless of the adapters in the destination managed system. The NPIV client does not establish multiple-queue when it is subsequently moved to a system that supports the multiple-queue environment.
    • When the AIX LPAR is migrated to an environment where the VIOS and the firmware support multiple-queue, but the FC adapters, such as 4 or 8 Gb Emulex, do not support multiple-queue, the sub-queue resources are retained by the AIX client. The AIX client can be used if the client is subsequently moved to an environment that supports multiple-queue. Performance issues might occur after migrating from a multiple-queue environment to an environment that does not support multiple-queue.
    • When the AIX LPAR is migrated to an environment where the VIOS is not capable of the multiple-queue feature, the sub-queues are lost and multiple queues are deprecated. The NPIV client runs in a single queue mode (similar to the NPIV setup in AIX 7200-04 or earlier, and VIOS Version 3.1.1, or earlier versions). The NPIV client does not establish multiple queues when it is subsequently moved to a system that supports multiple-queue environment.
    • When a multiple-queue NPIV client partition (AIX 7200-05, or later) is migrated from a POWER8® or POWER7® system to a POWER9 system with multiple-queue setup, the partition continues to operate in the NPIV single-channel mode because after you migrate a partition from a lower processor compatibility mode to a POWER9 system, the partition continues to run in a lower processor compatibility mode of POWER8 or POWER7 Systems. When the partition is booted with native mode on a POWER9 system, the NPIV multiple-queue is enabled during the NPIV configuration as part of the startup process.
      Note: POWER firmware level FW930, or later supports the sub-CRQ construct that is used for multiple-queue enablement. Hence, performing the LPM operation from a multiple-queue aware setup to a system with POWER firmware level FW930, or later and VIOS Version 3.1.2, or later preserves the sub-CRQ construct. Migrating this LPAR back to multiple-queue aware setup enables the multiple-queue feature.
  • LPM scenarios and multiple-queue behavior in an IBM i client
    • During the initial configuration of NPIV client, the IBM i NPIV client exchanges capabilities with the VFC host such as multiple-queue and migration support and configures itself appropriately. These capabilities are exchanged again during the LPM operation at the destination managed system. The multiple-queue feature is enabled or deprecated based on these capabilities.
    • When the IBM i LPAR is migrated from the source system with the NPIV multiple-queue environment to a destination system with the NPIV multiple-queue environment, the client continues to run in the multiple queues enabled environment.
      • The performance of IBM i NPIV client might be impacted if fewer queues are available on the destination managed system, or if the storage bandwidth at the destination managed system is less when compared to the source managed system.
      • While exchanging capabilities during the LPM operation, if the VFC host at the destination managed system reports a different number of queues than are configured on the source managed system, the IBM i NPIV client is reconfigured and continues to send I/O requests through the available queues.

        Examples:

        • If the number of queues that are configured at the source managed system is 8 and if the VFC host at the destination managed system reports 4 queues, only 4 queues are configured at the destination managed system. If the same LPAR is migrated back to managed system or migrated to another destination system where the VFC host reports 8 queues, the NPIV client is configured with 8 queues.
        • If the number of queues that are configured at the source managed system is 8 and if the VFC host at the destination managed system reports 16 queues, the VFC client is reconfigured to run with 16 queues. If the same LPAR is migrated back to managed system or migrated to another destination system where the VFC host reports 8 queues, the NPIV client is reconfigured with 8 queues.
    • When the IBM i LPAR is migrated from the NPIV multiple-queue environment to a managed system with an older firmware level, the multiple-queue resources are lost and the performance might degrade regardless of the adapters in the destination managed system. The NPIV client re-establishes a multiple-queue configuration if it is subsequently moved to a system that supports the multiple-queue environment.
    • When the IBM i LPAR is migrated to an environment where the VIOS and the firmware support the multiple-queue environment but the 8 Gb Multiplex FC adapters do not support the multiple-queue environment, the sub-queues are lost and multiple queues are deprecated. Performance issues might occur after migrating IBM i LPAR from a multiple-queue environment to an environment that does not support multiple-queue configuration. The NPIV client re-establishes a multiple-queue configuration if it is subsequently remapped to a multiple-queue capable adapter or if it is migrated to a system with multiple-queue capable adapters.
    • When the IBM i LPAR is migrated to an environment where the VIOS is not capable of the multiple-queue feature (VIOS Version 3.1.1, or earlier), the sub-queues are lost and multiple queues are deprecated. The NPIV client will re-establish multiple-queues if it is subsequently migrated to a system that supports the multiple-queue environment.
    • When an IBM i multiple-queue capable client partition (7.3 TR 12 or 7.4 TR 6) is migrated from an older system (POWER8 or POWER7 System) to a system that supports the multiple-queue environment, the NPIV client is reconfigured to enable NPIV multiple-queue support on the destination system.

VIOS Tunable Attributes

New VIOS tunable attributes are available in VIOS version 3.1.2, or later as part of the NPIV multiple-queue feature to provide flexibility with the number of FC adapter queues (physical queues) that each VFC host adapter uses. The NPIV multiple-queue feature also provides QoS type features and tunable attributes that are applicable to all the VFC host adapters. The num_per_range attribute can be set at the VIOS partition level and can be overridden at the individual VFC host adapter level.

The NPIV multiple-queue supports a new pseudo device called viosnpiv0. The partition wide tunable attributes are provided by the viosnpiv0 device. The local tunable attributes are provided by the VFC host adapter device. The following tables describe various tunable attributes that can be used for optimal performance:

Table 2. viosnpiv0 device attributes
Attribute Min value Max value Default value Description
num_per_range 4 64 8 A VIOS level tunable attribute. It indicates the number of FC SCSI queues that each VFC host uses.
num_local_cmds 1 64 5 Allows you to trade off memory resources and performance. A higher value might improve performance for fewer I/O workloads. It controls resources that are allocated for each specific queue that is in use by the VFC host adapter.
bufs_per_cmd 1 64 10 Allows you to trade off memory resource and performance. A higher value might improve performance for larger I/O workloads.
Table 3. vfchost attributes
Attribute name Min value Max value Default value Description
num_per_range 4 64 0 If this attribute is set to a nonzero value, it overrides the partition wide num_per_range attribute of the viosnpiv0 device. If the attribute value is 0, this tunable attribute is not in effect.
limit_intr Boolean (true or false) Boolean (true or false) false A local tunable attribute. If this attribute is set to true, it is expected to negatively impact the performance for a particular adapter. It reduces the number of processors and IOPS that are used to service the VFC host adapter. It takes precedence over the num_per_range attribute.
label N/A N/A "" Used to tag a VFC host adapter with a user-defined string identifier. After a successful LPM operation, the VFC host adapter on the destination VIOS will have the same label as the source VIOS.
Note: The attributes that are related to multiple-queue are lost if you are moving from a VIOS that supports multiple-queue to another VIOS that does not support multiple-queue (if the NPIV client is capable of such a mobility operation).

The local limit_intr attribute has the highest precedence. If the limit_intr is set to false, the local attribute num_per_range is effective. When the local num_per_range attribute is not set, the partition wide attribute num_per_range is effective.

The number of queues that a client uses depends on the FC adapter, FW level, and client capabilities, and also on the VIOS level and the tunable attributes of the VFC host adapter. After a successful LPM operation, if the client is using multiple queues, the local attribute num_per_range or the limit_intr attribute of the VFC host adapter is set on the destination managed system that is based on the value that is used at the source managed system.

Table 4. AIX VFC client tunable attributes
Attribute name Min value Max value Default value Description
lg_term_dma 1 MB 16 MB 8 MB Indicates the memory that is required by the virtual driver for its internal data structure. This attribute value can be modified or increased for the environment with large number of NPIV disks.
max_xfer_size 1 MB 16 MB 1 MB

Allows you to set the maximum transfer size for single I/O. This tunable attribute must be modified to suit I/O transfer size in different environments.

For example, Tape drives (sequential I/O) use large block sizes for I/O transfers.
num_cmd_elems 20 2048 1024 Determines the maximum number of active I/O operations at any given point of time.
num_io_queues 1 16 8 Determines the number of I/O queues that are used in the SCSI I/O communication.
label N/A N/A "" User-defined name to identify the adapter.
num_sp_cmd_elem 512 2048 512 Determines the maximum number of special command operations at any given point of time.
Note:
  • The number of queues that the NPIV client uses depends on several factors such as FC adapter, FW level, VIOS level, and tunable attributes of the VFC host adapter. During the initial configuration, the VFC client negotiates the number of queues with the VFC host and configures the minimum value of num_io_queues attribute and the number of queues that are reported by the VFC host.
  • After the initial configuration, the negotiated number is the maximum number of channels that the AIX VFC client can enable. If the VFC host renegotiates more channels after operations (such as remap, VIOS restart, and so on), the number of channels remains the same as the initially negotiated number. However, if the VFC host renegotiates with fewer channels, the AIX VFC client reduces its configured channels to this new lower number.

    For example, if the initial negotiated number of channels between the AIX VFC client and VFC host is 8, and later if the VFC host renegotiates the number of channels as 16, the AIX VFC client continues to run with 8 channels. If the VFC host renegotiates the number of channels as 4 channels, the AIX VFC client adjusts its number of configured channels to 4. However, if the VFC host renegotiates the number of channels as 8 channels, which result in increasing the number of configured channels to 8, the AIX VFC client must be reconfigured to renegotiate the number of channels from the client side.

  • When an I/O Processor (IOP) of the client adapter is reset, the IBM i VFC client reconfigures itself to use all available channels that are reported by the VFC host. In many cases, VIOS operations, which modify the number of available channels, resets an IOP automatically. If an IOP does not reset automatically, the IBM i VFC client continues to use the number of channels that were previously negotiated.

AIX VFC client adapter information

You can view the VFC client adapter information on an AIX client partition.

To display the host information of a VFC client adapter, type the following command, where X is the adapter instance number:
# cat /proc/sys/adapter/fc/fcsX/hostinfo
The host information is displayed in the following format:

VFC client adapter name            : <adapter instance name>
Host partition name (VIOS)         : <VIOS partition name>
VFC host adapter name              : <host adapter instance name>
VFC host adapter location code     : <host adapter instance location code>
FC adapter name on VIOS            : <physical adapter instance name on the VIOS>
FC adapter location code  on VIOS  : <physical adapter instance location code on the  VIOS>
The following example displays the host information of a VFC client adapter fcs0:
# cat /proc/sys/adapter/fc/fcs0/hostinfo

Output


VFC client adapter name            : fcs0
Host partition name (VIOS)         : vios062_v2_242
VFC host adapter name              : vfchost1
VFC host adapter location code     : U9009.41A.13F508W-V3-C15
FC adapter name on VIOS            : fcs2
FC adapter location code  on VIOS  : U78D2.001.WZS001U-P1-C7-T1