IBM i Virtual Fibre Channel Performance Best Practices

Content

This document applies to environments where the IOA is type 6B25 and the disk resource type is one of the following:
2145 - IBM SVC or Storwize Family
2107 - IBM DS8000 Family
D910 - EMC/Dell - Dell has not renewed their licensing agreement and is currently 'unsupported'.  It will likely work, but all troubleshooting/config/etc must be provided by Dell.  See the following link for this announcement - https://public.dhe.ibm.com/systems/support/planning/notices/IBM.i.Dell.Storage.Attach.Client.Notice.pdf
D840 - IBM Flash Systems

There are known issues that that can be reviewed and implemented, as listed below.

1.  Environment limitations. 

• Maximum number of disk paths for VIOS attached 512-byte sector storage using Virtual Fibre Channel (VFC/NPIV)    
  • Approximately 6000
• Maximum number of disk paths for VIOS attached storage using Virtual SCSI (VSCSI)    
  • Approximately 3000

• 512-byte sector disk resource types include:
  • 2145 - IBM SVC or Storwize Family
  • D910 - EMC

2.  Add additional interrupt handlers (VFC client/server adapters) 

The reasons outlined do not apply to 2107 devices.  In summary, the goal is to give each interrupt handler less work to perform - reducing delays experienced on the IBM i server itself.  This can be accomplished by adding more paths (up to 8 active paths) which spreads the work out, reducing the number of LUNs/volumes on each VFC client adapter (6B25 IOA - Dcxx).   At R740 OR R730 TR11 the design has changed to increase the number of tasks (a 3-fold increase) to help reduce long running interrupts.  At R730 TR11 or R740 we may still need to add additional VFC adapters to account for high IO levels.   The best practices outlined below still apply and should be considered when creating or changing this environment.   

To reduce the delays due to long running interrupt handlers, additional Virtual Fibre Channel adapters must be created - providing more interrupt handlers to handle the IO workload.   Conversely - reducing IO pressure would have the same end result.  This can be accomplished via workload tuning. 

• Internal storage and DS8k external storage uses 520-byte blocks by default - eliminating all of the following processing.
• OS/Storage Management tracks/manages this data
• Native Fibre Channel and Virtual Fiber Channel/NPIV both function by using 8/1 user/metadata blocks to address storage.
• Native Fibre Channel drivers are able to manage the user data blocks and metadata blocks together - interfacing directly with the Fibre Channel adapter to gather the appropriate data.
• Virtual Fibre Channel drivers are NOT able to directly interface with the physical Fibre Channel adapters since the physical adapters reside on VIOS and VIOS uses a different structure to organize storage.
  • As as result the IBM i Virtual Fibre Channel 'driver' has to build a buffer that contains user data AND metadata.
  • IBM i has to move data between this buffer and our native storage management data/structure.  This 'move'/processing requires CPU resources and a small portion of time for EACH storage related function.  
  • This work is performed on an interrupt basis.  Large reads are especially problematic (such as we see with large DB reads) - the 'movement' between buffer and native storage management takes time - it's possible that multiple reads are processed per one 'interrupt' causing a buildup of responses from storage (especially with low latency storage).   
  • Once the interrupt handler 'comes up for air' there may be a large group of responses to process leading to similar behavior when processing the responses from storage; this in turn may cause new read requests to be delayed while the responses are processed and quickly building on both sides leading to long IO response times.  

• Tune workloads to reduce volume and size of Disk Reads.  For instance - ensuring that query workloads are properly tuned/supported will help reduce the size and volume of reads (avoiding table/index scans).  This would reduce the overall IO pressure and as a result reduce the frequency of long running interrupts.
• Increasing the number of interrupt handlers will distribute this work - also reducing the impact of long running interrupt handlers.  There are two primary methods to increase the number of interrupt handlers - these config changes are performed on the  storage server/array and VIOS.  Note: the terminology below applies to Spectrum Virtualize and will be different if the storage is a different storage vendor.
  • Increase the number of paths per volume - up to a maximum of 8 total active paths.   No outage is required.
  • Create additional host(s) in Spectrum Virtualize and create new virtual fibre adapters for these hosts.   Spread the existing volumes evenly across all the hosts.   This will reduce the number of volumes per interrupt handler.   Each host would then have the limitation of 8 active paths as mentioned above.   However we are able to increase the total number of interrupt handlers on the IBM i system since we can have up to 8 active paths per Spectrum Virtualize host. This change WILL require an outage so is often not the first choice and is only implemented if an increase in paths does not provide enough relief OR workload is expected to grow significantly. 

• Ensure that all volumes are the same size to prevent unbalanced workloads on an individual volume basis as well as a path basis. 

• Use the mpcpprmode macro to enable low latency CPPR.

  Support is enable via the following PTFs:

  7.3:    MF61110

  7.2:    MF64978

• Other PTFs

  MA45410:  Removes a VPN lock in the LIC VFC interrupt handler that can cause reduced I/O performance.  This does not apply to 2107 device types.  

7.2:  MF61608

7.3:  MF61638

Ensure that the latest fixes/levels are applied to the VIOS servers

*Questions regarding VIOS config/changes will have to be answered by VIOS support. *

For the new levels of VIOS 3.1.2.x, MultiQ has been enabled to increase performance.  Please see the following link for current recommendations.

https://community.ibm.com/community/user/power/blogs/ninad-palsule1/2021/07/26/powervm-npiv-multi-queue-support 

•   Fixing other problems on the VIOS

Ensure the VIOS has adequate CPU resources and is using dedicated processors.  There may be dispatch delays is the VIOS servers do not have a full dedicated core. 

Verify that the physical Fibre Channel ports are not over utilized, etc.   

*Please note: Investigation performed by VIOS/Storage support MUST be during periods of poor disk response time.  Investigating during low/normal response times will not provide meaningful action.