IBM Support

Must Gather for Performance Problems with NPIV Connected Storage (Virtual Fibre Channel)

Question & Answer


Question

What type of data do I collect for diagnosing a performance problem with poor disk response times and the storage is connected with NPIV (Virtual Fibre Channel)?

Cause

Use this document if IBM i performance data confirms jobs are slowed down by poor disk response times, IOA type is 6B25 and the disk resource type is one of the following:

2145 - IBM SVC or Storwize Family

2107 - IBM DS8000 Family

D910 - EMC

D840 - IBM Flash Systems

Answer

In this configuration, volumes on the external storage are presented to the IBM i host system that uses N_Port ID Virtualization (NPIV). NPIV (also called Virtual Fibre Channel) allows multiple virtual servers to share a single physical fibre channel port (N_Port). IBM Power Systems implementation of NPIV requires at least one VIO Server (VIOS). Typically, more than one VIOS is used for redundancy. NPIV also requires the use of a SAN Switch in between the physical port on the VIOS and the physical port on the external storage system. Therefore, many layers exist in this configuration that can add latency felt at the application layer on IBM i. This document explains what to collect from the IO Device Driver (IODD) layer in IBM i out to the storage. Layers above the IODD are typically diagnosed by using wait bucket analysis available in Collection Services or Job Watcher.

IBM i
 

1. Normal Collection Services, Job Watcher, PEX used to diagnose performance problems. At a minimum, Collection Services data is needed.

2. PALs and LIC LOGs. This document describes how to collect SYSSNAP, which is one method that can be used to collect PALs (with hex and stats data) and VLOGs. Use this command and reference the document for more information on QMGTOOLS and SYSSNAP.

QMGTOOLS/SYSSNAP OUTPUT(*IFS) LICLOGS(Y) PALS(Y) QHST(Y) STRDATE(MMDDYY)

http://www-01.ibm.com/support/docview.wss?uid=nas8N1010353

3. Advanced Analysis Macros IODD and VIOFR

STRSST
Press the Enter key.
Sign in with a service tool profile and password that has authority to Display/Alter/Dump in SST.
Select Option 1 - Start a service tool, and press the Enter key.
Select Option 4 - Display/Alter/Dump, and press the Enter key.
Option 2 - Dump to printer, and press the Enter key.
Select Option 2 - Licensed Internal Code (LIC) data, and press the Enter key.
Select Option 14 - Advanced analysis, and press the Enter key.
Specify option 1 and type iodd, and press the Enter key
On the options screen, type dumpalldebug and press the Enter key
Press the Enter key until the spool file is created
Press F12 until you are back on the Advanced Analysis screen
Specify option 1 and type VIOFR, press the enter key
On the options screen type -vfc -detail -fr -ports -login -luns -bus -nexus -transporter -crq, press the enter key
Press then Enter key until the spool file is created
Back out of SST by selecting option F12

Send the two QPCSMPRT spool files to ECuRep.
VIOS PERFPMR and SNAP

Collect from all VIO Servers being used for Multipath.

First capture VIOS Performance Advisor data using the part command. Then, Collect PERFPMR while the problem is happening, lastly, collect SNAP data.

AIX Support Center Tools:  https://www.ibm.com/support/pages/node/6117130

SAN Switches
 

The most common switches are Brocade and Cisco. The basic commands are included, but check with your SAN team or switch documentation if assistance is required.

Brocade - supportsave

Cisco - show tech-support

Storage
It is important to collect snaps or state saves from the storage system during or as soon as possible after the performance problem disappears. This will prevent important data from being wrapped out which typically can happen within one to two hours. Contact your Storage Administrator, IBM Storage Support, or Third-party vendor for additional assistance or details.
 
2107 - IBM DS8000 Family
Collect a state save.

1. Generate a Statesave using DSCLI:
* Available in all Releases.

diagsi -action warmstart <Storage Image>

For example,  dscli> diagsi -action warmstart IBM.2107-xxxxxxx
Date/Time: May 5, 2015 6:48:46 PM BRT IBM DSCLI Version: 7.7.5.61 DS: IBM.2107-xxxxxxx
CMUD00023W diagsi: Are you sure you want to perform diagnostic control warmstart? [y/n]:y
CMUD00026I diagsi: Diagnostic control warmstart is successfully submitted. It will take some time for the system to complete the request.

-- or --

1. Generate an ODD Dump using DSCLI:
* Available on R7.3+ .

diagsi -action odd <Storage Image>

2. After these have been produced, DS8000 hardware support needs to dial in to the box and offload the files generated.
 
2145 - IBM SVC and Storwize Family

http://www-01.ibm.com/support/docview.wss?uid=ssg1S1005029
D840 - IBM Flash Systems
Download the Support Package containing standard logs using the Flash Systems GUI.
Contact Flash Systems Support if there are questions. 
 
D910 - EMC
Contact EMC Support or documentation for details.

[{"Line of Business":{"code":"LOB57","label":"Power"},"Business Unit":{"code":"BU058","label":"IBM Infrastructure w\/TPS"},"Product":{"code":"SWG60","label":"IBM i"},"ARM Category":[{"code":"a8m0z0000000CKiAAM","label":"Performance->External Storage"}],"ARM Case Number":"","Platform":[{"code":"PF012","label":"IBM i"}],"Version":"All Version(s)"}]

Document Information

Modified date:
14 April 2022

UID

nas8N1022401

Page Feedback