IBM Support

MustGather for Performance Problems with VSCSI Virtual Disks using VIOS

Question & Answer


Question

What do I collect for support if I am diagnosing a performance problem with poor disk response times and the storage is VSCSI virtual disks from a Virtual I/O Server (VIOS)?

Cause

Analyzing Collection Services, Job Watcher, PEX, and so on, leads to diagnosing poor disk response times and the IOA type is a 290A and the disk resource type is one of the following (based on the sector size presented to IBM i) then continue following this document for a data collection plan.

6B22-050: 512 bytes

6B22-055: 520 bytes

6B4E-050: 4096 bytes

6B4E-055: 4160 bytes

Answer

In this configuration, there can be various storages backing the virtual disks presented to IBM i. Everything from internal hard disk or solid-state drives to Fibre Channel connected external storage. This document describes what to collect from IBM i, the VIOS, and for supported external storage devices that can connect to IBM i as VSCSI devices through VIOS.

IBM i

1. Normal Collection Services, Job Watcher, PEX used to diagnose performance problems. This normally is collected during initial PD before it is determined the problem is with the disk subsystem. At a minimum, Collection Services are needed.

Parent Data Collection document for IBM i Performance: http://www-01.ibm.com/support/docview.wss?uid=nas8N1012115

2. PALs and LIC LOGs. This document describes how to collect SYSSNAP, which is one method that can be used to collect PALs (with hex and stats data) and VLOGs. Use this command and reference the document for more information on QMGTOOLS and SYSSNAP.

QMGTOOLS/SYSSNAP OUTPUT(*IFS) LICLOGS(Y) PALS(Y) QHST(Y) STRDATE(MMDDYY)

http://www-01.ibm.com/support/docview.wss?uid=nas8N1010353

3. Advanced Analysis Macros IODD and VIOFR

STRSST
Press the Enter key.
Sign in with a service tool profile and password that has authority to Display/Alter/Dump in SST.
Select Option 1 - Start a service tool, and press the Enter key.
Select Option 4 - Display/Alter/Dump, and press the Enter key.
Option 2 - Dump to printer, and press the Enter key.
Select Option 2 - Licensed Internal Code (LIC) data, and press the Enter key.
Select Option 14 - Advanced analysis, and press the Enter key.
Specify option 1 and type iodd, and press the Enter key
On the options screen, type dumpalldebug and press the Enter key
Press the Enter key until the spool file is created
Press F12 until you are back on the Advanced Analysis screen
Specify option 1 and type VIOFR, press the enter key
On the options screen type -client -detail -fr -engine -login -bus -transporter -crq, press the enter key
Press then Enter key until the spool file is created
Back out of SST using F12

Send the two QPCSMPRT spool files to ECuRep.
HMC-Managed Systems

Collect the virtual I/O topology for HMC-managed systems. Replace <managed server name> with the name of the server as it appears on the HMC GUI.  From an HMC command line:  lshwres -m <managed server name> -r virtualio --rsubtype scsi --level lpar -F topology

VIOS PERFPMR and SNAP

Collect from all VIO Servers being used for Multipath.

Per the VIOS MustGather, first capture VIOS Performance Advisor data using the part command. Then, Collect PERFPMR while the problem is happening, and finally, collect SNAP data.

AIX Support Center Tools:  https://www.ibm.com/support/pages/node/6117130

If HDISKs are backed by external storage:
SAN Switches
The most common switches are Brocade and Cisco. The basic commands are included for each switch provider, but check with your SAN team or switch documentation for further instructions.

Brocade - support save

Cisco - show tech-support

Storage
It is important to collect snaps or a statesave from the storage system during or as soon as possible after the performance problem has cleared. This prevents important data from being wrapped out, which typically can happen within one to two hours. Contact your Storage Administrator, IBM Storage Support, or Third-party vendor for more assistance or details.
 
2107 - IBM DS8000
Collect a statesave.

1. Generate a statesave using DSCLI:
* Available in all Releases.
diagsi -action warmstart <Storage Image>

For example, dscli> diagsi -action warmstart IBM.2107-xxxxxxx
Date/Time: May 5, 2021 6:48:46 PM BRT IBM DSCLI Version: 7.7.5.61 DS: IBM.2107-xxxxxxx
CMUD00023W diagsi: Are you sure you want to perform diagnostic control warmstart? [y/n]:y
CMUD00026I diagsi: Diagnostic control warmstart is successfully submitted. It will take time for the system to complete the request.

-- or --

1. Generate an ODD dump using DSCLI:
* Available on R7.3+ .
diagsi -action odd <Storage Image>

2. After these have been produced, DS8000 hardware support needs to dial in to the box and offload the files generated.
 
2145 - IBM SVC and Storwize

2145 SVC and Storwize Family

http://www-01.ibm.com/support/docview.wss?uid=ssg1S1005029
D840 - IBM Flash Systems
Download the Support Package containing standard logs using the Flash Systems GUI.

Contact Flash Systems Support for details. 


D910 - EMC
Contact EMC Support or documentation for details.

[{"Line of Business":{"code":"LOB57","label":"Power"},"Business Unit":{"code":"BU058","label":"IBM Infrastructure w\/TPS"},"Product":{"code":"SWG60","label":"IBM i"},"ARM Category":[{"code":"a8m0z0000000CKiAAM","label":"Performance->External Storage"}],"Platform":[{"code":"PF012","label":"IBM i"}],"Version":"All Versions","Type":"MASTER"}]

Document Information

Modified date:
14 April 2022

UID

nas8N1022539

Page Feedback