mmlsqos command

Displays the I/O performance values of a file system, when you enable Quality of Service for I/O operations (QoS) with the mmchqos command.

Synopsis

mmlsqos Device
   [-Y | --fine-stats DisplayRange]
   [--pool {all | Pool}] 
   [--seconds Seconds] 
   [--sum-classes {yes | no}] 
   [--sum-nodes {yes | no}]

Availability

Available on all IBM Spectrum Scale editions.

Description

With the mmlsqos command, you can display the consumption of I/O operations by processes that access designated storage pools. With the mmchqos command, you can regulate I/O access to a specified storage pool by allocating shares of I/O operations to two QoS classes:

maintenance: The default QoS class for some I/O intensive, potentially long-running GPFS commands, such as mmbackup, mmrestore
other: The default QoS class for all other processes.

A third class, misc, is used to count the IOPS that some critical file system processes consume. You cannot assign IOPS to this class, but its count of IOPS is displayed in the output of the mmlsqos command.

Remember the following points:

Allocations persist across unmounting and remounting the file system.
QoS stops applying allocations when you unmount the file system and resumes when you remount it.
When you change allocations or mount the file system, a brief delay due to reconfiguration occurs before QoS starts applying allocations.

For more information about this command, see Setting the Quality of Service for I/O operations (QoS).

When the file system is mounted, the command displays information about the QoS classes of both explicitly named pools and unnamed pools. Unnamed pools are storage pools that you have not specified by name in any mmchqos command. When the file system is unmounted, the command displays information about only the QoS classes of explicitly named pools.

Parameters

Device: The device name of the file system to which the QoS action applies.
-Y: Displays the command output in a parseable format with a colon (:) as a field delimiter. Each column is described by a header.
Note: Fields that have a colon (:) are encoded to prevent confusion. For the set of characters that might be encoded, see the command documentation of mmclidecode. Use the mmclidecode command to decode the field.
--fine-stats DisplayRange: Displays the fine-grained statistics that are currently in memory. The DisplayRange values specify the indexes of the first and last blocks of statistics in the range of blocks that you want to display, such as 0-4. If you specify only one integer, such as 2, the command displays all the blocks in memory starting with that index and going to the end. The last line of the output displays the index of the next block in memory. You can avoid re-displaying statistics by having the next mmlsqos command display statistics beginning at this block index.
Fine-grained statistics are taken at 1-second intervals and contain more information than regular statistics. They are intended to be used as input for programs that analyze and display data, such as the example plotting program at /usr/lpp/mmfs/samples/charts/qosplotfine.pl. For the content of the statistics, see the subtopic later in this topic.
--pool: Displays the I/O performance values for all QoS pools if all is specified, or for the named pool if a pool name is specified. The default is all.
--seconds: Displays the I/O performance values for the previous number of seconds. The valid range of seconds is 1-999. The default value is 60 seconds. The values are displayed for subperiods within the period that you specify. The subperiods might be every 5 seconds over the last 60 seconds, or every 60 seconds over the last 600 seconds. You cannot configure the number or length of subperiods.
--sum-classes: Displays the I/O performance for each QoS class separately if no is specified, or summed across all the QoS classes if yes is specified. The default is no.
--sum-nodes: If yes is specified, displays the I/O performance summed across all the nodes in the cluster. If no is specified, displays the I/O performance for each node separately. The default is yes.

Exit status

0: Successful completion.

Nonzero: A failure occurred.

Security

You must have root authority to run the mmlsqos command.

The node on which you enter the command must be able to execute remote shell commands on any other administration node in the cluster. It must be able to do so without the use of a password and without producing any extraneous messages. For more information, see Requirements for administering a GPFS file system.

Analyzing regular output from mmlsqos

The mmlsqos command always shows the first three lines of output:

QOS config::

Indicates whether QoS is actively regulating I/O consumption (enabled) or is quiescent (disabled).

QOS values::

Displays, for each storage pool that you configured, the name of the storage pool and the IOPS that you assigned to the other class and the maintenance class. In the following example fragment, the command shows that the system storage pool is configured with the value of inf for both QoS classes:

QOS values:: pool=system,other=inf,maintenance/all_local=inf

The qualifier /all_local after maintenance indicates that the maintenance IOPS are applied to all the files systems owned by the cluster. This value is the default for the maintenance class.

QOS status::

Indicates whether QoS is regulating the consumption of IOPS ("throttling") and also whether QoS is recording ("monitoring") the consumption of IOPS of each storage pool.

The following sample output is complete:

# mmlsqos fs --seconds 30
QOS config::     enabled
QOS values::     pool=system,other=inf,maintenance/all_local=inf:pool=fpodata,other=inf,maintenance/all_local=inf
QOS status::     throttling active, monitoring active
=== for pool fpodata
01:31:45  misc iops=11 ioql=0.016539 qsdl=1.2e-06 et=5
=== for pool system
01:31:45  misc iops=8.2 ioql=0.013774 qsdl=2e-06 et=5

The command mmlsqos fs0 --seconds 30 requests a display of I/O performance values for all QoS pools over the previous 30 seconds. Because the parameters --sum_classes and --sum_nodes are missing, the command also requests I/O performance for each storage pool separately and summed across all the nodes of the cluster.

The information that is displayed for the two configured pools, fpodata and system, indicates that IOPS occurred only for processes in the misc class. The meaning of the categories in each line is as follows:

First column: The time when the measurement period ends.
Second column: The QoS class for which the measurement is made.
iops=: The performance of the class in I/O operations per second.
ioql=: The average number of I/O requests in the class that are pending for reasons other than being queued by QoS. This number includes, for example, I/O requests that are waiting for network or storage device servicing.
qsdl=: The average number of I/O requests in the class that are queued by QoS. When the QoS system receives an I/O request from the file system, QoS first finds the class to which the I/O request belongs. It then finds whether the class has any I/O operations available for consumption. If not, then QoS queues the request until more I/O operations become available for the class. The Qsdl value is the average number of I/O requests that are held in this queue.
et=: The interval in seconds during which the measurement was made.

You can calculate the average service time for an I/O operation as (Ioql + Qsdl)/Iops. For a system that is running IO-intensive applications, you can interpret the value (Ioql + Qsdl) as the number of threads in the I/O-intensive applications. This interpretation assumes that each thread spends most of its time in waiting for an I/O operation to complete.

Analyzing fine-grained output from mmlsqos

The following code block shows an example of fine-grained output. Each line displays the sum of the data for all the QoS programs that are running on the specified node. If the --pid-stats parameter is specified in the mmchqos command, then each line displays the data for one QoS program that is running on the specified node.

Time, Class, Node, Iops, TotSctrs, Pool, Pid, RW, SctrI, AvgTm, SsTm, MinTm, MaxTm, AvgQd, SsQd
1480606227,misc,172.20.0.21,285,2360,system,0,R,16,0.000260,0.000197,0.000006,0.005894,0.000001,0.000000
1480606228,misc,172.20.0.21,631,5048,system,0,R,16,0.000038,0.000034,0.000006,0.002463,0.000000,0.000000
1480606239,other,172.20.0.21,13,112,system,26724,R,16,0.000012,0.000000,0.000008,0.000022,0.000001,0.000000
1480606239,other,172.20.0.21,1,512,system,26724,R,512,0.000375,0.000000,0.000375,0.000375,0.000002,0.000000
1480606239,other,172.20.0.21,30,15360,system,26724,W,512,0.000910,0.000004,0.000451,0.002042,0.000001,0.000000
1480606241,misc,172.20.0.21,5,48,system,14680072,W,16,0.000135,0.000000,0.000025,0.000258,0.000000,0.000000
1480606240,other,172.20.0.21,4,32,system,26724,R,16,0.000017,0.000000,0.000009,0.000022,0.000001,0.000000
1480606240,other,172.20.0.21,48,24576,system,26724,W,512,0.000916,0.000006,0.000490,0.002141,0.000001,0.000000
1480606241,other,172.20.0.21,10,80,system,26724,R,16,0.000017,0.000000,0.000007,0.000039,0.000001,0.000000
1480606241,other,172.20.0.21,34,17408,system,26724,W,512,0.000873,0.000007,0.000312,0.001957,0.000001,0.000000
1480606241,misc,172.20.0.21,12,104,system,15597572,W,16,0.000042,0.000000,0.000024,0.000096,0.000000,0.000000
1480606241,misc,172.20.0.21,1,512,system,15597572,W,512,0.000192,0.000000,0.000192,0.000192,0.000000,0.000000
1480606241,misc,172.20.0.21,9,72,system,14680071,W,16,0.000029,0.000000,0.000024,0.000040,0.000000,0.000000
## conti=4

The following list describes the content in each column:

Time: The time when the measurement was made, expressed in seconds since the epoch.

Class

The QoS class of the process.

Node

The IP address of the node on which the process was running.

Iops

The number of IOPS that the process consumed during the sample period.

TotSctrs

The number of sectors for which data was read or written. By default, one sector is 512 bytes.

Note: The number of TotSctrs from one process or one node might not be equal to the real bytes read or written from applications. For cache I/O in IBM Spectrum Scale, if the modified data from one application I/O is less than fgdbRangeSize (by default, it is 4KB), IBM Spectrum Scale writes bytes defined by fgdbRangeSize from pagepool to backend disks. Therefore, if the application updates only a byte of the file, it shows eight sectors from the mmlsqos output when taking the default fgdbRangeSize.

Pool

The memory pool for which the IOPS were used.

Pid

The process ID of the process for which the IOPS were used. When the --pid-stats parameter of the mmchqos command is not specified, this value is always 0.

RW

Whether the process was doing a read operation or a write operation.

SctrI

The number of sectors for which the IOPS were done. One of the following categories is displayed. The exact value of a category depends on the disk block size.

A single sector.
1/32 or less of a full block.
Less than a full block.
A full block.

For example, if the disk block size is 512, the command displays one of the following values: 1, 16, 511, or 512.

AvgTm

The mean time that was required for an I/O operation to be completed.

SsTm

The sum of the squares of differences from the mean value that is displayed for AvgTm. You can use this value to calculate the variance of I/O service times.

MinTm

The minimum time that was required for an I/O operation to be completed.

MaxTm

The maximum time that was required for an I/O operation to be completed.

AvgQd

The mean time for which QoS imposed a delay of the read or write operation.

SsQd

The sum of the squares of differences from the mean value that is displayed for AvgQd.

The last line in the example indicates that the index of the next block to be displayed is 4. You can avoid re-displaying statistics by having the next mmlsqos command display statistics beginning at this block index.

Examples

The following command displays the I/O performance values for all the pools in the file system over the previous 60 seconds. It does so for each QoS class separately and summed across all the nodes in the cluster.
```
mmlsqos fs0 --seconds 60
```
The following command displays the I/O performance values for the named pool over the previous 60 seconds. It does so for each QoS class separately and for each node separately.
```
mmlsqos fs0 --pool pname0 --sum-nodes no
```

Location

/usr/lpp/mmfs/bin