Setting the Quality of Service for I/O operations (QoS)

QoS limits the effect of I/O-intensive GPFS™ maintenance commands on overall system I/O performance.

With QoS, you can prevent I/O-intensive, long-running GPFS maintenance commands from dominating file system I/O performance and significantly delaying other tasks. Commands like the examples in Figure 1 can generate hundreds or thousands of requests for I/O operations per second. The high demand can greatly slow down normal tasks that are competing for the same I/O resources.
Figure 1. Examples of long-running, IO-intensive GPFS commands
mmrestripefs fsname -N
mmapplypolicy fsname -N all ...

The I/O intensive, potentially long-running GPFS commands are collectively called maintenance commands and are listed in the help topic for the mmchqos command.

With QoS configured, you can assign an instance of a maintenance command to a QoS class that has a lower I/O priority. Although the instance now takes longer to run to completion, normal tasks have greater access to I/O resources and run more quickly.

For more information, see the descriptions of the QoS commands:
Note:
  • QoS requires the file system to be at V4.2.0.0 or later. To check the file system level, enter the following command:
    mmlsfs fileSystemName -V
  • QoS works with asynchronous I/O, memory-mapped I/O, cached I/O, and buffered I/O. However, with direct I/O, QoS counts the IOPS but does not regulate them.

Overview of using QoS

The following steps provide an overview of how to use QoS. In this overview, assume that the file system fs0 contains 5 nodes and has two storage pools: the system storage pool (system) and another storage pool sp1.
  1. Monitor your file system with the mmlsqos command to determine its maximum capacity in I/O operations per second (IOPS). Follow these steps:
    1. Enable QoS without placing any limits on I/O consumption. The following command sets the QoS classes of both storage pools to unlimited:
      Table 1. Set QoS classes to unlimited
      Storage pool QoS class: maintenance QoS class: other
      system unlimited unlimited
      sp1 unlimited unlimited 
      mmchqos fs0 --enable --reset
    2. Run some maintenance commands that drive I/O on all nodes and disks.
    3. Run the mmlsqos command to observe how many IOPS are consumed:
      mmlsqos fs0 --seconds 60
  2. Run the mmchqos command to allocate the available IOPS among the storage pools.
    1. Allocate a smaller share of IOPS to the maintenance class, perhaps 15 percent. For example, if you determined in Step 1 that the maximum is 10,000 IOPS, then you might allocate 1500 IOPS to the maintenance class.
      If there is more than one storage pool, then divide the IOPS among the maintenance classes of the storage pools. In this overview, suppose that you decide to allocate 1000 IOPS to the maintenance class of the system pool and 500 IOPS to the maintenance class of the sp1 storage pool. See the second column of the table below.
      Note: Make sure that the virtual storage Logical Unit Numbers (LUNs) of different storage pools do not map to the same physical devices.

      By default, QoS divides specific allocations of IOPS evenly among the nodes in the file system. In this overview there are 5 nodes. So QoS allocates 200 IOPS to the maintenance class of the system pool and 100 IOPS to the maintenance class of the sp1 storage pool on each node.

      Note: You can also divide IOPS among a list of nodes or among the nodes of a node class. For example, you can use the mmcrnodeclass command to create a class of nodes that do maintenance commands. You can than divide IOPS among the members of the node class by entering a command like the following one:
      mmchqos fs0 --enable -N nodeClass pool=sp2,maintenance=880IOPS,other=unlimited
      If the file system serves remote clusters, you can divide IOPS among the members of a remote cluster by entering a command like the following one:
      mmchqos fs0 --enable -C remoteCluster pool=sp3,maintenance=1000IOPS,other=unlimited
    2. Allocate the remaining IOPS to the other classes. It is a good idea to accomplish this task by setting other to unlimited in each storage class. Then normal tasks can absorb all the IOPS of the system when no maintenance commands are running. See the third column of the following table:
    Table 2. Allocate the available IOPS
    Storage pool QoS class: maintenance QoS class: other
    system 1000 IOPS (200 IOPS per node) unlimited
    sp1 500 IOPS (100 IOPS per node) unlimited
    The command is on one line:
    mmchqos fs0 --enable pool=system,maintenance=1000IOPS,other=unlimited 
 pool=sp1,maintenance=500IOPS,other=unlimited
  3. When you run a maintenance command, QoS by default assigns it to the maintenance class:
    mmdeldisk fs0 nsd12
    All maintenance command instances that are running at the same time and that access the same storage pool compete for the IOPS that you allocated to the maintenance class of that storage pool. If the IOPS limit of the class is exceeded, then QoS queues the extra I/O requests until more IOPS become available.
    To run a maintenance command without I/O restrictions, you can explicitly assign it to the other class:
    mmdeldisk fs0 nsd12 --qos other
  4. You can disable QoS at any time without losing your IOPS allocations:
    mmchqos fs0 --disable
    When you reenable QoS it starts applying the allocations again:
    mmchqos fs0 --enable
  5. You can change the IOPS allocations at any time. The following command is on one line:
    mmchqos fs0 --enable pool=system,maintenance=750IOPS,other=unlimited 
 pool=sp1,maintenance=750IOPS,other=unlimited
    When you change allocations, mount the file system, or reenable QoS, a brief delay due to reconfiguration occurs before QoS starts applying allocations.

  6. To monitor the consumption of IOPS while a maintenance command is running, run the mmlsqos command. The following command displays the statistics for the preceding 60 seconds during which a maintenance command was running:
    mmlsqos fs0 --seconds 60

See also

Parent topic: Managing file systems