Queue depth in iSCSI hosts

In iSCSI networks, the queue depth is the number of I/O operations that can be run in parallel on a device.

Configure your servers to limit the queue depth on all of the paths to the volumes in configurations that contain many servers or volumes.

Where a number of servers in the configuration are idle or do not initiate the calculated quantity of I/O operations, queue depth might not need to be limited.

Queue depth limits in iSCSI networks

The queue depth is the number of I/O operations that can be run in parallel on a device. If you are designing a configuration for an iSCSI network, you must estimate the queue depth for each node to avoid application failures.

If a node reaches the maximum number of queued commands, it returns error codes to the host such as Resource unavailable. Many operating systems cannot recover if the situation persists for more than 15 seconds. This inability to recover can result in one or more servers that present errors to applications and application failures on the servers.

The formula for queue depth calculation considers the following factors:
  • The maximum number of queued commands is calculated on a per-node basis. An I/O group contains two nodes. The system must continue to function when one of the nodes in an I/O group is not available.
  • If a volume is mapped to be seen by more than one server, each of the servers can send commands to it.
  • If a device driver times out of a command, it immediately reissues the command. The commands are stored in the command queue.

After you calculate the queue depth limit, you must apply it. Each operating system has a particular method of limiting the queue depth on a per-volume basis; see your host operating system documentation to change queue depth.

Homogeneous queue depth calculation in iSCSI hosts

Set the homogeneous queue depth to a calculated value in iSCSI hosts.

Set the queue depth for each volume on the servers by using the calculation:

  • q = (n/(v * c)) * 0.75
where
  • n = the allowed command pool per I/O group; this value is fixed at 4000
  • v = number of volumes per server for the particular I/O group under consideration
  • c = number of servers in an I/O group

Therefore, q = (4000/(v * c)) * 0.75.

Example

Consider a system with 20 host servers (c = 20), each with 20 volumes mapped (v = 20). For this configuration, the value of the queue depth is q = (4000/(20 * 20)) * 0.75 = 7.5 (or the lower boundary limit of 7).

Use the minimum value of 2 if the value of derived queue depth exceeds the maximum allowed value on a particular host operating system.

Some common host server configurations are summarized in Table 1, considering 20 volumes mapped to each host server.

Table 1. Common queue depth configurations
Number of iSCSI hosts per I/O group <20 20-40 64
Recommended queue depth Seven (or the maximum number that is allowed by the host OS, whichever is smaller) Three to four (or the max number that is allowed by the host OS, whichever is smaller) Two

Nonhomogeneous queue depth calculation in iSCSI hosts

After you understand certain criteria for nonhomogeneous queues, you can calculate their queue depth in iSCSI hosts.

Nonhomogeneous queues meet one of the following criteria:
  • One or more servers must be allocated more resources so that they can queue more commands.
  • Volumes must not be distributed evenly among the I/O groups in the clustered system.

For each volume, consider each server to which that volume has a mapping in an I/O group. This mapping results in a set of server/volume pairs. If the sum of the server and volume queue depth for all of the pairs is less than 4000, the server does not experience problems due to a full queue.