IBM Support

IBM MQ: Considerations for EFS on AWS

Troubleshooting


Problem

IBM MQ provides guidance on the functional capabilities for a file system and the testing a user should complete in their own environment. If a file system has been verified by the lab, or known issues identified, these are documented here: https://www.ibm.com/support/pages/testing-statement-ibm-mq-multi-instance-queue-manager-file-systems

This provides functional verification, but users often need to understand the performance characteristics and if any bottlenecks occur when scaling. The recommended approach is for users to complete a comprehensive performance test of their own environment to verify that it produces the desired behavior, and a stress test to understand the characteristics in extreme situations.

In the case of AWS EFS (https://aws.amazon.com/efs/), we have identified several aspects that are important to consider and might affect the scalability and suitability of using EFS in general:

  • EFS Storage Class: there are various options AWS provide to define the throughput of EFS storage. IBM MQ is sensitive to the performance of the file system, especially in the case of persistent messaging. AWS also documents upper limits associated with the total throughput possible from an NFS client which could impact some scenarios. For the latest information consult https://docs.aws.amazon.com/efs/latest/ug/limits.html#limits-client-specific
  • File Locks: although AWS provides the capability to select the storage class to change the characteristics of the throughput, these do NOT change the overall limits associated with the number of file locks an NFS client can use. As of November 2019 the limits are these:

“Each unique mount on the client can acquire up to a total of 8,192 locks across a maximum of 256 unique file-process pairs. For example, a single process can acquire one or more locks on 256 separate files. As another example, eight processes can each acquire one or more locks on 32 files.”

https://docs.aws.amazon.com/efs/latest/ug/limits.html#limits-client-specific

To understand the impact of this restriction you need to understand how MQ uses file locks. MQ gets file locks in many situations, for example:

  • When the Queue Manager starts, file locks are obtained to complete logic such as logging, reading configuration and administrative operations. This often translates to at least 20 file locks being consumed before handling any messaging traffic. Depending on your individual configuration this can be slightly higher or lower.
  • As queues are opened by an application locks are taken against the individual queue files representing those queues.
  • When a transaction is being processed, locks are taken to ensure the consistency of the processing across failure situations.
  • For persistent messaging operations the queue manager periodically checks for consistency between its recovery logs and the queue files. This operation is called checkpointing and can take locks on the log and queue files.

For example, 250 clients each accessing a unique queue would likely exceed the EFS lock limit.

The file locks can be observed by using standard operating system utilities such as lslocks on Linux:

root@ubuntu:~# lslocks | grep mqm

amqzmuf0 19757 POSIX 47B   READ  0  0  0  /var/mqm/qmgrs/QM1/active

amqfcxba 19813 POSIX 47B   READ  0  0  0  /var/mqm/qmgrs/QM1/active

amqzxma0 19710 POSIX 47B   WRITE 0  0  0  /var/mqm/qmgrs/QM1/master

amqzxma0 19710 POSIX 47B   READ  0  0  0  /var/mqm/qmgrs/QM1/active

amqzmuc0 19727 POSIX 47B   READ  0  0  0  /var/mqm/qmgrs/QM1/active

amqzmuc0 19727 POSIX 6.1K  READ  0  0  0  /var/mqm/log/QM1/amqhlctl.lfh

amqzmur0 19742 POSIX 47B   READ  0  0  0  /var/mqm/qmgrs/QM1/active

amqzfuma 19719 POSIX 47B   READ  0  0  0  /var/mqm/qmgrs/QM1/active

amqrrmfa 19760 POSIX 47B   READ  0  0  0  /var/mqm/qmgrs/QM1/active

amqzlaa0 19792 POSIX 47B   READ  0  0  0  /var/mqm/qmgrs/QM1/active

amqfqpub 19796 POSIX 47B   READ  0  0  0  /var/mqm/qmgrs/QM1/active

java     19852 POSIX 47B   READ  0  0  0  /var/mqm/qmgrs/QM1/active

Predicting when the limit will be exceeded is complicated, because of the nature of the limit being tiered, allowing 256 file / process pairs and an additional limit on the absolute number of locks across all files (for instance 8,192). Therefore it is recommended that performance and stress testing is completed for your scenario with careful monitoring of the associated file locks to ensure you do not approach or reachthe AWS EFS limit.

Document Location

Worldwide

[{"Business Unit":{"code":"BU053","label":"Cloud & Data Platform"},"Product":{"code":"SSYHRD","label":"IBM MQ"},"Component":"","Platform":[{"code":"PF025","label":"Platform Independent"}],"Version":"All Versions","Edition":"","Line of Business":{"code":"LOB45","label":"Automation"}}]

Product Synonym

WebSphere MQ;MQ

Document Information

Modified date:
09 December 2019

UID

ibm11125177