RabbitMQ 3.7 is coming to an end on September 10, 2020.
That means that on that date, current IBM Cloud Databases users will no longer be able to provision RabbitMQ 3.7.
By September 10, 2020, Messages for RabbitMQ instances on version 3.7 that are still active will be disabled. Users are expected to be either on RabbitMQ 3.8 or the latest preferred version of the database.
Impact to Messages for RabbitMQ users
Customers can upgrade by following the directions found in the documentation.
What are quorum queues?
With the release of RabbitMQ version 3.8, a new queue type called quorum queue got introduced. Using quorum queues can significantly improve high availability of a deployment depending on its use case.
The main feature of quorum queues is a FIFO queue that makes use of Raft to provide high availability via replication in a cluster. In a quorum, the available majority of nodes decides which of the existing followers gets elected as leader. Previously existing queues of type “classic” have to rely on the mirrored queues concept to achieve high availability, which has to be applied and managed over the whole cluster lifecycle.
Quorum queues and classic queues
Using mirrored queues, a short disconnect or restart causes queues to dismiss their content and resynchronize their content from the current master. While the queues are synchronizing, they can’t proceed incoming messages and become unavailable until the synchronization has finished. Synchronization requires a considerable amount of memory and can have other negative side effects (especially for larger queues when using automatic synchronization), while having manual synchronization comes with possible data loss.
Quorum queues are durable, and every node keeps its messages on disk as well as in-memory, so there is no need for that kind of synchronization. However, that makes it more memory- and disk-intensive, and features like fanout exchange have a great—until almost unusable—impact for storing all the duplicate messages. The performance of quorum queues is considered equal or better than mirrored queues when enough resources are available.
Quorum queue use cases
Quorum queues can be considered for use cases that depend on long-running queues that require data integrity and high availability rather than flexibility and short-lived life cycles. Not all features are available for quorum queues, so it is recommended to verify the feature matrix when considering quorum queues.
The following points have to be considered when using quorum queues:
Clients
Must make use of manual acknowledgements and publisher confirms.
Resources
A quorum queue requires a service instance with a generous amount of memory, disk, and IOPs to keep its promise. Check out RabbitMQ’s recommendation to determine your resource needs for running quorum queues.
Configuration
- delivery-limit: Handles messages that cause a consumer to continuously redeliver messages without consuming them completely.
- max-length-bytes: As with every queue, the queue length must not grow unconditionally and needs to be well maintained and monitored.
References:
- Quorum Queues Documentation
- Lifting the lid on Quorum Queues
- RabbitMQ 3.8 Feature Focus – Quorum Queues
Upgrading Messages for RabbitMQ
If you want to move over to Messages for RabbitMQ version 3.8, it is pretty simple to transfer your workload. It all starts with creating a backup of your current Messages for RabbitMQ deployment. You can do that from the IBM Cloud API or via the IBM Cloud UI.
For the UI, you’ll select the Backups tab at the top of your deployment’s management view. Click the Backup Now button to initiate a manual backup. Once the backup has been completed, select that backup from your deployment’s backup list:
From here, click the blue Restore button to restore. This will allow you to create a new deployment from your backed up data and select RabbitMQ version 3.8 as the new version.
Make sure that you scale up your new deployment to what your current RabbitMQ deployment uses. After you’re done selecting the options you want set, click on the blue Restore button to start provisioning the new deployment. Once your new deployment has provisioned, you can start using it immediately.
Resource requirements for quorum queues
As mentioned above, quorum queues have a lot of benefits but they are resource-heavy, so you’ll need to make sure that you have enough available. We recommend following RabbitMQ’s guidelines for determining the amount of resources that you’ll need to scale to use quorum queries effectively.
You can also use the autoscale feature to add more resources automatically as your needs grow. To do that, select the Resources tab from your new Messages for RabbitMQ 3.8 deployment and then scroll to the bottom where you’ll see the following:
You have a number of options on how you want to set up autoscaling based on when you want to scale and how much you want to scale by. We’ve selected the defaults above by clicking on the boxes indicated by blue checkmarks. Once you’ve selected those boxes, make sure you click on the blue Save Changes button to initiate autoscaling for your deployment.
Remember, when scaling disk, you cannot scale disk down, only up. So, make sure that you watch your disk allocation closely.