How to Migrate from Compose for RabbitMQ to IBM Cloud

6 min read

Migrating your Compose for RabbitMQ message broker over to IBM Cloud Messages for RabbitMQ with the Shovel plugin.

In our series of Compose migration articles, we've discussed how to move your databases from Compose to IBM Cloud Databases. In this article, we'll divert a little and focus on how to migrate your Compose for RabbitMQ message broker over to IBM Cloud Messages for RabbitMQ

The process is simple enough using the Shovel plugin, which is available for both IBM Cloud Messages and Compose for RabbitMQ users. Using Shovel, we'll show you the migration process for your RabbitMQ exchanges and queues using the RabbitMQ management UI and also through the HTTP API. 

Provisioning 

In order to migrate to IBM Cloud Messages for RabbitMQ, you must have an instance up and running. You can set it up via the IBM Cloud Catalog or using the IBM Cloud CLI. We recommend taking a look at our documentation before provisioning so that you can view some of the provisioning options available.

The basic command to provision a RabbitMQ instance using the IBM Cloud CLI is the following:

ibmcloud resource service-instance-create <service-name> <service-id> <service-plan-id> <region> \
-p '{"version": <version of RabbitMQ>,"members_disk_allocation_mb": "<size>", "members_memory_allocation_mb": "<size>"}'

In this case, it may look like this for you:

ibmcloud resource service-instance-create example-rmq messages-for-rabbitmq us-south \
-p '{ "version": "3.8", "members_disk_allocation_mb": "9216", "members_memory_allocation_mb": "6144"}'

Note that the values for members_disk_allocation_mb and members_memory_allocation_mb are the total amount for all members. When we allocate 9216 GB of disk, it is spread out across three members. For RAM, you can scale up and down, but for disk, you can only scale up.

Before provisioning, make sure that you set up your IBM Cloud Messages for RabbitMQ instance to have the same, if not more, disk and RAM as your Compose instance. With IBM Cloud Messages, you need to calculate the amount of IOPs you'll need—which depends on your instance's workload—and you need to allocate the appropriate amount of RAM as well. 

We have have an article titled "Getting Started with Messages for RabbitMQ" that explains the differences between Compose for RabbitMQ and IBM Cloud Messages for RabbitMQ, so check that article out for a deeper discussion on the differences between services.

Once your instance is ready to use, make sure you change the administrator password so that you can access Messages for RabbitMQ.

Creating the Shovel in the RabbitMQ UI

What is Shovel? Shovel is a plugin for moving messages from one message broker to another. It enables you to configure a source and destination message broker, which allows you to read and write messages and handles connection failures, too. You can use your AMQPS connection strings from your instances to connect to queues or exchanges from the source to destination brokers so that when your Compose instance receives a message, that message automatically gets sent to the new Messages for RabbitMQ instance exchange or queue.

The Shovel plugin is enabled by default on Compose. You can set up a Shovel either using the Compose for RabbitMQ administration UI or via the RabbitMQ HTTP API. From the administration UI in your Compose instance click on the Admin tab and select Shovel Management:

From the administration UI in your Compose instance click on the Admin tab and select Shovel Management:

That will take you to the Shovel Management view were you can set up the shovel. When creating a new shovel, click on the Add shovel button and that will present a form where you create a name for the shovel and add the source and destination fields. You'll also have to select whether you want the shovel to forward messages from a queue or an exchange:

That will take you to the Shovel Management view were you can set up the shovel. When creating a new shovel, click on the Add shovel button and that will present a form where you create a name for the shovel and add the source and destination fields. You'll also have to select whether you want the shovel to forward messages from a queue or an exchange:

In the example above, I've named the shovel compose-to-messages-rmq. We've set up the shovel to use AMQP 0.9.1 and the URI is your Compose for RabbitMQ AMQPS connection string with the default virtual host added at the end. I've set this up to forward the exchange in my Compose instance called compose-exchange, which has the routing key hello. So, any messages with the hello routing key going to this exchange will be routed here. 

Typically, this will be the same exchange on Compose that you've been using for your applications already. We just need to designate that we want to move messages also from that exchange over to our Messages for RabbitMQ instance. 

Next, for the destination, I've also used the AMQP 0.9.1 protocol and used my Messages for RabbitMQ URI—also with the default virtual host, which is /. Since I am forwarding messages from the exchange on Compose, I will also need to create an exchange in Messages for RabbitMQ. I've set that up and called it messages-rmq

Note that you will have to create the exchange in your Messages for RabbitMQ instance first before creating the shovel, or you will see an error because the exchange that you're forwarding messages to will not exist until you've created it. For the messages-rmq exchanges, I also used the same routing key hello. After that, click on the Add shovel button. 

With a shovel in place, you will see something like the following from the Shovel Management view:

With a shovel in place, you will see something like the following from the Shovel Management view:

As you can see, it shows that you've set up a new shovel, and any subsequent shovels that you add will also be shown here. Now, you can click on the Shovel Status tab on the right and view if the shovel has been successfully created and running. As you can see below, the state should show that it's running:

As you can see below, the state should show that it's running:

If the state ever shows that it has stopped—or if you want to restart the shovel—click on the Restart button under Operations

To create a shovel for a queue, it's essentially the same process that we went though for the exchange; however, you don't have to have to create the queue in the destination message broker. If your queue doesn't exist in the destination broker, RabbitMQ will create that for you, which is a pretty nice feature.

Creating a Shovel using the RabbitMQ HTTP API

When creating a shovel using the HTTP API, you'll follow a different process. With an exchange already created in both your source and destination brokers that we'll call compose-api-exchange and messages-api-exchange, respectively, and using helloapi as the compose-api-exchange routing key and newapi as the messages-api-exchange routing key. I've set it up with different routing keys to show that you don't necessarily need the same routing key. Now, using the cURL command in your terminal, you can create the shovel in your default virtual host by supplying your own deployment's information:

curl -XPUT \ 
-H "content-type:application/json" \
'https://<username>:<password>@<host>:<port>/<vhost>' \
-d '{
     "value": {
     "src-protocol": "amqp091", 
     "src-uri":  "amqps://<username>:<password>@<host>:<port>/<vhost>
     "src-exchange":  "<exchange name>", 
     "src-exchange-key": "<routing key>", 
     "dest-protocol": "amqp091", 
     "dest-uri": "amqps://<username>:<password>@<host>:<port>/<vhost>", 
     "dest-exchange": "<exchange name>", 
     "dest-exchange-key": "<routing key>", 
     "dest-add-forward-headers": true
     }
}'

Once you've ran that from the terminal, you'll see something like the following in your RabbitMQ administration UI:

Once you've ran that from the terminal, you'll see something like the following in your RabbitMQ administration UI:

Getting messages

Shovel works with all types of exchanges. In this example, I've set up exchanges as direct exchanges which use routing keys. However, if you have fanout exchanges, those will work as well—but you won't indicate a routing key when setting up the shovel for the exchange. 

To demonstarate how you can send messages, let's look at the shovels we created for the first exchange that we set up—compose-to-messages-rmq. This shovel connects to our Compose for RabbitMQ exchange compose-exchange with our messages-rmq exchange in our Messages for RabbitMQ instance. I've also created two queues to send messages to: compose-queue in the Compose instance and messages-queue in the Messages for RabbitMQ instance. Since the shovel is sending messages to the two exchanges, the queues have been bound to the exchanges so that the messages are received and queued up.

Let's send a message using cURL to the compose-exchange

curl -XPOST \
-H "content-type:application/json" \
'https://<username>:<password>@<host>:<port>/api/exchanges/<vhost>/compose-exchange/publish' \
-d '{"properties":{},"routing_key":"hello","payload":"this is a message from compose","payload_encoding":"string"}'

Using the routing key hello, we'll send a message—this is a message from compose—as a string to the compose-exchange. Once this is sent, you will receive an acknowledgement: {"routed":true}. Since we sent the message to the compose-exchange, the queues that use the hello routing key in the Compose instance will receive the message, and the messages-exchange in the Messages for RabbitMQ instance will also get the message and direct that to the messages-queue

We can see this if we look at the messages in both queues:

curl -XPOST \
-H "content-type:application/json" \
'https://<username>:<password>@<host>:<port>/api/queues/<vhost>/compose-queue/get' \
-d '{"count":5,"ackmode":"ack_requeue_true","encoding":"auto","truncate":50000}'


[
  {
    "payload_bytes": 30,
    "redelivered": true,
    "exchange": "compose-exchange",
    "routing_key": "hello",
    "message_count": 0,
    "properties": [],
    "payload": "this is a message from compose",
    "payload_encoding": "string"
  }
]

Also, when we look at the message in the Messages for RabbitMQ instance, you will see the following:

curl -XPOST --cacert <path/to/CA/cert> \
-H "content-type:application/json" \
'https://<username>:<password>@<host>:<port>/api/queues/<vhost>/messages-queue/get' \
-d '{"count":5,"ackmode":"ack_requeue_true","encoding":"auto","truncate":50000}'


[
  {
    "payload_bytes": 30,
    "redelivered": true,
    "exchange": "messages-rmq",
    "routing_key": "hello",
    "message_count": 0,
    "properties": [],
    "payload": "this is a message from compose",
    "payload_encoding": "string"
  }
]

As you can see, both queues received the message from both message brokers connected only via shovel. 

With data flowing to both message brokers, you'll be able to switch the consumer applications over to Messages for RabbitMQ. After testing, you can start switching over producers, as well, and test further. Once your testing has completed, you're then able to deprovision your Compose instance and you'll be all set on IBM Cloud Messages for RabbitMQ.

Summary

The Shovel plugin for RabbitMQ makes it really easy for you to migrate your message broker from Compose to IBM Cloud Messages for RabbitMQ. You can create as many shovels as you need for each exchange—and even for each queue.

This quick guide is part of a series of articles that explains how to migrate your databases from Compose to IBM Cloud Databases—the latest generation of open-source databases on IBM Cloud. We focused on Compose for RabbitMQ in this article, but if you’re looking to migrate other databases from Compose to IBM Cloud Databases, we’ve got you covered in the following articles and documentation:

If you still have questions, reach out—our support team is always ready to lend a hand.

Be the first to hear about news, product updates, and innovation from IBM Cloud