Getting to know MQTT
Why MQTT is one of the best network protocols for the Internet of Things
For Internet of Things (IoT) devices, connecting to the internet is a requirement. Connecting to the internet allows the devices to work with each other and with backend services. The underlying network protocol of the internet is TCP/IP. Built on top of the TCP/IP stack, MQTT (Message Queue Telemetry Transport) has become the standard for IoT communications.
MQTT was originally invented and developed by IBM in the late 1990's. Its original application was to link sensors on oil pipelines with satellites. As its name suggests, it is a messaging protocol that supports asynchronous communication between parties. An asynchronous messaging protocol de-couples the message sender and receiver in both space and time, and hence is scalable in unreliable network environments. Despite its name, it has nothing to do with messaging queues, and uses a publish and subscribe model instead. In late 2014, it officially became an OASIS open standard, and it is supported in popular programming languages by using multiple open source implementations.
MQTT is a lightweight and flexible network protocol that strikes the right balance for IoT developers:
- The lightweight protocol allows it to be implemented on both heavily constrained device hardware and high latency / limited bandwidth networks.
- Its flexibility makes it possible to support diverse application scenarios for IoT devices and services.
To understand why MQTT is such a good fit for IoT developers, let's first examine why other popular network protocols failed in IoT.
Why not ... the rest of the alphabet soup of network protocols
Most developers are already familiar with HTTP web services. So, why not have IoT devices connect to web services? The device could send in its data as an HTTP request and receive updates from the system as the HTTP response. This request and response pattern does have some severe limitations:
- HTTP is a synchronous protocol. The client waits for the server to respond. Web browsers have this requirement, but it comes at the cost of poor scalability. In the world of IoT, the large number of devices and most likely an unreliable or high latency network have made synchronous communication problematic. An asynchronous messaging protocol is much more suitable for IoT applications. The sensors can send in readings, and let the network figure out the optimal path and timing for delivery to its destination devices and services.
- HTTP is one way. The client must initiate the connection. In an IoT application, the devices or sensors are typically clients, which means that they cannot passively receive commands from the network.
- HTTP is a 1-1 protocol. The client makes a request, and the server responds. It is difficult and expensive to broadcast a message to all devices on the network, which is a common use case in IoT applications.
- HTTP is a heavy weight protocol with many headers and rules. It is not suitable for constrained networks.
For the above reasons, most high-performance scalable systems use an asynchronous messaging bus, rather than web services, for internal data exchange. In fact, the most popular messaging protocol that is used in enterprise middleware systems is called AMQP (Advanced Message Queuing Protocol). However, in the high-performance environment, computing power and network latency are typically not a concern. AMQP is designed for reliability and interoperability in enterprise applications. It has a rich feature set, but it is not suitable for resource-constrained IoT applications.
Besides AMQP, there are other popular messaging protocols. For example, the XMPP (Extensible Messaging and Presence Protocol) is a peer-to-peer instant messaging (IM) protocol. It is heavy on features that support IM use cases, such as presence and media attachments. Compared with MQTT, it requires much more resources both on the device and on the network.
So, what makes the MQTT so lightweight and flexible? A key feature of the MQTT protocol is its publish and subscribe model. As with all messaging protocols, it decouples the publisher and consumer of data.
The publish and subscribe model
The MQTT protocol defines two types of entities in the network: a message broker and a number of clients. The broker is a server that receives all messages from the clients and then routes those messages to relevant destination clients. A client is anything that can interact with the broker to send and receive messages. A client can be an IoT sensor in the field or an application in a data center that processes IoT data.
- The client connects to the broker. It can subscribe to any message "topic" in the broker. This connection can be a plain TCP/IP connection or an encrypted TLS connection for sensitive messages.
- The client publishes messages under a topic by sending the message and topic to the broker.
- The broker then forwards the message to all clients that subscribe to that topic.
Since MQTT messages are organized by topics, the application developer has the flexibility to specify that certain clients can only interact with certain messages. For example, sensors will publish their readings under the "sensor_data" topic, and subscribe to the "config_change" topic. Data processing applications that save sensor data into a backend database will subscribe to the "sensor_data" topic. An admin console application could receive system admin's commands to adjust the sensors' configurations, such as sensitivity and sample frequency, and publish those changes to the "config_change" topic. (See Figure 1.)
Figure 1. The MQTT publish and subscribe model for IoT sensors
At the same time, MQTT is lightweight. It has a simple header to specify the message type, a text-based topic, and then an arbitrary binary payload. The application can use any data format for the payload, such as JSON, XML, encrypted binary, or Base64, as long as the destination clients can parse the payload.
Getting started with MQTT development
The easiest tool to get started with MQTT development is the Python mosquitto module, which is part of the Eclipse Paho project that provides MQTT SDKs and libraries in multiple programming languages. It contains an MQTT broker that can run on your local computer and command line tools to interact with the broker by using messages. You can download and install the mosquitto module from the mosquitto website.
The mosquitto command runs the MQTT broker on your local computer. You can optionally use the -d option to run it in the background.
$ mosquitto -d
Next, in another terminal window, you can use the mosquitto_sub command to connect to the local broker and subscribe to a topic. After the command runs, it will wait and print out any message it receives from the subscription as they come in.
$ mosquitto_sub -t "dw/demo"
In yet another terminal window, you can use the mosquitto_pub command to connect to the local broker and then publish a message to a topic.
$ mosquitto_pub -t "dw/demo" -m "hello world!"
Now, the terminal that runs mosquitto_sub should now print out "hello world!" on the screen. You just sent and received a message by using an MQTT broker!
Of course, in a production system, you cannot use a local computer as your broker. Instead, you might use the IBM Bluemix Internet of Things Platform service, which is a reliable and on-demand service that functions like an MQTT broker. (You can read more about how this Bluemix service integrates and uses MQTT as its protocol for communicating with devices and applications in the service documentation.)
The IBM Bluemix Internet of Things Platform service works as follows.
- From the Bluemix console, you can create an instance of the Internet of Things Platform service on demand.
- Then, you can add devices that can connect the service instance by using MQTT. Each device will have an ID and name. Only listed devices can access the service, and the Watson IoT Platform dashboard will report the traffic and usage information on these devices.
- For each device client, Bluemix will assign a host name, user name, and password to connect to your service instance (the MQTT broker). (On Bluemix, the user name is always use-token-auth, and the password is the token that is shown in Figure 2 for each connected device.)
Figure 2. Creating an Internet of Things Platform service instance in IBM Bluemix
When using a remote MQTT broker, you will need to pass in the broker's host name and authentication credentials to your mosquitto_sub and mosquitto_pub commands. For example, the following command subscribes to the demo topic on our Internet of Things Platform service with the user name and password that is provided by Bluemix:
$ mosquitto_sub -t "demo" -h host.iotp.mqtt.bluemix.com -u username -P
For more options on how to use the mosquitto tools and also on how to use the mosquitto API to create your own MQTT client applications, see the documentation on the mosquitto website.
Now that you have the necessary tools, let's dive deeper into the MQTT protocol.
Understanding the MQTT protocol
MQTT is a wire protocol that specifies how data bytes are organized and transmitted over the TCP/IP network. But for practical purposes, developers do not need to understand the wire protocol. All we need to know is that each message has a command and data payload. The command defines the message type (for example, a CONNECT message or a SUBSCRIBE message). All MQTT libraries and tools provide simple ways to manipulate those messages directly and can automatically populate some required fields, such as message and client IDs.
First, the client connects to the broker by sending a CONNECT message. The CONNECT message asks to establish a connection from the client to the broker. The CONNECT message has the following content parameters.
Table 1. CONNECT message parameters
|cleanSession||This flag specifies whether the connection is persistent or not. A persistent session stores all the subscriptions and potentially missed messages (depending on QoS) in the broker. (See Table 3 for a description of QoS.)|
|username||The broker's authentication and authorization credentials.|
|password||The broker's authentication and authorization credentials.|
|lastWillTopic||When the connection is dropped unexpectedly, the broker will automatically publish a "last will" message to a topic.|
|lastWillQos||The "last will" message's QoS. (See Table 3 for a description of QoS.)|
|lastWillMessage||The "last will" message itself.|
|keepAlive||This is the time interval that the client needs to ping the broker to keep the connection alive.|
The client will receive a CONNACK message from the broker. The CONNACK message has the following content parameters.
Table 2. CONNACK message parameters
|sessionPresent||This indicates whether the connection already has a persistent session. That is, the connection already has subscribed topics, and will receive the delivery of missed messages.|
|returnCode||0 indicates success. Other values identify the cause of failure.|
After a connection is established, the client can then send one or more SUBSCRIBE messages to the broker to indicate that it will receive messages from the broker for certain topics. The message can have one or multiple repeats of the following parameters.
Table 3. SUBSCRIBE message parameters
|qos|| The qos (quality of service, or QoS) flag indicates how
consistently the messages under this topic needs to be delivered
to clients. |
|topic||A topic to subscribe to. A topic can have multiple levels separated by the slash character. For example, "dw/demo" and "ibm/bluemix/mqtt" are valid topics.|
After a client has successfully subscribed to a topic, the broker returns a SUBACK message with one or more returnCode parameters.
Table 4. SUBACK message parameters
|returnCode|| A return code exists for each of the topics in the SUBCRIBE
command. The return values are as follows. |
Corresponding to the SUBSCRIBE message, the client can also UNSUBSCRIBE from a topic or multiple topics.
Table 5. UNSUBSCRIBE message parameters
|topic||This parameter can repeat for multiple topics.|
The client can send PUBLISH messages to the broker. The message contains a topic and data payload. The broker then forwards the message to all clients that subscribe to that topic.
Table 6. PUBLISH message parameters
|topicName||The topic under which the message is published.|
|qos||The quality of service level of the message delivery. (See Table 3 for a description of QoS.)|
|retainFlag||This flag indicates whether the broker will retain the message as the last known message for this topic.|
|payload||The actual data in the message. It can be a text string or a binary blob of data.|
Tips and workarounds
The power of MQTT is its simplicity. You have no constraints on the kind of topics or message payloads that you can use. That allows for some interesting use cases. For example, consider these questions:
How do I do 1-1 messages with MQTT? Both parties can agree to use a topic that is unique to them. For example, the topic name could contain the IDs of both clients to ensure its uniqueness.
How does a client broadcast its presence status? The system could have an agreed upon naming convention for "presence" topics. For example, the "presence/client-id" topic could have the presence information for a client. The client sets the message to true when it connects and false when it disconnects. The client can also set a last will message to false so that it is set when the connection drops. The message can be retained by the broker so that new clients can read the topic and find out the presence status.
How do I secure the communications? The client-to-broker connection can be an encrypted TLS connection to protect the data in transit. In addition, since the MQTT protocol imposes no constraints on the payload data format, the system could have an agreed upon encryption method and key update mechanism. After that, all content in the payload could be encrypted binary data of the actual JSON or XML messages.
In this article, I gave a technical introduction to the MQTT protocol. You learned what is MQTT, what makes MQTT suitable for IoT applications, and how to get started with developing apps that use MQTT.
- The official website of MQTT
- MQTT v3.1.1 is an official OASIS standard
- The Eclipse Paho project, includes popular open source MQTT implementations
- The mosquitto project, an open source Python MQTT broker and client library
- MQTT documentation for the Internet of Things Platform service