MQTT concepts
MQTT is a client-server messaging protocol for software to publish and subscribe to messages. It is optimized for the Internet of Things (IoT). Its key features are simplicity and ease of use.
FairCom MQ is a high-speed, highly reliable, low-maintenance MQTT broker written in C++ with a built-in JSON/SQL database. It fully supports MQTT 3.1, MQTT 3.1.1, and MQTT 5. It is suitable for mission-critical solutions collecting data on the edge of the network, such as manufacturing, remote facilities, smart hospitals, smart cities, and smart buildings.
Why MQTT?
MQTT is designed for the Internet of Things (IoT) including manufacturing 4.0, smart buildings, and smart hospitals. Similar to how the request-response paradigm of HTTP transforms the Internet into a human web, the publish-subscribe model of MQTT transforms IoT into a machine web.
Low-cost communication between machines requires adopting a single industry standard, and this is where the lightweight, scalable, and easy-to-use MQTT protocol comes into play. MQTT 5 extends the feature set of MQTT in ways that make it more powerful for IoT and more capable for IT. IoT has already adopted MQTT as its primary standard, and more and more IT shops are adopting MQTT because it is so easy to use and implement.
Use cases
MQTT is designed for Machine-to-Machine (M2M) and Internet of Things (IoT) applications.
MQTT excels in scenarios where guaranteed message delivery is important, such as an application using distributed components. For example, an IoT application gathers data from and sends commands to distributed devices, which may become unavailable, intermittently connected, or disconnected by a less reliable wireless network. MQTT can guarantee delivery of messages to and from distributed components.
MQTT is a simple push technology, which makes it easy to build distributed chat, alert, and command processing applications.
Industrial automation
Gathering telemetry
Sending commands
Notification services
Chat applications
Remote monitoring
Manufacturing
Automotive
Healthcare
Telecom
Energy
Smart cities
Smart buildings
What is MQTT?
MQTT is a client-server messaging protocol for software to publish and subscribe to messages. Easy to use client libraries exist for all major programming languages. MQTT is an open Internet technology and runs on the TCP stack.
An MQTT client publishes each message to a topic. To receive messages, a client subscribes to one or more topics. The same client can publish and subscribe to topics. An MQTT broker takes each message sent to a topic and delivers it to each subscriber of that topic.
MQTT is a brokered push technology. Clients push messages to a broker and a broker pushes messages to clients. A client receives an entire message as an asynchronous event – including its binary payload of up to 255 MB. This pure push model sets MQTT apart from AMQP, Kafka, and other messaging technologies.
An MQTT broker also pushes the entire message to each subscriber as an event. This eliminates the complexity of clients going to the broker, choosing a message queue, and filtering messages in the queue.
MQTT allows a client to stay connected to a broker. This allows the client to receive messages from the broker as they are published in real-time. It also allows the broker to track the online status of a client. A client can also disconnect and reconnect as desired. When a client reconnects, a broker delivers the messages that were sent when the client was disconnected.
MQTT 3, MQTT 3.1, and MQTT 5 are open OASIS standards. An MQTT 5 broker supports MQTT versions 3, 3.1, and 5, and allows clients to use any of these versions. Thus, an MQTT 5 broker allows a client to connect using any MQTT version. An MQTT client can send messages to any other MQTT client - regardless of its version.
Pub/sub decoupling
MQTT uses a brokered publish-subscribe pattern (pub/sub). A client assigns a topic to a message and publishes it to a broker, and the broker delivers a message to all clients that subscribe to that topic. The broker decouples publishers from subscribers, which means clients do not know about each other.
Decoupling makes integration fast, flexible, dynamic, scalable, and highly reliable. In contrast, clients in point-to-point integrations become tightly interconnected in complex ways, and this slows integration, limits integration options, makes change difficult, limits scalability, and reduces availability.
Space decoupling
Clients are unaware of each other
Time decoupling
Clients can be connected at different times
Sync decoupling
Clients send and receive messages asynchronously without blocking operations
MQTT’s ability to decouple makes it attractive for integration use cases. It speeds integration and makes change easy and flexible. It supports intermittently connected equipment and vehicles. It reliably delivers alerts, commands, and telemetry as soon as devices reconnect.
MQTT protocol
MQTT works on top of the TCP/IP stack and communicates by exchanging packets.
CONNECT
DISCONNECT
PUBLISH
SUBSCRIBE
UNSUBSCRIBE
The other ten packet types are used for internal message flows to acknowledge requests, ensure quality of service, keep a connection alive, and enhance authentication.
Packet type | Description |
|---|---|
CONNECT | Client sends a connection request to the broker. |
CONNACK | Broker acknowledges connection. |
PUBLISH | Client sends a message about a topic to the broker, or a broker sends a message to a subscriber of a topic. |
PUBACK | QoS 1 PUBLISH acknowledgement. |
PUBREC | QoS 2 PUBLISH acknowledgement. |
PUBREL | QoS 2 PUBREC acknowledgement. |
PUBCOMP | QoS 2 PUBREL acknowledgement. |
SUBSCRIBE | Client sends a request to a broker to subscribe to topics. |
SUBACK | Broker acknowledges subscription. |
UNSUBSCRIBE | Client sends a request to a broker to unsubscribe from topics. |
UNSUBACK | Broker acknowledges unsubscription. |
PINGREQ | Client and broker periodically send a keepalive heartbeat message to ensure both are still listening. |
PINGRESP | Receiver acknowledges heartbeat. |
DISCONNECT | Client sends a message to a broker to disconnect gracefully. |
AUTH | Client optionally sends an MQTT5 message to authenticate. |
Topics
A topic is a string of properties separated internally by the forward slash character, such as "factory1/line1/machineGroup1/command/machine1”. A topic can be a long UTF-8 string of up to 64K characters, which allows a topic to contain a large number of human-readable, descriptive properties. When subscribing to topics, an MQTT client can use wildcards to match topics that contain one or more of these properties – no matter where the property occurs in the topic. This makes the subscription process similar to querying data.
MQTT designed topics to be data rich and full of context so that message payloads can be simple. MQTT also supports simple topics and complex payloads.
Clients publish and subscribe to topics. A broker delivers each message sent to a topic to each subscriber of that topic. Unlike other message technologies, an MQTT client can simply send a message to a topic and the broker will create the topic.
An MQTT broker remembers the topics that a client subscribes to. This frees a client from tracking its subscriptions and resubscribing each time it connects.
Each topic is used to publish a specific type of message. This allows subscribers to know what content to expect in the topic’s payload.
Telemetry
Status
Notification
Alert
Command
Any client can send any message to any topic. This flexibility requires integrators and systems to choose a pattern for naming and using MQTT topics.
A best practice is for each client to publish messages to its own unique topics. A client uses a different topic for each type of purpose, such as telemetry, status, notifications, alerts, and commands. Each topic has a hierarchy of topic levels that identify the context of the sender, the purpose of the topic, and the sender’s unique identifier, such as factory1/line1/group1/telemetry/machineId1. Subscribers specify which topics they want to receive and they can use wildcards to subscribe to groups of topics. Because the topic defines the context of its messages, the message payload can be a simple, predefined type of value, such as the following JSON telemetry document:
{ "temperature": 19 }Another practice is to use a common set of topics to collect messages. Each topic has a general purpose, such as
"alerts". Because the topic contains less information, the payload of each message contains more information, such as the following JSON alert document:{ "source": { "factory": "factory1", "line": "line1", "group": "group1", "device": "machineId1" }, "alertCode": "2", "alertDescription": "failure" }An antipattern is to use one central topic for all messages. Because the topic contains no information, the payload of each message must contain all information. All clients subscribe to the one topic, receive every message, and filter out the messages they do not want. This is an antipattern because it does not leverage an MQTT broker's ability to filter messages for subscribers using specific topics and topic wildcards. The payload of each message must contain all information, such as the following generic JSON document:
{ "source": { "factory": "1", "line": "2", "group": "3", "device": "4" }, "metrics": [ { "name": "Temperature", "value": "19" } ], "alerts": [ { "alertCode": "2", "alertDescription": "failure" } ], "status": [ { "stateCode": "2", "stateDescription": "down" } ], "commands": [ { "method": "sendMetrics", "params": {} } ], "metadata": [ { } ] }
MQTT topic wildcards
A topic consists of one or more properties. Each property is separated by a forward slash. MQTT calls these properties "topic levels". Properties can be hierarchical, but they are not required to be hierarchical.
When subscribing to a topic, a client can replace a property with a wildcard as a way to subscribe to many topics. Wildcarded subscriptions enable integration to be flexible and dynamic.
Wildcard | Symbol | Description |
|---|---|---|
Single-level wildcard | + | It matches one complete topic level. It may be used more than once in a topic subscription. |
Multi-level wildcard | # | It matches all subsequent topic levels. It must be the last character of a topic subscription. |
Important
The + and # characters can only be used in a topic name when subscribing.
Property-matching wildcards
The single-level wildcard (+) allows properties in a topic to be treated as independent attributes – they are not required to be hierarchical. The number and order of properties are fixed, which is important because properties do not have names.
A client can subscribe to a topic filter that contains many single-level (+) wildcards. This turns the subscription process into a pattern matching process. For example, the topic, "+/RedSox/+/+/+" matches all topics containing the "RedSox" property.
This example uses topics relevant to a trading system about sports, players, collectables, etc. These topics also include actions, such as getting information as well as receiving requests to buy and bid on items.
1-sport/2-team/3-item/4-action/5-player:basketball/bulls/jersey/getInfo/MichaelJordanbasketball/wizards/player/getInfo/MichaelJordanbasketball/tarheels/collectables/bid/MichaelJordanbaseball/barons/collectables/buy/MichaelJordanbaseball/RedSox/player/getInfo/BabeRuthbaseball/Yankees/player/getInfo/BabeRuthbaseball/Braves/collectables/bid/BabeRuth
baseball/+/collectables/+/+matchesbaseball/barons/collectables/buy/MichaelJordanbaseball/Braves/collectables/bid/BabeRuth
+/+/player/getInfo/+matchesbasketball/wizards/player/getInfo/MichaelJordanbaseball/RedSox/player/getInfo/BabeRuthbaseball/Yankees/player/getInfo/BabeRuth
Hierarchy-matching wildcards
A single # wildcard matches all topics.
factory1/line1/telemetry/machine1factory1/line1/command/#factory1/line1/alert/#factory1/line1/#factory1/##
Combining wildcards
If you want to combine single- and multi-level wildcards, put hierarchical properties at the end of the topic and put non-hierarchical properties at the beginning. This works because the multi-level wildcard (#) must occur at the end of a topic, but the single-level (+) wildcard can occur at any level in a topic.
1-machineType/2-manufacturer/3-factory/4-line/5-station/6-action/7-specificMachineIdstamper/PrimeRobotics/PlantDetroit3/lineA/station1/telemetry/stamper315cnc/Hurco/PlantDetroit3/LineA/station2/alert/cnc2bwelder/Lincoln/PlantYork2/Line1/stationB/command/welder17welder/Lincoln/PlantYork2/Line1/stationB/alert/welder17welder/Lincoln/PlantYork2/Line1/stationB/telemetry/welder17
welder/+/PlantYork2/Line1/stationB/command/#welder/+/PlantYork2/+/+/command/#+/Hurco/PlantDetroit3/+/+/alert/#+/+/PlantDetroit3/LineA/station2/alert/#+/+/PlantDetroit3/LineA/+/alert/#+/+/PlantDetroit3/+/+/alert/#+/+/+/+/+/alert/#welder/+/PlantYork2/Line1/stationB/telemetry/#welder/+/PlantYork2/Line1/stationB/#welder/+/PlantYork2/Line1/#welder/+/PlantYork2/#welder/Lincoln/#+/Lincoln/#welder/+/#welder/#
Note
The best practice is to make topics as simple as possible while including enough properties in the right order to make subscriptions flexible and dynamic.
Wildcard use cases
Topic levels and wildcards enable IoT environments to create a hierarchy of topics that make it easy to adapt to change.
factory
area
line
cell
messagetype
machine
station
Clients can subscribe to higher topic layers to automatically monitor many devices as they come and go. Clients can subscribe to lower topic layers to receive commands sent specifically to them or to their subsystem.
QoS levels
Each MQTT message is published with one of three Quality of Service (QoS) levels. Each level provides different delivery guarantees.
Important
In MQ Explorer there is an option to pick which QOS level is used for MQTT messages (QoS 0, QoS 1, and QoS 2). However when forwarding, if QoS 2 is chosen and the broker is forwarding to AWS, the broker will keep connecting and disconnecting.
QoS level | Description |
|---|---|
0 | At most once delivery A message is delivered zero or one time. This is the fastest QoS because the receiver does not acknowledge receipt. The quality of delivery relies on the reliability of TCP. No retransmission takes place. |
1 | At least once delivery A message is delivered one or more times. This is the best option to guarantee delivery with the least overhead, such as sending telemetry or an alert. The sender resends a message until the receiver acknowledges receipt. Duplicate messages can occur. |
2 | Exactly once delivery A message is delivered one time. This is the best option when neither loss nor duplication of messages are acceptable, such as sending a command. It requires the sender and receiver to resend the message and its acknowledgments until they are all acknowledged. |
Last Will and Testament
Monitoring and management software need to receive notifications when equipment or software fails unexpectedly. To meet this need, an MQTT broker always tracks the connection state of each client and knows when it disconnects intentionally or unintentionally. A broker can send a Last Will and Testament (LWT) message, also known as Death Certificate message, on behalf of a disconnected client. The topic, payload, and properties of the LWT message is optionally defined by each client when it connects.
Advanced MQTT brokers also provide dashboards to track disconnected clients. They may also allow authorized clients to query the broker for disconnected clients and/or to register LWT messages on behalf of other clients. This is useful when a client cannot register its own LWT message or when an organization needs to standardize the topics and payloads of LWT messages.
Retained messages
When a client newly subscribes to a topic, it typically wants to know it is connected to the right topic and wants to know the state of that topic. It can wait for the next message to be sent, but that may take a while.
MQTT allows a publisher to tag any message that it publishes as a retained message, which means that the broker retains the message for the purpose of sending it on behalf of the client to new subscribers. Each topic may have one retained message. Any client can make any message a retained message; thus, many clients can set the retained message of a topic.
Each message published to MQTT can be flagged as a retained message. A broker stores the retained message. Each topic can have one retrained message. Each time a new client subscribes to a topic or to a wildcard that matches the topic, the broker will immediately send the last retained message of that topic.
A retained message is useful for communicating the last known state. It can be a "coming online" message (Birth Certificate), an alert, a telemetry message, or a planned "going offline" message (Death Certificate).
Sessions
An MQTT broker is responsible for storing session information of each client so it can guarantee message delivery. It also makes implementing MQTT very simple for a client.
All subscriptions of the client
All QoS 1 & 2 messages which are not yet processed
All QoS 1 & 2 messages the client missed while offline
MQTT clients typically use persistent sessions to ensure they do not miss any messages for certain topics—not even when they are disconnected. When a client reconnects, the broker sends all missed messages for a subscription with a QoS Level of 1 or 2. Even a publishing-only client benefits from a persistent session because the broker tracks its online or offline state and can send birth and death certificates on its behalf.
When a client connects to an MQTT broker, it may set the CLEAN flag to request a clean session, which is useful for removing client state from the broker. It is also useful for creating a temporary development or monitoring session where the broker persists no session state.
Heartbeats
To allow a broker to track the state of a client connection and to free resources when a client has disconnected abnormally, an MQTT client specifies a keepAlive value when connecting to the broker. The maximum value is 65,535 seconds (18h 12m 15s). If a client does not send a message within 150% of the keepAlive value, a broker terminates the connection. A client sends a PINGREQ message to the broker within the keepAlive interval if no other message exchange is occurring. The broker responds with a PINGRESP message.