Event streaming concepts
Event streaming combines messaging with an event log.
Messaging uses the pub/sub model where applications publish messages to topics on a message broker. They also subscribe to topics on the broker to receive each message published on those topics. Messaging provides real-time push notifications, fast delivery, and guaranteed delivery. The problem with messaging is that once a message broker delivers a message, there is no record of the message.
Event streaming uses an event log to persist each message in the order the broker receives it. Applications read the event log to replay messages. They can replay messages from the beginning or end. They can paginate through messages efficiently in batches and filter messages to process only those messages that interest them.
FairCom DB and MQ seamlessly converge FairCom's database engine with its MQTT broker. The messaging engine persists each message to a special integration table that has built-in features designed for messaging. The engine creates an integration table for each topic. Each message is a record in the integration table, and each table can transform incoming messages using the built-in V8 JavaScript engine. It also has the unique ability to deliver each message to various outputs and APIs including MQTT, REST, SQL, JSON DB API, etc. The integration table persists delivery state to guarantee delivery to offline subscribers when they come online.
Applications can read the records in each integration table to retrieve messages from the beginning, end, or at any point along the way. The engine can track where applications left off reading and start where they left off. Applications can use SQL and JSON DB to filter messages to process messages that interest them.
The MQ API makes it easier to use messaging in any application.
MQTT drivers provide many ways to use MQTT incorrectly or inefficiently. The MQ API provides a single standard way to use messaging that is efficient and easy to use correctly.
It provides intuitive names and simplifies using advanced MQTT 5 features.
You can use any REST tool, such as POSTman, to publish and subscribe to MQTT.
All messages published to topics are received by all clients subscribing to those topics. It does not matter if you are using the MQTT protocol or the MQ API.
The MQ API makes it easy to manage the settings of all MQ clients centrally.
MQTT expects each client to configure its session settings, which is difficult and expensive.
The MQ API provides the
"alterMqttSession"action to alter the session settings of any MQTT and MQ API client.
The MQ API guarantees you retrieve messages in published order.
MQTT clients have rare cases in which messages are delivered out of order.
The MQ API efficiently retrieves messages in batches.
MQTT pushes messages one at a time.
The MQ API delivers messages when you are ready to process them.
MQTT pushes messages as soon as they are published. This requires you to write more complex asynchronous code.
The MQ API allows your software to retrieve a batch of messages when it is ready to process them. This is much simpler to program.
The MQ API simplifies and standardizes security.
The MQTT 3.1 and 5 specifications leave security largely undefined, making security prone to misconfiguration and creating security incompatibilities across brokers.
The MQ API uses HTTPS and a simple, standard, and proven security model for authentication that supports client certificates or usernames and passwords.
The MQ API works through firewalls.
Some organizations' security policies reject MQTT from passing through their firewalls.
The MQ API eliminates the need for browser-based applications to use WSS to send and receive MQTT messages.
Browser-based applications cannot directly use MQTT, but they can upgrade HTTPS to WSS and tunnel MQTT over WSS to a broker that supports this capability.
Client certificates do not work over WSS.
When a request succeeds, the server sets the "errorCode" property to 0.
When a request fails, the server sets the "errorCode" property to a FairCom Error Number.
The server sets the "errorMessage" property to a string describing the particular failure and suggesting how to correct it.
In MQTT and the MQ API, the "clientName" uniquely identifies an MQ session but does not authorize it. Another authentication technique authorizes the connection, such as an account name and password or a client certificate. Thus, any account may be used to authorize an MQ connection. Each account is assigned to roles that authorize access to specific topics, client names, actions, etc.
The MQ API uses the "createSession" JSON Action to authenticate an account. This account must have appropriate permissions before the MQ API allows it to perform actions, such as publishing to a topic, retrieving messages from a topic or subscription, and modifying a connection and subscription.
The MQTT protocol is similar. It requires a client to create an authorized MQTT connection before publishing and subscribing to topics.
An MQ session contains information about a client's subscriptions, settings, and queue position of each topic to which the client is subscribed. The FairCom MQ engine stores this information and retrieves it when a client connects.
Each MQTT client has a unique identifier called the "clientName" that associates it with precisely one MQTT session.
The MQTT protocol is stateful and exclusive. If an authorized client uses the same client identifier as another connected client, the MQTT broker connects the new client and disconnects the existing client.
The JSON MQ Message API works differently because it is stateless. It allows multiple processes to simultaneously publish and retrieve messages using the same client identifier, which is specified using the "clientName" property. Unlike the MQTT protocol, the JSON MQ Message API does not disconnect an existing MQTT session that uses the same client name. This allows a broker administrator to publish messages on behalf of a device without disconnecting the device.
Warning
Be sure to take extra care when using the stateful "getMessagesFromMqttSession" action to ensure that no other process is using the same MQTT client identifier; otherwise, each process that retrieves messages denies other processes from retrieving them. If your usage of MQTT allows multiple processes to use the same client identifier, use the "getMessagesFromMqttTopic" to retrieve messages safely.
When you omit "clientName", the "createMqttSession" action generates and returns a unique client name. You should persist and reuse generated client names to avoid polluting the MQTT broker with abandoned sessions.