Event streaming concepts
Event streaming combines messaging with an event log
Messaging uses the pub/sub model where applications publish messages to topics on a message broker. They also subscribe to topics on the broker to receive each message published on those topics. Messaging provides real-time push notifications, fast delivery, and guaranteed delivery. The problem with messaging is that once a message broker delivers a message, there is no record of the message.
Event streaming uses an event log to persist each message in the order the broker receives it. Applications read the event log to replay messages. They can replay messages from the beginning or end. They can paginate through messages efficiently in batches and filter messages to process only those messages that interest them.
FairCom DB and MQ seamlessly combine FairCom's database engine with its MQTT broker. The messaging engine persists each message to a special integration table that has built-in features designed for messaging. The engine creates an integration table for each topic. Each message is a record in the integration table, and each table can transform incoming messages using the built-in V8 JavaScript engine. It also has the unique ability to deliver each message to various outputs and APIs including MQTT, REST, SQL, JSON DB API, etc. The integration table persists delivery state to guarantee delivery to offline subscribers when they come online.
Applications can read the records in each integration table to retrieve messages from the beginning, end, or at any point along the way. The engine can track where applications left off reading and start where they left off. Applications can use SQL and JSON DB to filter messages to process messages that interest them.
The MQ API makes it easier to use messaging in any application.
MQTT drivers provide many ways to use MQTT incorrectly or inefficiently. The MQ API provides a single standard way to use messaging that is efficient and easy to use correctly.
It provides intuitive names and simplifies using advanced MQTT 5 features.
You can use any REST tool, such as POSTman, to publish and subscribe to messages.
All messages published to topics are received by all clients subscribing to those topics. It does not matter if you are using the MQTT protocol or the MQ API.
The MQ API makes it easy to manage the settings of all MQ clients centrally.
MQTT expects each client to configure its session settings, which is difficult and expensive.
The MQ API provides the
"alterMqSession"action to alter the session settings of any MQTT and MQ API client.
The MQ API guarantees you retrieve messages in published order.
MQTT clients have rare cases in which messages are delivered out of order.
The MQ API efficiently retrieves messages in batches.
MQTT pushes messages one at a time.
The MQ API retrieves messages when you are ready to process them.
MQTT pushes messages as soon as they are published. This requires you to write more complex asynchronous code.
The MQ API allows your software to retrieve a batch of messages when it is ready to process them. This is much simpler to program.
The MQ API simplifies and standardizes security.
The MQTT 3.1 and 5 specifications leave security largely undefined, making security prone to misconfiguration and creating security incompatibilities across MQTT brokers.
The MQ API uses HTTPS and a simple, standard, and proven security model for authentication that supports client certificates or usernames and passwords.
The MQ API works through firewalls.
Some organizations' security policies reject MQTT from passing through their firewalls.
The MQ API eliminates the need for browser-based applications to use WSS to send and receive MQTT messages.
Browser-based applications cannot directly use MQTT, but they can upgrade HTTPS to WSS and tunnel MQTT over WSS to an MQTT broker that supports this capability, such as the FairCom server.
Client certificates do not work over WSS.
When a request succeeds, the server sets the "errorCode" property to 0.
When a request fails, the server sets the "errorCode" property to a FairCom Error Number.
The server sets the "errorMessage" property to a string describing the particular failure and suggesting how to correct it.
In MQTT and the MQ API, the "clientName" uniquely identifies an MQ session but does not authorize it. Another authentication technique authorizes the connection, such as an account name and password or a client certificate. Thus, any account may be used to authorize an MQ connection. Each account is assigned to roles that authorize access to specific topics, client names, actions, etc.
The MQ API uses the "createSession" JSON Action to authenticate an account. This account must have appropriate permissions before the MQ API allows it to perform actions, such as publishing to a topic, retrieving messages from a topic or subscription, and modifying a connection and subscription.
The MQTT protocol is similar. It requires a client to create an authorized MQTT connection before publishing and subscribing to topics.
An MQ session is stateful and contains information about a client's settings, subscriptions, published topics, and queue position of each topic to which the client is subscribed. The FairCom MQ engine stores this information and retrieves it when a client connects.
Each MQ client has a unique identifier called the "clientName" that associates it with precisely one MQ session. You can think of a "clientName" being the unique name of an MQ API of MQTT session. FairCom uses "clientName" for compatibility with MQTT terminology, which identifies a session by the name of the client device or software that connects to the server.
The MQTT protocol is stateful and exclusive. If an authorized client uses the same client identifier as another connected client, an MQTT broker connects the new client and disconnects the existing client.
The MQ API allows multiple processes to simultaneously publish and retrieve messages using the same client identifier, which is specified using the "clientName" property. You must use MQ client names carefully because multiple processes can use the same client name with the "getMessagesFromMqApiSession" action to remove messages from the client's queues, which denies other processes using the same client name from receiving those messages. This technique is useful for distributed message processing. If this is undesirable for your use case, use "getMessagesFromMqTopic" or ensure each process uses a different client name so that each process has its own subscription message queue for each topic.
A session created by the MQ API and one created by MQTT use different wire protocols that are fundamentally incompatible. Thus, "getMessagesFromMqApiSession" works only with a "clientName" associated with the MQ API.
When you omit "clientName", the "createMqApiSession" action generates and returns a unique client name. You should save and use each generated client name to avoid polluting the MQ engine with abandoned sessions. You can use the "deleteMqSession" action to remove abandoned sessions.