Event streaming concepts
Concepts for event streaming and messaging with the MQ API
Event streaming combines messaging with an event log
Messaging uses the pub/sub model, where applications publish messages to a message broker and subscribe to topics on the broker to receive each message published on those topics. Messages can contain commands, queries, alerts, status, and data. A message broker streams messages using real-time push notifications, fast delivery, and guaranteed delivery. The problem with traditional messaging is that once a message broker delivers a message, there is no record of the message. Event streaming solves this problem by using an event log to persist each message in the order the broker receives it.
FairCom achieves event streaming by converging its database engine with its MQTT broker. While the broker delivers messages to a topic, the database engine persists them in a special integration table assigned to that topic. Each message is a record in that integration table that applications can replay from any point.
Because integration tables persist the delivery state of messages, they guarantee delivery to offline subscribers when they come online. Additionally, the database engine tracks where applications last stopped reading so they can start where they left off. Applications can also filter messages using SQL and JSON DB to process only the messages that pertain to them and paginate through messages efficiently in batches. The FairCom server can also transform incoming messages to enrich, reshape, reformat, and recalculate their information.
Customers can use event streaming for chat, collaboration, behavior tracking, logistics, alerts, push notifications, distributed command processing, data synchronization, performance monitoring, etc.
Event streaming applies to all industries and organizations in use cases that require continuous, real-time data processing:
Process financial transactions within and across stock, bank, and insurance exchanges.
Manage logistics continuously across cars, trucks, planes, fleets, and ships.
Analyze sensor data in real time from IoT devices and equipment in factories, wind parks, solar farms, smart cities, etc.
Monitor patients in hospital care, predict emergency conditions, and send alerts.
Interactively respond to customers in real time in retail, hotel, and travel applications.
Exchange information within and between companies with real-time and batch processing.
Monitor the performance of sales, customers, shipping, and servers in real time.
Send push notifications, such as advertisements and sales events, to increase customer engagement.
Add chat, alerting, and push notifications to your applications with little development effort.
Add distributed command processing to your applications to build efficient, scalable, low-cost solutions.
Integrate IT software platforms with reduced costs by consolidating databases, message brokers, and ETL servers.
The MQ API makes it easier to use messaging and event streaming in any application.
MQTT drivers provide many ways to use MQTT incorrectly or inefficiently. The MQ API provides a single standard way to use messaging that is efficient and easy to use correctly.
It provides intuitive names and simplifies using advanced MQTT 5 features.
You can use any REST tool, such as POSTman, to publish and subscribe to messages.
All messages published to topics are received by all clients subscribing to those topics. It does not matter if you are using the MQTT protocol or the MQ API.
The MQ API makes it easy to centrally manage the settings of all MQ clients.
MQTT expects each client to configure its session settings, which is difficult and expensive.
The MQ API provides the
"alterMqSession"action to alter the session settings of any MQTT and MQ API client.
The MQ API guarantees you will retrieve messages in published order.
MQTT clients have rare cases in which messages are delivered out of order.
The MQ API efficiently retrieves messages in batches.
MQTT pushes messages one at a time.
The MQ API retrieves messages when you are ready to process them.
MQTT pushes messages as soon as they are published. This requires you to write more complex asynchronous code.
The MQ API allows your software to retrieve a batch of messages when it is ready to process them. This is much simpler to program.
The MQ API simplifies and standardizes security.
The MQTT 3.1 and 5 specifications leave security largely undefined, making security prone to misconfiguration and creating security incompatibilities across MQTT brokers.
The MQ API uses HTTPS and a simple, standard, and proven security model for authentication that supports client certificates or usernames and passwords.
The MQ API works through firewalls.
Some organizations' security policies reject MQTT from passing through their firewalls.
The MQ API eliminates the need for browser-based applications to use WSS to send and receive MQTT messages.
Browser-based applications cannot directly use MQTT, but they can upgrade HTTPS to WSS and tunnel MQTT over WSS to an MQTT broker that supports this capability, such as the FairCom server.
Client certificates do not work over WSS.
An event is a record of something happening. It is also called a message. A producer publishes events to topics on the event server, and a consumer subscribes to topics on the event server. The event server takes a topic's events and delivers them to subscribers.
An event server durably stores events so clients can replay, query, filter, enrich, and batch process them.
An event record has an identifier, topic, value, timestamp, and optional metadata.
An event server automatically generates the event's unique identifier and timestamp when receiving the event.
An event's value is typically JSON but can be any type of data, including XML, image, PDF, video, binary data, etc.
Event Description | Event Topic | Event Payload (JSON) |
|---|---|---|
Alysha used Venmo to pay $200 to Rob at 1:30 on January 7, 2010. |
| {
"api": "hub",
"action": "createInput",
"params": {
"inputName": "modbusRTU",
"serviceName": "modbus",
"dataPersistenceStrategy": "onChange",
"dataCollectionIntervalMilliseconds": 5000,
"immediatelyCollectDataOnStart": true,
"dataCollectionBufferCount": 2,
"settings": {
"modbusProtocol": "RTU",
"modbusSerialPort": "COM2",
"modbusBaudRate": 19200,
"modbusParity": "None",
"modbusDataBits": 8,
"modbusStopBits": 2,
"propertyMapList": [
{
"propertyPath": "temperature",
"modbusDataAccess": "holdingregister",
"modbusDataAddress": 1199,
"modbusUnitId": 5,
"modbusDataLen": 1
}
]
},
"tableName": "modbusTableRTU"
},
"authToken": "replaceWithAuthTokenFromCreateSession"
}
{
"source_payload": {
"eventType": "payment",
"payer": "Alysha",
"payee": "Rob",
"amount": 200,
"currency": "USD",
"transactionTimestamp": "2025-01-07T13:30:00.000",
"financialSystem": "paypal"
},
"id": 1,
"ts": "2025-01-07T14:46:03.314",
"metadata": {}
}
|
Sam's heart monitor triggered a heart attack alert at 3:30 PM on January 7, 2010. |
| {
"source_payload": {
"eventType": "alert",
"patient": "Sam",
"alertDescription": "myocardial infarction symptoms",
"alertCode": 5117,
"alertTimestamp": "2025-01-07T15:30:00.000"
},
"id": 1,
"ts": "2025-01-07T15:30:07.497",
"metadata": {}
}
|
Event producers are client applications that publish events to the event server. Consumers subscribe to events.
Producers and consumers do not know about each other because they publish and subscribe to topics. This decoupling makes solutions resilient to change, reduces the cost of building and maintaining them, and enables them to grow to a massive scale. Because producers never wait for consumers to process their messages, the event server must guarantee each event is delivered exactly once in the correct order to each subscriber.
The event server organizes event history by topic. Many producers and consumers can publish and subscribe to the same topic. For example, one producer can publish messages on a topic subscribed to by many consumers, or many producers can publish messages on a topic subscribed to by one consumer. Consumers can read a topic's previous events as often as desired because the server retains them for as long as you specify.
The event server can replicate events across servers in the same data center and across data centers, cloud regions, and cloud providers. This feature makes your data scalable and highly available.
When a request succeeds, the server sets the "errorCode" property to 0.
When a request fails, the server sets the "errorCode" property to a FairCom Error Number.
The server sets the "errorMessage" property to a string describing the particular failure and suggesting how to correct it.
In MQTT and the MQ API, the "clientName" uniquely identifies an MQ session but does not authorize it. Another authentication technique authorizes the connection, such as an account name and password, or a client certificate. Thus, any account may be used to authorize an MQ connection. Each account is assigned to roles that authorize access to specific topics, client names, actions, etc.
The MQ API authenticates an account using the "createSession" JSON Action. This account must have appropriate permissions before the MQ API allows it to perform actions, such as publishing to a topic, retrieving messages from a topic or subscription, and modifying a connection and subscription.
The MQTT protocol is similar. A client must create an authorized MQTT connection before publishing and subscribing to topics.
An MQ session is stateful and contains information about a client's settings, subscriptions, published topics, and queue position of each topic to which the client is subscribed. The FairCom MQ engine stores and retrieves this information when a client connects.
Each MQ client has a unique identifier called the "clientName" that associates it with precisely one MQ session. A "clientName" is the unique name of an MQ API or MQTT session. FairCom uses "clientName" for compatibility with MQTT terminology, which identifies a session by the name of the client device or software that connects to the server.
The MQTT protocol is stateful and exclusive. If an authorized client uses the same client identifier as another connected client, an MQTT broker connects the new client and disconnects the existing client.
The MQ API allows multiple processes to publish and retrieve messages using the same client identifier specified using the "clientName" property. You must use MQ client names carefully because multiple processes can use the same client name with the "getMessagesFromMqApiSession" action to remove messages from the client's queues, which denies other processes using the same client name from receiving those messages. This technique is useful for distributed message processing. If this is undesirable for your use case, use "getMessagesFromMqTopic" or ensure each process uses a different client name so that each process has its own subscription message queue for each topic.
A session created by the MQ API and one created by MQTT use different wire protocols that are fundamentally incompatible. Thus, "getMessagesFromMqApiSession" works only with a "clientName" associated with the MQ API.
When you omit "clientName", the "createMqApiSession" action generates and returns a unique client name. You should save and use each generated client name to avoid polluting the MQ engine with abandoned sessions. You can use the "deleteMqSession" action to remove abandoned sessions.