Documentation

FairCom EDGE IoT Database

Previous Topic

Next Topic

MQTT Store and Forward

For specifically configured topics, the database can store MQTT messages and forward them to subscribers. These topics are called "Store & Forward Topics." This feature enhances MQTT to guarantee delivery for specific topics even when a subscriber is unavailable to receive them.

When an existing subscriber to a Store & Forward Topic becomes unavailable and later becomes available, the MQTT Broker forwards messages to the subscriber starting after the last successfully delivered message.

Learn more...

This feature enhances MQTT to guarantee delivery for specific topics even when a subscriber is unavailable to receive them.

This requires the MQTT Broker to do the following:

  • Track the Client ID for each subscriber.
  • Track the table record ID of the last message successfully sent to each subscriber.

Unless the Store & Forward Topic is configured with "retentionUnit": "forever", the MQTT Broker automatically removes expired messages.

Expired messages can no longer be forwarded. Thus, when a subscriber is unavailable longer than the retention period and later becomes available, the MQTT Broker can only send it unexpired messages.

For configured topics, the MQTT Broker collects and stores messages in anticipation of future subscribers.

The MQTT Broker delivers messages in the order they were originally received.

An MQTT message for a Store & Forward Topic may contain any binary content up to 2GB in length per message.

  • The MQTT Broker stores the message in a variable-length binary field in a table created specifically for the Store & Forward Topic.
  • It does not modify the message.
  • It does not assume the message has a specific format, such as JSON, XML, etc.

This feature is available only for topics that are configured specifically for this purpose.

Because storing messages impacts performance and storage, the MQTT Broker only stores and forwards messages for Store & Forward Topics.

The broker can be configured to do the following with Store & Forward Topics:

  • Create a Store & Forward Topic.
  • Remove a Store & Forward Topic. This deletes all its tables and data and all subscriber tracking. This action cannot be undone.

Creating a Store & Forward Topic

A Store & Forward Topic is created by setting the storeAndForward property to True in the CreatePersistenceTopic operation. The following optional configuration properties can be set:

  • storeAndForward: "True" | "False"
  • "maxDeliveryRatePerSecond"
  • "newSubscriberDeliveryMode"
  • "newSubscriberMaxStoredMessages"

The default rate per second for delivering stored messages is 100.

"maxDeliveryRatePerSecond": 100 - This is the maximum delivery rate for messages being delivered to a subscriber when it becomes available or when it is a new subscriber and asks for all stored messages to be delivered to it. The actual rate will be bounded by the network, local resources, and server performance. This keyword provides an artificial “throttling” of the delivery rate to avoid saturating the network and other resources.

  • The default mode for delivering messages to a new subscriber is to deliver only newly received messages.

    "newSubscriberDeliveryMode": "NewMessages"

    Values for "newSubscriberDeliveryMode":

    "NewMessages" - A new subscriber only receives new messages published to the topic.

    "StoredMessages" - A new subscriber receives all stored messages published to the topic.

    Depending on the quantity of stored messages, it may take a significant amount of time before current messages are delivered.

    A new subscriber may not be sent all messages ever received by the MQTT Broker.

    • A topic stores a finite number of messages capped by the retentionPeriod and retentionUnit configuration properties.
    • A topic does not store its messages until it is configured to be a Store & Forward Topic. Thus, all messages sent to the MQTT Broker prior to that point cannot be delivered to future subscribers.
  • The default maximum number of stored messages to deliver to a new subscriber is 1,000.

    "newSubscriberMaxStoredMessages": 0 - This applies only to new subscribers and it limits the number of previously stored messages that the MQTT Broker will send. Thus, it applies only when the Store & Forward Topic is configured as "newSubscriberDeliveryMode": "StoredMessages"

    "newSubscriberMaxStoredMessages": "all" is used to set no limits on how many messages will be sent to a new subscriber.

    "newSubscriberMaxStoredMessages": 0 is used when no previous messages should be sent to a new subscriber.

Example 1:

Configuration message (JSON) to create a Store & Forward Topic:

{

"operation": "CreatePersistenceTopic",

"persistenceTopic": "Example1",

"tableName": "Example1",

"storeAndForward": "True",

"tableRetentionPeriod": 3,

"tableRetentionUnit": "month",

"maxDeliveryRatePerSecond": 100,

"newSubscriberDeliveryMode": "StoredMessages",

"newSubscriberMaxStoredMessages": 1000

}

The Example 1 topic will store 3 months' worth of messages in the table named Example1. If an outage occurs, the messages missed by existing subscribers will be delivered at the throttled rate of 100 messages per second, to keep from saturating the system. New subscribers will receive the last 1000 stored messages.

Example 2:

Configuration Message to Remove a Store & Forward Topic

{

"operation": "RemoveStoreForwardTopic",

"storeForwardTopic": "Example1"

}

In Example 2, “myTopic” will remove the Topic, which will remove all currently stored messages and will also remove all files on disk used for storing the messages.

TOCIndex