Skip to main content

"getMessagesFromMqTopic" (JSON Action)

Retrieve messages published on a specific topic

This action uses the event streaming technique to retrieve messages published on a specific topic by the MQ API and MQTT. It is stateless and does not remove messages from the topic's queue. Use it to tail the end of the queue, retrieve all messages from the beginning of the queue, retrieve messages starting from a timestamp, or paginate through a queue. 

This action allows multiple processes to retrieve messages simultaneously from a topic without moving the client's message position or removing shared messages.

Instead of using a client's message position to retrieve undelivered messages, it retrieves messages from a specified starting location in the topic's message queue. You can retrieve messages from the end or beginning of the queue, a specific timestamp, or a queue position ID.

This action does not disconnect MQTT clients. 

Call this action to retrieve the "maxMessages" number of messages from a topic starting from the position specified by "startFrom". The message payload is encoded according to "binaryFormat". 

You must use one but not both of the following properties to identify the message queue from which you want to retrieve messages: 

  • Use "topic" to specify a topic from which you want to retrieve messages. A topic cannot contain wildcards.

  • Use "tableName" to specify the name of an integration table containing messages. You may also set the table's "databaseName" and "ownerName"; otherwise, the server will use the JSON Action Session's default values.

The optional "startFrom" property controls the starting point where the server delivers messages. 

  • Set "startFrom" to the default of ["tail"] to retrieve the most recently sent messages.

  • Set "startFrom" to "head" to retrieve messages from the beginning. 

  • Set "startFrom" to an ISO 8601 DateTime, such as  "2024-07-11T21:18:01.000", to retrieve messages starting on or after that date and time. If the time component is missing, it defaults to "00:00:00.000". 

  • Set "startFrom" to an integer number, such as 52233, to retrieve messages starting with the message with that ID number. 

Tip

To retrieve the next set of messages, set "startFrom" to one plus the ID number of the last message you received.

Optionally, set  "expiredMessagesFilter" to "include", "exclude", or "only" to include or exclude expired messages from the results. Use "only" to include only expired messages in the results. It defaults to "include".

Optionally, set  "includeMqttProperties": true to include the message's MQTT properties. It defaults to false.

Optionally, set "includeTransformFields": true to include the custom fields used for the transformation process's inputs and outputs. It defaults to false.

Tip

For more query options, use the JSON DB "getRecords..." actions. You can use the "describeMqSession" to return information about a client's subscriptions, which includes a map of topic names to table names. You can query those tables to see their messages. You can use SQL to run ad hoc queries. You can find a record using partial, exact, or inequality matches. You can move and skip forward and backward through the records in index-sorted order.

Request examples

Minimal

{
  "api": "mq",
  "action": "getMessagesFromMqTopic",
  "params": {
    "topic": "test/topic1"
  },
  "authToken": "replaceWithAuthTokenFromCreateSession"
}

Maximal

{
  "api": "mq",
  "action": "getMessagesFromMqTopic",
  "params": {
    "topic": "test/topic1",
    "databaseName": "faircom",
    "ownerName": "admin",
    "tableName": "mqtt_msg_test_topic1",
    "startFrom": "head",
    "maxMessages": 20,
    "binaryFormat": "hex",
    "expiredMessagesFilter": "include",
    "includeMessageMetadata": true,
    "includeMqttProperties": true,
    "includeTransformFields": true,
    "includeOtherFields": true
  },
  "responseOptions": {
    "dataFormat": "arrays", 
    "numberFormat": "number",
    "binaryFormat": "base64",
    "variantFormat": "json"
  },
  "apiVersion": "1.0",
  "debug": "max",
  "requestId": "optionalUniqueRequestIdFromTheClient",
  "authToken": "replaceWithAuthTokenFromCreateSession"
}

Retrieve the most recently sent message to topic1 with all metadata and the message payload formatted as JSON

{
  "api": "mq",
  "action": "getMessagesFromMqTopic",
  "params":
  {
    "maxMessages": 1,
    "binaryFormat": "json",

    "topic": "test/topic1",
    "expiredMessagesFilter": "include",

    "includeMessageMetadata": true,
    "includeMqttProperties": true,
    "includeTransformFields": true
  },
  "requestId": "optionalUniqueRequestIdFromTheClient",
  "authToken": "replaceWithAuthTokenFromCreateSession"
}

Retrieve the last 10 messages sent to topic1 with the message payload formatted as base64 and no metadata

{
  "api": "mq",
  "action": "getMessagesFromMqTopic",
  "params":
  {
    "topic": "test/topic1",

    "startFrom": "tail",
    "maxMessages": 10,
    "binaryFormat": "base64"
  },
  "requestId": "optionalUniqueRequestIdFromTheClient",
  "authToken": "replaceWithAuthTokenFromCreateSession"
}

Retrieve the last message sent to all topics, format the message payload as hexadecimal, and include all MQTT properties

{
 "api": "mq",
 "action": "getMessagesFromMqTopic",
 "params":
 {
   "topic": "#",
   "maxMessages": 1,
   "binaryFormat": "hex",

   "includeMessageMetadata": true,
   "includeExpiredMessages": true,
   "includeMqttProperties": true,
   "includeTransformFields": true
 },
   
    "authToken": "replaceWithAuthTokenFromCreateSession"
}

Retrieve the first 50 non-purged messages sent to topic1 with the message payload formatted as JSON

{
  "api": "mq",
  "action": "getMessagesFromMqTopic",
  "params":
  {
    "topic": "test/topic1",
    "startFrom": "head",
    "maxMessages": 50,
    "binaryFormat": "json"
  },
  "authToken": "replaceWithAuthTokenFromCreateSession"
}

Retrieve the next 50 non-purged messages sent to topic1

{
  "api": "mq",
  "action": "getMessagesFromMqTopic",
  "params":
  {
    "topic": "test/topic1",
    "startFrom": 51,
    "maxMessages": 50,
    "binaryFormat": "json"
  },
  "authToken": "replaceWithAuthTokenFromCreateSession"
}

Retrieve 5000 messages sent to topic1 on or after "2024-07-11T21:18:01.000"

{
  "api": "mq",
  "action": "getMessagesFromMqTopic",
  "params":
  {
    "topic": "test/topic1",
    "startFrom": "2024-07-11T21:18:01.000",
    "maxMessages": 5000,
    "binaryFormat": "json"
  },
  "authToken": "replaceWithAuthTokenFromCreateSession"
}

Response example

{
  "authToken": "authToken",
  "result": {
    "messages": [
    ]
  },
  "requestId": "2",
}

The getMessagesFromMqTopic JSON action provides a stateless, event-streaming method to retrieve messages published on a specific topic (or from an integration table), unlike getMessagesFromMqApiSession which works with client subscriptions. This action offers flexible starting points for message retrieval using the startFrom parameter, allowing users to fetch messages from the "head" (beginning), "tail" (most recent), a specific timestamp, or a message ID, without altering a client's message position or removing shared messages. You can control the number of messages returned via maxMessages, specify the binaryFormat of the payload, and filter for expiredMessagesFilter. Optional parameters also allow including message metadata, MQTT properties, and custom transform fields for more detailed insights into the retrieved messages.

getMessagesFromMqTopicget messages from mq topicJSON ActionMQ APIMQ get messages from topicMQ topic event streamingmessage retrieval APIread messages from MQ topicget messages from integration tableMQ topic message replayMQ API get messagesretrieve topic messagesmessage queue API topicmessages by timestampget messages from topic

"params"

Table 1. "params" property summaries

Property

Description

Default

Type

Limits (inclusive)

binaryFormat

(optional) specifies how the "payload" property is encoded

"base64"

string

One of the following: "base64", "hex", or "byteArray".

expiredMessagesFilter

(optional) specifies whether or not to include expired messages in the result

"include"

string

"include"
"exclude"
"only"

includeMessageMetadata

(optional) returns the message's "id", "timestamp", "error", and "log" properties in the response

false

Boolean

true
false

includeMqttProperties

(optional) returns MQTT properties in the response

false

Boolean

true
false

includeOtherFields

(optional) returns additional fields that are not the built-in integration table, transform, and MQTT fields in the response

false

Boolean

true
false

includeTransformFields

(optional) returns custom transform fields in the response

false

Boolean

true
false

maxMessages

(optional) specifies the number of messages to return from a "getMessages..." action

20

integer

0 to 100000

startFrom

(optional) sets the starting message position for a message filter

"tail"

string

"head"
"tail"
string ISO 8601 DateTime value
message ID number

topic

(optional) specifies the name of a topic

""

UTF-8 string



"binaryFormat"

"expiredMessagesFilter"

The "expiredMessagesFilter" property is optional. It is an enumerated string that contains one of the following values: "include", "exclude", and "only". It defaults to "include".

  • "include" (default) returns expired messages in the results.

  • "exclude" omits expired messages from the results.

  • "only" returns only expired messages in the results.

"includeMessageMetadata"

Set the "includeMessageMetadata" property to true to include the message's "id", "timestamp", "error", and "log" properties.

"includeMqttProperties"

Set the "includeMqttProperties" property to true to include the MQTT properties in the response. Otherwise, omit the property or set it to false or null. MQTT properties are helpful when you need to troubleshoot MQTT messages.

"includeOtherFields"

Set the "includeOtherFields" property to true to include additional fields that are not the built-in integration table fields, transform fields, and MQTT fields. Otherwise, omit the property or set it to false or null.

"includeTransformFields"

Set the "includeTransformFields" property to true to include custom transform fields (if any) that a customer added to the integration table. Otherwise, omit the property or set it to false or null. Custom transform fields are helpful when you want to see the outputs and inputs of transformations assigned to the integration table.maxMessages.

"maxMessages"

The "maxMessages" property is optional and defaults to 20. It has a minimum value of 1 and a maximum value of 100000. It specifies the number of messages to return from a "getMessages…" action. The number of returned messages will be equal to or less than this maximum.

"startFrom"

The "startFrom" property is an optional JSON string with an enumerated value of "head", "tail", a string ISO 8601 DateTime value, or a message ID number. The default value is "tail". It sets the starting message position for a message filter in the "getMessagesFromMqTopic" action. Use it to retrieve historical messages on or after the specified date and time.

  • When its value is "tail", the server returns the "maxMessages" number of messages before and including the last message. The server returns messages in order of the last message toward the first.

  • When its value is "head", the server returns the "maxMessages" number of messages after and including the first message that has not been purged from the server. The server returns messages in order of the first message toward the first.

  • When it is a string ISO 8601 DateTime value, such as "2024-07-11T21:18:01.000", the server returns the "maxMessages" number of messages starting on or after that date and time. If the time component is missing, it defaults to "00:00:00.000".

  • When it is an ID number, such as 52233, the server returns the "maxMessages" number of messages starting on or after that ID. Messages are returned in ID ascending order. Do not embed an ID number in a string so the server can tell the difference between an ID and a DateTime.

"topic"

The "topic" property is a required, case-sensitive UTF-8 string containing an MQTT topic. It may not contain MQTT wildcard characters, #, or +. It may contain multiple / characters, which segment a topic into a set of positional properties.

"responseOptions"

Table 2. "responseOptions" property summaries

Property

Description

Default

Type

Limits (inclusive)

binaryFormat

specifies how binary values are returned

"hex"

string

One of the following: "base64", "hex", or "byteArray".

dataFormat

specifies what format the results will be returned in

"arrays"

string

"default"
"arrays"
"objects"

excludeFields

(optional) specifies which fields are excluded in the response message

[]

When the array is empty or the property is not specified, the includeFields behavior applies

array

excludePaths

specifies which paths are excluded in the response message

[]

When the array is empty or the property is not specified, the includePaths behavior applies

array

includeFields

(optional) specifies which fields are returned in the response message

[]

When the array is empty or the property is not specified, all fields are returned

array

includePaths

specifies which paths are included in the response message

[]

When the array is empty or the property is not specified, all paths are returned

array

numberFormat

specifies how numbers are formatted in the JSON response message

"number"

string

"number"
"string"

omit

specifies which properties are excluded in the response message

{}

object

variantFormat

(optional) specifies how the server will represent the values of variant fields in its response to your request

"json"

string

"json"
"binary"
"string"
"variantObject"


"binaryFormat"

"dataFormat"

"numberFormat"

The "numberFormat" property is an optional, case-insensitive string enum. It defines the format of JSON numbers. The default value for "numberFormat" is the "defaultNumberFormat" defined in the "createSession" or "alterSession" actions. If it is omitted there, it defaults to the value of the "defaultNumberFormat" property in the <faircom>/config/services.json file.

When "numberFormat" occurs in "mapOfPropertiesToFields", it tells the server how to encode or decode a number assigned to a JSON property.

For example, including "numberFormat" in a "tableFieldsToJson" transform step controls if the server encodes a number in a JSON property as a number or a number embedded in a string.

Possible values:

  • "number"

    This causes the server to return numeric values as JSON numbers, such as -18446744073709551616.000144722494 .

    This is most efficient.

    JSON represents numbers are base-ten numbers that may have any number of digits.

    Large numbers, such as 18446744073709551616.000144722494 are known to cause problems with JSON parsers and some programming languages, such as JavaScript, which are limited to the smaller range and binary rounding errors of IEEE floating point numbers.

  • "string"

    This returns the server to embed numeric values in JSON strings, such as "18446744073709551616.000144722494" .

    This is slightly less efficient because it includes two extra double quote characters

    Returning numbers embedded in strings ensures JSON parsers and programming languages will not automatically convert the number to a numeric representation that loses precision, introduces rounding errors, truncates values, or generates errors. This allows your application to control how numbers are processed.

  • Omitted or set to null

    This defaults to "number".

Example request

{
  "action": "someAction",
  "responseOptions":
  {
    "numberFormat": "string"
  },
  "authToken": "replaceWithAuthTokenFromCreateSession"
}

"variantFormat"

The "variantFormat" property tells the server how to format the values of variant fields in its response to your request.

The "variantFormat" property has one of the following values: "binary", "json", "string", and "variantObject". It tells the server how to store and return values stored in variant fields. The default value is "json".

Things to know

  • The server applies the "variantFormat" property to all variant fields affected by a JSON action.

  • If you want to control the variant format of each field, set the "variantFormat" property to "variantObject" and use variant objects to set the variant type and encoding of each variant field.

The following sections describe each value of the "variantFormat" property.

"variantFormat": "binary"

  • Use the "variantFormat": "binary" setting to encode a variant field's value as a JSON string.

  • The "binaryFormat" property specifies how to encode the binary value.

    • The "binaryFormat": "hex" setting uses hexadecimal to encode a binary value in a JSON string.

    • The "binaryFormat": "base64" setting uses Base64 to encode a binary value in a JSON string.

    • The "binaryFormat": "byteArray" setting uses an array of integer numbers to encode each byte in the binary value.

  • When "variantFormat" is "binary", the server sets the variant's type to "binary" and stores raw bytes in the variant field.

  • The "binary" variant type ensures maximum compatibility and performance across all FairCom APIs, including JSON DB, SQL, CTDB, and ISAM.

  • The server does not validate the binary value, which can be any sequence of bytes.

"variantFormat": "json"

  • Use the "variantFormat": "json" setting to encode a variant field's value as a JSON value.

    • A JSON value can be an object, array, string, number, true, false, or null.

    • A binary field value is returned in a JSON string.

    • The "binaryFormat" property specifies the format of a binary variant value embedded in a JSON string.

    • The "numberFormat" property causes the server to return numbers as JSON numbers or JSON strings.

  • When "variantFormat" is "json", the server sets the variant's type to "json" and stores a JSON value in the variant field.

    • The server validates JSON before it assigns it to a variant field.

    • The server converts a variant field to valid JSON before it returns the value of a variant field.

  • When "variantFormat" is set to "json" in "responseOptions" and the "type" of variant object is "binary", the JSON API returns the binary variant value embedded in a string or as a byte array depending on the "binaryFormat" property in "responseOptions". In other words, when "variantFormat" is set to "json" in "responseOptions", the API returns a JSON value. All variant values can be returned as native JSON values except for binary values because JSON cannot represent them. Thus, when a variant has a binary value and the response option is JSON, the API embeds the binary value in a JSON string according to the value of "binaryFormat".

  • The server converts variant values to and from JSON as follows:

    • JSON null: The server does not store a JSON null; instead it marks the variant field as NULL. If the field is not nullable, the API returns an error.

    • JSON string: The server stores a JSON string in the variant field as is. It includes the double quotes before and after the string's value and stores escaped characters without decoding them. For example, the server stores the JSON string, "my // string",  as "my // string". The server returns a JSON string exactly as it stores it.

    • JSON number: The server stores a JSON number in the variant field as is. A number consists of the following ASCII characters 1234567890+-.eE. For example, the server stores the JSON number -123.45  as the ASCII characters -123.45. The server returns a JSON string exactly as it stores it.

    • JSON true & false: The server stores a JSON true and false in the variant field as is. It stores and returns the ASCII characters true or false.

    • JSON object & array: The server stores a JSON object and array as is. When it validates the JSON, it minimizes whitespace while preserving the UTF-8 characters representing the actual object or array. For example, the server stores the array [1,2,3]  as the UTF-8 characters [1,2,3].

"variantFormat": "variantObject"

  • Use the "variantFormat": "variantObject" setting to encode a variant field's value as a variant object.

  • When the "insertRecords" and "updateRecords" actions have "variantFormat": "variantObject" in "params", you must assign a variant object to each variant field.

  • A JSON action returns a variant object for a variant field value when you put "variantFormat": "variantObject" in "responseOptions".

"variantFormat": "string"

  • Use the "variantFormat": "string" setting to encode a variant field's value as a JSON string.

  • This setting in the "responseOptions" property causes all "getRecords..." actions to return string values from variant fields.

    • Binary values are returned as a JSON string containing an embedded binary value. The "binaryFormat" property specifies how to encode the binary value.

    • Number values are returned as a JSON number embedded in a string, such as "123.45". The server ignores the "numberFormat" property in the "responseOptions" object.

    • Boolean values are returned as a "true" or "false" string.

    • String values are returned as a JSON string, which may contain JSON escaped characters.

    • JSON values are returned embedded in a string.

      • A JSON object is escaped, encoded, and embedded in a string, such as "{\"key\": \"value\"}".

      • A JSON array is escaped, encoded, and embedded in a string, such as "[1, \"my string\", true ]".

      • A JSON string is returned as a JSON string.

      • A JSON number is embedded in a JSON string, such as "-123.456".

      • A JSON true is returned as "true".

      • A JSON false is returned as "false".

      • A JSON null is returned as "null".

  • This setting in the "params" property for "insertRecords" and "updateRecords" actions requires a string value to be assigned to each variant field.

    • FairCom does not recommend this setting because it can only store strings in variant fields. Consider using the other "variantFormat" options to store many more types of data, such as binary, JSON objects, JSON arrays, numbers, etc.