Skip to main content

"alterIntegrationTable" (JSON Action)

JSON hub "alterIntegrationTable" action modifies settings for existing tables that are safe to modify

The "alterIntegrationTable" action alters table settings that are safe to modify.

  • The settings that are safe to modify include:

    • Renaming a table

    • Adding fields

    • Increasing field size

    • Changing table retention policy

    • Changing table metadata

    • Changing a table's transformation pipeline

  • As you refine your integration processes, you may want to rename an integration table to better label the data it holds.

    You can use the "alterIntegrationTable" action with the "newTableName" property to rename an integration table.

    Tip

    You can also use the "tableName" property of the "configureTopic" action to rename an integration table that is automatically created by an MQTT message. This is easy because you can rename the integration table using its MQTT topic.

  • This action cannot shrink the size of fields because this destroys data.

  • This action cannot rename fields because it breaks compatibility with bridges between services (integrations / configurations).

Request examples

Rename table

{
  "api": "hub",
  "apiVersion": "1.0",
  "requestId": "5",
  "action": "alterIntegrationTable",
  "params": 
  {
    "databaseName": "faircom",
    "tableName": "test2",
    "newTableName": "test3"
  },
  "authToken": "replaceWithAuthTokenFromCreateSession"
}
{
  "api": "hub",
  "apiVersion": "1.0",
  "requestId": "1",
  "action": "alterIntegrationTable",
  "params": 
  {
    "databaseName": "faircom",
    "tableName": "test2",
    "retentionPeriod": 2,
    "retentionUnit": "month",
    "metadata": {
      "tags": ["test", "deleteme"] 
    }
  },
  "authToken": "replaceWithAuthTokenFromCreateSession"
}
{
  "api": "hub",
  "apiVersion": "1.0",
  "requestId": "2",
  "action": "alterIntegrationTable",
  "params": {
    "databaseName": "faircom",
    "tableName": "test2",
    "alterFields": [
      {
        "name": "t10",
        "type": "VARCHAR",
        "length": 65500,
        "nullable": true
      }
    ],
    "retentionPeriod": 45,
    "retentionUnit": "day"
  },
  "authToken": "replaceWithAuthTokenFromCreateSession"
}
{
  "api": "hub",
  "apiVersion": "1.0",
  "requestId": "3",
  "action": "alterIntegrationTable",
  "params": 
  {
    "databaseName": "faircom",
    "tableName": "test2",
    "addFields": 
    [
      {
        "name": "new_field1",
        "type": "LVARCHAR",
        "nullable": true
      },
      {
        "name": "new_field2",
        "type": "TIMESTAMP",
        "nullable": true
      }
    ]
  },
  "authToken": "replaceWithAuthTokenFromCreateSession"
}
{
  "api": "hub",
  "apiVersion": "1.0",
  "requestId": "4",
  "action": "alterIntegrationTable",
  "params": 
  {
    "databaseName": "faircom",
    "tableName": "test2",
    "deleteFields": 
    [
      "new_field1", "new_field2"
    ]
  },
  "authToken": "replaceWithAuthTokenFromCreateSession"
}
{
  "api": "hub",
  "apiVersion": "1.0",
  "requestId": "5",
  "action": "alterIntegrationTable",
  "params": 
  {
    "databaseName": "faircom",
    "tableName": "test2",
    "transformName": "firstCreateMe"
  },
  "authToken": "replaceWithAuthTokenFromCreateSession"
}
{
    "result": {},
    "requestId": "1",
    "errorCode": 0,
    "errorMessage": "",
    "authToken": "replaceWithAuthTokenFromCreateSession"
}
{
    "result": {},
    "requestId": "5",
    "errorCode": 100,
    "errorMessage": "Not able to find integration table by name [test3/admin/faircom].",
    "authToken": "replaceWithAuthTokenFromCreateSession"
}
{
    "result": {},
    "requestId": "5",
    "errorCode": 100,
    "errorMessage": "Transform pipeline [firstCreateMe] was not found.",
    "authToken": "replaceWithAuthTokenFromCreateSession"
}
{
  "authToken": "replaceWithAuthTokenFromCreateSession",
  "result": {},
  "requestId": "00000001",
  "debugInfo": {
    "request": {
      "authToken": "replaceWithAuthTokenFromCreateSession",
      "api": "hub",
      "action": "alterIntegrationTable",
      "params": {
        "databaseName": "faircom",
        "tableName": "test1",
        "rebuildTable": false,
        "retentionPolicy": "doNotPersist",
        "retentionPeriod": 1,
        "retentionUnit": "day"
      },
      "requestId": "00000001",
      "debug": "max"
    }
  },
  "errorCode": 12006,
  "errorMessage": "Before you can change the data retention policy, period, or unit, you must use "alterIntegrationTable" and add "rebuildTable": true to the request. WARNING: rebuilding a table disrupts data collection while it rewrites all records."
}

Use the alterIntegrationTable JSON API action to alter settings for existing tables that are safe to modify

API actionsJSON hub APIJSON Actionintegration tables actionsalter integration tablealterIntegrationTable

The "params" property is an object that contains an action's parameters. Each action defines its own required and optional properties. See System limits for a comprehensive look at property requirements and limitations.

Table 1. "params" property summaries

Property

Description

Default

Type

Limits (inclusive)

alterFields

(optional) defines how to alter a field in a table

[]

array of objects

databaseName

(optional) specifies the name of a database

Defaults to the "defaultDatabaseName" value that is set during "createSession". If no default is set during "createSession", then "faircom" is used.

string

1 to 64 bytes

rebuildTable

(optional) rebuilds the table when set to true.

WARNING: Rebuilding a table interrupts data collection until the table is rebuilt.

false

Boolean

true
false

retentionPolicy

(optional) specifies how messages persist

"autoPurge"

string

"autoPurge"
"neverPurge"

retentionPeriod

(optional) specifies the number of retention units, which controls how long data is retained – see "retentionUnit"

4

integer

1 to 100

retentionUnit

(optional) purges expired messages each time this unit cycles – see "retentionPeriod"

"week"

string

"minute"
"hour"
"day"
"week"
"month"
"year"

sourcePayloadFormat

(optional) specifies the variant type format of the "source_payload" field

"binary"

string

"json"
"xml"
"binary"
"jpeg"
"siemensUdt"
"utf8"

tableName

(optional) specifies the name of a table

""

string

1 to 64 bytes

transformName

(optional) specifies the name of a transform process you have created. See the "createTransform" and "alterTransform" actions

""

string

1 to 64 bytes



The "databaseName" property is an optional string that specifies the database that contains the tables. It defaults to the database name supplied at login.

Note

In the API Explorer, "defaultDatabaseName" is set to "ctreeSQL" in the "createSession" action that happens at login.

  • A zero-length "databaseName" is invalid.

  • Its limits are from 0 to 64 bytes.

  • If the "databaseName" property is omitted or set to null, the server will use the default database name specified at login.

  • If no default database is specified during "createSession", "defaultDatabaseName" will be set to the "defaultDatabaseName" value that is specified in the services.json file.

The "rebuildTable" property is an optional Boolean that defaults to false. When true, the server stops accepting read, insert, and update requests and rebuilds the table. Rebuilding a table creates new table files and writes existing records to the new files.

Warning

Rebuilding a table interrupts data collection until the rebuilt operation finishes. The rebuild time is proportional to the number of table records.

Tip

You may want to rebuild the table in the following cases:

  • To apply a transform to previously inserted or updated records.

  • To change the table's retention policy, period, or unit.

  • To repair a data file corrupted by a storage failure.

The "retentionPeriod" property specifies how many units of data to retain. It must be an integer value from 1 to 100. It refers to the unit of time specified by the "retentionUnit" property — for example, if "retentionPeriod" is 14 and "retentionUnit" is "day", then message data is retained for 14 days. This property is optional.

  • Periodically, the system increments the age of each table partition.

    • "Minute" units are incremented at the zero second of each minute, whether the table partition was created 1 second before or 59 seconds before.

    • "Day" units are incremented at midnight GMT, not midnight of the local UTC time zone. A table partition becomes one day old at midnight GMT, whether it was created one second before or 23 hours and 59 seconds before.

    • "Week" units are incremented at midnight GMT on the last day of each week, even if the table partition was created one second before. The last day of the week is Saturday GMT.

    • "Month" units are incremented at midnight GMT on the last day of each month.

    • "Year" units are incremented at midnight GMT on Dec 31.

  • "retentionPeriod" implicitly calculates an upper bound on how many partitions are able to accumulate on your hard drive. However, partitions are not deleted just because this calculated number of partitions is reached. The system also does not restrict the deletion of all partitions.

  • If the FairCom database is shut down for a month, when the database is started up again, all partitions that are retained for less than one month are immediately deleted.

  • If someone purposely or accidentally sets the date to a future date, all partitions immediately become older, and auto-deletion ensues accordingly.

When partitions are auto-purged, some data are maintained "forever" in a global index. Auto-purge does not prevent these files from growing without bounds and filling up your hard drive.

If not specified, the default found in the services.json file is used. Initially, it is 4 (weeks).

Automatically purging data is important to ensure that retained data does not consume all storage and shut down the computer. The default value of 4 weeks allows FairCom's servers to store 1 TB of messages when 200 topics send one 2K message per second.

Note

  • If the value is not an integer from 1 to 100, FairCom's servers set it to the default value.

  • Smaller numbers improve SQL performance.

  • Each time the "retentionPeriod" cycles, FairCom's servers automatically and efficiently delete expired data.

  • The "retentionUnit" and "retentionPeriod" properties are used only when the "retentionPolicy" is set to "autoPurge".

  • FairCom's servers only use the "retentionPeriod" property when the "retentionPolicy" is "autoPurge".

  • The "retentionPeriod" can be changed to retain fewer or more messages. Changing it does not necessarily destroy existing data, but data may expire more quickly or be retained longer.

  • The "retentionPeriod" and "retentionUnit" properties control data granularity as well as the retention time.  In other words, "retentionPeriod" defines how many sets of data are stored, and "retentionUnit" defines how often data is purged.

    For example, if "rententionPeriod" is set to 14 , the server stores 14 sets of data. At the beginning of the 15th cycle, the server automatically purges the oldest set of data. If "retentionUnit" is set to day, then data will be purged daily. If set to "week", then data will be purged weekly.

  • The current calendar date affects purging.

    FairCom's servers automatically purge all retained data that has expired. This is noticeable when FairCom's servers come online after having been offline for a long time. When a server comes back online, it automatically purges all expired data.

    For example, if a FairCom server is offline for four weeks when it comes online, it will completely purge all retained data that has a retention time of less than 4 weeks.

The "retentionPolicy" property controls how messages are persisted. This property is optional.

If not specified, the default found in the services.json file is used. Initially, it is "autoPurge".

retentionPolicy values:
  • "autoPurge"

    This is the default. It is automatically applied when a new topic is created. It is preferred because it allows FairCom's servers to automatically remove messages that are older than the retention time. This helps ensure message data does not consume all storage space. It also minimizes storage costs and speeds up data access. The server partitions a table into multiple files so it can efficiently delete expired files.

  • "neverPurge"

    This stores messages on disk and never removes them. This is useful when you need the entire history of the message stream. If message velocity is high, this can consume all storage space and cause an outage. The server creates a non-partitioned table, which is slightly faster than a partitioned table because it stores all records in one file.

Each time this unit cycles, FairCom purges expired messages. For example, if you want a week's worth of messages to be purged once a week, set "retentionUnit" to "week" . This property is optional.

If not specified, the default found in the services.json file is used. Initially, it is "week"

  • This property is used in concert with "retentionPeriod" to determine retention time.

  • "retentionUnit" values:

    • "minute"

    • "hour"

    • "day"

    • "week"

    • "month"

    • "year"

    • "forever"

Note

  • For best performance, set the "retentionUnit" to a value that keeps "retentionPeriod" between 5 and 30

  • When you set "retentionUnit" property to "forever" the server will not purge messages. This setting is the same as setting "retentionPolicy" to "neverPurge".

  • The "retentionUnit" and "retentionPeriod" properties are used only when the "retentionPolicy" is set to "autoPurge".

The "sourcePayloadFormat" property is an optional string that specifies the expected data type of the MQTT payload. When omitted or set to null, it defaults to "binary". When the server receives an MQTT message, it always takes the binary value from the MQTT payload and stores it in the source_payload field. The source_payload field is a variant, which means it stores the type of data it contains. If there are no errors in validating the MQTT payload to match the type specified in "sourcePayloadFormat", the server assigns the value of "sourcePayloadFormat" as the variant field's type. 

Depending on the values of the "sourcePayloadFormat" and "validateMqttPayload" properties, the server may validate the MQTT payload to ensure it matches the type. The default value for the "validateMqttPayload" property is true. The server may also convert the MQTT payload into another form before storing it. If the MQTT payload does not match the type or cannot be converted, the server stores the payload in the source_payload field, sets its variant type to "binary", and sets the error field to 1 to indicate there is an error. It also adds an error entry to the JSON array in the log field. 

FairCom's MQTT server does the following when a topic receives an MQTT message and "sourcePayloadFormat" is set to one of the following values:

"sourcePayloadFormat" Value

Description

"binary"

The server stores the MQTT payload in the source_payload field with no validation or conversion and assigns the "binary" type to the field. It ignores the value of "validateMqttPayload".

"json"

The server stores the MQTT payload in the source_payload field.

  • If "validateMqttPayload" is true, it validates the MQTT payload as JSON.

    • If successful, it assigns the "json" type to the source_payload field.

    • If unsuccessful, the server assigns a "binary" type to the source_payload field and sets the error field to 1 to indicate there is an error. It also adds an error entry to the JSON array in the log field.

  • If "validateMqttPayload" is false, it does not validate the MQTT payload and assigns the "json" type to the source_payload field. The publisher must send MQTT messages containing valid JSON, or database queries may return errors.

"utf8"

The server stores the MQTT payload in the source_payload field.

  • If "validateMqttPayload" is true, it validates the MQTT payload as a UTF-8 string.

    • If successful, it assigns the "utf8" type to the source_payload field.

    • If unsuccessful, the server assigns a "binary" type to the field and sets the error field to 1 to indicate there is an error. It also adds an error entry to the JSON array in the log field.

  • If "validateMqttPayload" is false, it does not validate the MQTT payload and assigns the "utf8" type to the source_payload field. The publisher must send MQTT messages containing valid UTF-8, or database queries may return errors.

"variantObject"

The server validates the payload as a variant object.

  • If successful, the server extracts the value from the variant object’s "value" property and assigns it to the value of the source_payload field. It also assigns the variant object’s type to the type of the source_payload field.

  • If unsuccessful, the server writes the MQTT payload to the source_payload field, assigns a  "binary"  type to the field, and sets the error field to  1, to indicate that it also adds an error entry to the JSON array in the log field.

The server ignores the value of "validateMqttPayload". The server always validates "variantObject".

"siemensUdt"

The server stores the MQTT payload in the source_payload field without validation or conversion and assigns the "binary" type to the field. It ignores the value of "validateMqttPayload".

"jpeg"

The server stores the MQTT payload in the source_payload field without validation or conversion and assigns the "binary" type to the field. It ignores the value of "validateMqttPayload".

"xml"

The server stores the MQTT payload in the source_payload field without validation or conversion and assigns the "binary" type to the field. It ignores the value of "validateMqttPayload".

  • This property is a hint to the server about the format and type of the MQTT message payload.

  • This property does not cause server to validate the MQTT payload to see if it matches the type you set. The server stores the payload as is without validation. For example, setting the type to "json", it does not stop the server from receiving and storing a non-JSON value or invalid JSON document in the source payload.

  • The FairCom Edge Explorer application may use the value of this property to determine the default way to display a topic's payload.

  • The transform engine may use the value of this property to help it transform the source payload.

The "tableName" property contains the name of the table in the database where the event occurred. It is a non-zero-length string.

See table name in System limits for table naming requirements and limitations.

A table in DBnotify is defined by "databaseName", "ownerName" and "tableName" or by "dataFilePath".

The "transformName" property is an optional string that contains the unique name of a transform process, which consists of one or more transform steps.

The value of "transformName" must be the name of a transform process you created. It cannot be the name of a FairCom-provided transform method.

You must create a transform before you can assign it to an integration table. See the "createTransform" and "alterTransform" actions.

  • A transform is a process that works like a pipeline where the output of one transformation can become the input for another transformation.

  • The following actions use the "transformName" property to assign a transform to an integration table:

    • "configureTopic"

    • "createInput"

    • "alterInput"

    • "createIntegrationTable"

    • "alterIntegrationTable"