"rerunIntegrationTableTransformSteps"
Rerun transform steps after updating them
The "rerunIntegrationTableTransformSteps" action causes the server to rerun an integration table's transform steps. You can limit its scope to specific records or all existing records. The "scope" property is required.
This action is valuable when transform steps are added or altered after records have been inserted. Benefits include:
Experimentation: Observe the behavior of various transform steps on your dataset.
Bug Resolution: Easily correct a bug in a transform step and re-run the steps to reprocess the original data with the corrected transform, negating the need to undo any prior damage.
Data Correction: Fix problematic data in the
source_payloadfield.Resilience: If an interruption occurs during the re-running of transform steps (e.g., power failure), you can simply restart the process.
To ensure this action runs successfully, you should preserve the original value of the source_payload field.
In an Integration Table, the
source_payloadfield contains the original event data from connectors, MQTT clients, SQL inserts, the JSON DB"insertRecords"action, etc.Before inserting the record, the server runs transform steps that read the
source_payloadfield and write to other fields in the table and perhaps other tables.Typically, a transform step does not overwrite the original data in the
source_payloadfield; otherwise, it would destroy the original data, change the historical record, break the audit trail, and potentially prevent the"rerunIntegrationTableTransformSteps"and"testIntegrationTableTransformSteps"actions from running successfully.Preserving the original data in the
source_payloadfield allows you to rerun transform steps as many times as you want.If data in the
source_payloadfield has an incorrect value, you can create transform steps to fix it. The first transform step can back up the value of thesource_payloadfield by copying it into a property in themetadatafield. The next transform step can correct the problem and update the original value in thesource_payloadfield. You can test the fix using the"testIntegrationTableTransformSteps"action. You can apply the fix using the"rerunIntegrationTableTransformSteps"action.
Note
Updating a large number of existing records may take significant time and may slow the intake of newly inserted records.
This action is not atomic. If it fails part way through, some records will be updated and some will not. In this case, you must rerun this action to finish the process. Thus, it is important that the source_payload field contains good data so the transform steps will run properly.
Use the "rerunTransformScope" property (required) to define the scope of this action:
"allRecords"returns immediately after starting a background thread that runs the transform steps on all records in the specified table."stop"stops the table's background thread that reruns transform steps on all its records. If there is no background thread, the action does nothing and returns an error."firstRecord"returns the first record's fields before and after being modified by the transform steps. It runs synchronously and does not persist any changes."lastRecord"returns the last record's fields before and after being modified by the transform steps. It runs synchronously and does not persist any changes."specificRecords"returns the specified record's fields before and after being modified by the transform steps. It runs synchronously and does not persist any changes.
Limitations on the "allRecords" scope:
The server runs one
"allRecords"scope at a time. This applies to both the"rerunIntegrationTableTransformSteps"and"testIntegrationTableTransformSteps"actions.While running transform steps on all records in an integration table, you cannot alter the structure of the table.
There are a variety of times when you want to run transform steps on existing records.
A bug in a transform step may cause records to contain bad data. After you fix the bug, you may want to apply the fix to existing records.
If you enhance transform steps, you may want to apply the enhancement to existing records.
If you alter table fields, you may want to rerun the transform steps on existing records to take advantage of new field capabilities. For example, you may change an integer field to a double so it can store floating point numbers.
If you disable the transform steps, you may later want to enable them and apply the changes to existing records.
A device sometimes produces bad data. After you fix that data, you may want to run the transform steps on existing records to reprocess data changes.
Updating existing records cannot be undone, but it can be stopped.
The action runs asynchronously on all records; thus, it returns immediately.
It starts with the first record in the table and proceeds to the last.
After running this action, you can view or query the initial records in the table to verify it is working as expected. If not, you can stop the action.
You can stop the action when the specified table is currently processing existing records by calling this action with
"rerunTransformScope": "stop". This stops the action from processing the remaining records, but it cannot undo the changes it has already performed.
Before running this action, use the
"testIntegrationTableTransformSteps"action to verify the transform steps work. This is important because this action modifies thousands of records a second and these changes cannot be undone.To get the status of the
"rerunIntegrationTableTransformSteps"action, run the"describeIntegrationTables"action on the integration table. When this action is actively processing existing records, it returns"transformStatus": "running"and a"transformPercentComplete"property set to an integer between1and99.Avoid running the
"alterIntegrationTable"action while this action is running because it may change what subsequent transform steps do.The
"logTransformOverwrites"property, whentrue, causes the action to log attempts by JavaScript to overwrite a field value.
Request Examples
Minimal example stops rerunning transform steps on existing records
The following example stops a background thread that is currently rerunning transform steps on records in the myTable1 table. It defaults to using the database and owner specified in the current JSON action session (see "createSession" and "alterSession"). It returns an error if a background thread is not currently rerunning transform steps on the records in myTable1.
{
"api": "hub",
"action": "rerunIntegrationTableTransformSteps",
"params": {
"tableName": "myTable1",
"rerunTransformScope": "stop"
},
"authToken": "replaceWithAuthTokenFromCreateSession"
}The following minimal example reruns the transform steps on the earliest inserted record in myTable1 that has not been purged by the retention policy. It uses the database and owner specified in the current JSON action session (see "createSession" and "alterSession").
{
"api": "hub",
"action": "rerunIntegrationTableTransformSteps",
"params": {
"tableName": "myTable1",
"rerunTransformScope": "firstRecord"
},
"authToken": "replaceWithAuthTokenFromCreateSession"
}
The following example reruns the transform steps on the most recently inserted record in myTable1. It uses fully qualified table names that specify the database and owner of the table.
{
"api": "hub",
"action": "rerunIntegrationTableTransformSteps",
"params": {
"databaseName": "myDb1",
"ownerName": "myOwner1",
"tableName": "myTable1",
"rerunTransformScope": "lastRecord"
},
"authToken": "replaceWithAuthTokenFromCreateSession"
}
The following example reruns the transform steps assigned to myTable1 on all existing records.
{
"api": "hub",
"action": "rerunIntegrationTableTransformSteps",
"params": {
"tableName": "myTable1",
"rerunTransformScope": "allRecords"
},
"authToken": "replaceWithAuthTokenFromCreateSession"
}
The following example reruns the transform steps on the records in myTable1 with the specified identifiers.
{
"api": "hub",
"action": "rerunIntegrationTableTransformSteps",
"params": {
"tableName": "myTable1",
"rerunTransformScope": "specificRecords",
"ids": [1,3]
},
"authToken": "replaceWithAuthTokenFromCreateSession"
}
"params" property summariesProperty | Definition | Default | Type | Limits (inclusive) | |||||
|---|---|---|---|---|---|---|---|---|---|
(optional) specifies the database of the source table where this action runs the transform steps. | Defaults to the | string | 1 to 64 bytes | ||||||
(conditional) lists identifiers that specify a table row, indicating which records the action affects. | Required when the | array | 0 or more identifiers | ||||||
(optional) specifies the owner account of the source table where this action runs the transform steps. | Defaults to the | string | 1 to 64 bytes | ||||||
defines the scope of this action. | Required - No default value | string enum |
| ||||||
specifies the name of the source table where this action runs the transform steps. | Required - No default value | string | 1 to 64 bytes |
The optional "databaseName" property specifies the database of the source table where this action runs the transform steps.
When set to null or omitted, the server uses the default "databaseName" defined in the "createSession" action. For more information, see the "createSession" action and the "defaultDatabaseName" property.
Note
The fully qualified name of a table includes the database, owner, and table names.
The "ids" property is an array. Each identifier in the array uniquely specifies a table row, indicating which records the action affects. It is required when the "rerunTransformScope" property is set to "specificRecords".
An integration table's id field contains the unique identifier of each record.
The optional "ownerName" property specifies the owner account of the source table where this action runs the transform steps.
When set to null or omitted, the server uses the default "ownerName" defined in the "createSession" action. For more information, see the "createSession" action and the "defaultOwnerName" property.
Note
The fully qualified name of a table includes the database, owner, and table names.
The required "rerunTransformScope" property defines the scope of this action. It is a string enum with one of the following values:
"allRecords"returns immediately and asynchronously creates a background thread that runs the transform steps on all records in that table.You can stop this action by calling the
"rerunIntegrationTableTransformSteps"action with"rerunTransformScope": "stop".The server regularly updates the
"transformStatus"and the"transformPercentComplete"properties, which are returned by the"describeIntegrationTables"action.If an integration table's background thread is already rerunning the transform steps, the action returns an error indicating the
"rerunIntegrationTableTransformSteps"action is already running on the integration table. The user can decide if he wants to wait or stop the action.
"stop"stops the table's background thread that reruns transform steps on all its records. If the background thread is not running, the action does nothing and returns an error."firstRecord"immediately and synchronously runs the transform steps on the first record in the table.This option does not interfere with a background thread that is currently rerunning the transform steps.
"lastRecord"immediately and synchronously runs the transform steps on the last record in the table.This option does not interfere with a background thread that is currently rerunning the transform steps.
"specificRecords"immediately and synchronously runs the transform steps on the specified records in the table.Each record has an
"id"field containing an integer number that uniquely identifies it.The action uses the
"ids"property to specify the identifiers of one or more records to be transformed.This option does not interfere with a background thread that is currently rerunning the transform steps.
The required "tableName" property specifies the name of the source table where this action runs the transform steps.
Note
The fully qualified name of a table includes the database, owner, and table names.