FairCom Edge IoT Database

Replication with FairCom Edge

In the IoT it is very common to stream data to the cloud for storage and other processing, such as running reports. FairCom Edge allows you to store and process data on the edge device, saving the time it takes to move data to the cloud. It also has the benefit replicating data to the cloud.

For full documentation see the Replication Manager User Guide.

In This Chapter Cloud Replication Source Server Target Server Replication Manager Ports

Cloud Replication

Replicating from FairCom Edge to the cloud consists of three pieces:

1. A source server on the edge collecting data into one or more tables.
2. A target server in the cloud receiving all the data changes from the source server.
3. A replication process that connects to both and moves the data between them.

Replication differs from streaming in that, if a connectivity outage occurs, replication will pick up where it left off when the outage is over and no data will be skipped. With streaming, data generated during the outage most often is lost. Replication also moves data in the same transaction chunks that were originally added to the source server, preserving atomicity on the target server.

You can configure your replication using graphical tool, the FairCom Replication Manager. The configuration is stored in the Memphis database and propagated throughout the system.

To simplify setup, FairCom Edge and the files it creates are replication-aware.

Source Server

Install a FairCom server on the source machine (this can be FairCom DB, FairCom Edge, or c-treeRTG). See the FairCom Installation Guide for detailed steps on installing and configuring a server.

To enable replication, you must do either of the following:

• Specify REPLICATE in the MQTT (Message Queuing Telemetry Transport) persist JSON data (see examples in Node-RED Using the FairCom Edge MQTT Broker and Python and FairCom Edge MQTT).

  OR

• Add the following line to the ctsrvr.cfg file, replacing "<table name>" with the name of the table(s) you wish to replicate:
  REPLICATE ./ctreeSQL.dbs/<table name>.dat

If you wish to replicate more than one table, you can add multiple lines or use wildcards such as "*.dat". The REPLICATE keyword identifies tables that will be replicated for the source server. The path is relative to the server's data folder. Tables created via MQTT persistence will be located in data/ctreeSQL.dbs/ by default.

Add REPL_SRLSEG_ALLOW_UNQKEY YES to the ctsrvr.cfg file. All tables to be replicated must have a unique index. This keyword allows simple tables to be replicated by using our built in identity index.

Connect the source server to the Replication Manager server by editing the /server/config/ctagent.json file. Change the "memphis_host": "localhost", line by replacing "localhost" with the IP of the target server.

Start the source server.

Target Server

Install a FairCom Server on the target machine.

If this server is on the same machine as the source server (e.g., a development environment), you will have to change its name. Look for the SERVER_NAME keyword in the ctsrvr.cfg file in the config folder. Make sure this name is unique (we sum up the ASCII equivalent of the name, so SRV1 and 1SRV will be identical, which will cause problems).

Replication Manager

If you have not already done so, you will need to install Replication Manager, as described in the Replication Manager User Guide.

Open a web browser and point it to this URL:

https://<target server IP>:8443/ReplicationManager/

You should see the login page for the Replication Manager. The user name and password should be filled in automatically.

User: "admin"

Password: "ADMIN"

You should see the Replication Manager load. Verify that both the source and target servers appear.

Drag a line between the FairCom Edge source server and the target server. You will be warned that the source server must push the changes, this is normal for replication from a FairCom Edge server.

Create a new publication with the table from the source server.

If you want all data to stay on the target server, make sure Ignore Partition is checked under Advanced Options.

Subscribe to the publication just created.

Right-click on the publication line and select Deploy.

You should see the table being created on the target server and the data moving over to it.

Ports

If there is a VPN or firewall between any of the machines you will have to provide network access from the machine the Replication Agent is running on to both the target and source server. The port you will need to open is the ISAM (NoSQL) port each server is using. The port the server uses is based on the server name. For the default name of FAIRCOMS, the port will be 5597. For other server names, check the ctstatus.fcs file in the servers "data" directory for the port in use by searching for a line similar to:

"Server using TCP/IP Socket Port Number: 5597"

See the Replication Manager Guide for more about installing, configuring, and using replication.