Documentation

c-treeEDGE IoT Database

Previous Topic

Next Topic

Replication with c-treeEDGE

In the IoT it is very common to stream data to the cloud for storage and other processing, such as running reports. c-treeEDGE allows you to store and process data on the edge device, saving the time it takes to move data to the cloud. It also has the benefit replicating data to the cloud.

Cloud Replication

Replicating from the edge to the cloud consists of three pieces:

  1. A source server on the edge collecting data into one or more tables.
  2. A target server in the cloud receiving all the data changes from the source server.
  3. A replication process that connects to both and moves the data between them.

Replication differs from streaming in that, if a connectivity outage occurs, replication will pick up where it left off when the outage is over and no data will be skipped. With streaming, data generated during the outage most often is lost. Replication also moves data in the same transaction chunks that were originally added to the source server, preserving atomicity on the target server.

You can configure your replication using a graphical tool, the Replication Manager. The configuration is stored in the Memphis database and propagated throughout the system. To simplify setup, c-treeEDGE and the files it creates are replication-aware.

For full documentation see the Replication Manager Guide in the FairCom online documentation.

Source Server

Install a c-treeEDGE database on the source machine. See the c-treeEDGE Administration Guide for detailed steps on installing and configuring a server. To enable replication, you must do either of the following:

  • Specify REPLICATE in the MQTT persist JSON data (see examples in Node-RED Using the c-treeEDGE MQTT Broker and Python and c-treeEDGE MQTT).

    OR

  • Add the following line to the ctsrvr.cfg file in the source server's directory, replacing "<table name>" with the name of the table(s) you wish to replicate:
    REPLICATE ./ctreeSQL.dbs/<table name>.dat

If you wish to replicate more than one table, you can add multiple lines or use wildcards such as "*.dat". The REPLICATE keyword identifies tables that will be replicated for the source server. The path is relative to the server's data folder. Tables created via MQTT persistence will be located in data/ctreeSQL.dbs/ by default.

Add REPL_SRLSEG_ALLOW_UNQKEY YES to the ctsrvr.cfg file. All tables to be replicated must have a unique index. This keyword allows simple tables to be replicated by using our built in identity index.

Connect the source server to the Replication Manager server by editing the /server/config/ctagent.json file. Change the "memphis_host": "localhost", line by replacing "localhost" with the IP of the target server.

Start the source server.

Target Server

Install a c-treeACE Server on the target machine. If this server is on the same machine as the source server, you will have to change its name. Look for the SERVER_NAME keyword in the ctsrvr.cfg file in the server folder. Make sure this name is unique (we sum up the ASCII equivalent of the name, so SRV1 and 1SRV will be identical, which will cause problems).

Replication Manager

Open a browser window and point it to this URL:

https://<target server IP>:8443/ReplicationManager/

You should see the Replication Manager app load. Verify that both servers appear.

ops_mgr01

Drag a line between the c-treeEDGE source server and the c-treeACE target server. You will be warned that the source server must push the changes, this is normal for replication from a c-treeEDGE server.

Create a new publication with the table from the source server.

ops_mgr03

If you want all data to stay on the target server, make sure Ignore Partition is checked under Advanced Options.

ops_mgr04

Subscribe to the publication just created.

ops_mgr05

Right-click on the publication line and select Deploy.

ops_mgr06

You should see the table be created on the target server and the data moved over.

Ports:

If there is a VPN or firewall between any of the machines you will have to provide network access from the machine the Replication Agent is running on to both the target and source server. The port you will need to open is the ISAM (NoSQL) port each server is using. The port the server uses is based on the server name. For the default name of FAIRCOMS, the port will be 5597. For other server names, check the ctstatus.fcs file in the servers "data" directory for the port in use by searching for a line similar to:

"Server using TCP/IP Socket Port Number: 5597"

See the Replication Manager Guide for more about installing, configuring, and using replication.

TOCIndex