Using Auto-Purge

The Automatic Data Aggregation family of features simplifies the collection and aggregation of sensor data. Many real-time sensors generate large streams of data, which can quickly fill available storage on smaller devices. The Auto-Purge feature can purge the oldest data to make room for new data. Before purging, the data can be sent to the cloud for storage — depending on your environment, either the entire set of raw data can be streamed to the cloud or, if bandwidth is a consideration, a smaller set of aggregated values can be sent.

c-tree leverages the advanced partitioning capabilities of the FairCom Database Engine to accomplish this. Partitioning allows you to divide a database into smaller partitioned files. Records are inserted into the database and sorted into the partitions based on defined criteria, such as time or date. For example, for sensors producing many records, you can set a partition to accommodate an hour's worth of records: Each hour, a new partition is created to hold that hour's records.

The Auto-Purge feature allows automatic purging of data so the storage available on your device does not fill up. Using this feature, you might purge aged data so that (for example) only 24 hours of current data are kept on the device. In setting each partition to hold data for 1 hour, 24 individual file partitions are created on the device. As soon as the 25th partition is begun, the 1st partition is purged. When the 26th partition is begun, the 2nd partition is purged. And so on. The data collection demands of your application determine how much data is stored in a partition and how many partitions to keep.

In the case of "time series" data, where a new record is constantly inserted into the table (such as a stream of sensor data), new partitions will be created constantly, and Auto-Purge will constantly remove old partitions. If you stop inserting records, the old partitions will never be removed.