FairCom DB Notify
FairCom DB Notify and FairCom MQ non-secure and secure communications - tutorial
Use non-secure or secure communications between FairCom DB Notify and FairCom MQ tutorial
FairCom DB Notify allows you to define "event criteria '' on any FairCom DB database file. The file is actively monitored and a FairCom DB Notify processing event is triggered when the criteria are satisfied. When triggered, FairCom DB Notify sends a configurable MQTT message to a specified MQTT Broker.
This tutorial describes how to set up these event criteria either with non-secure communication or with secure 'dbnotify' communications between a FairCom DB Server and a FairCom MQ broker using OpenSSL and x509 authentication.
Ensure that FairCom DBand FairCom MQ are installed and running.
Note
To run both products on the same computer, remap the ports used by one or both of the products. Change SQL_PORT (ctsrvr.cfg) and the enabled ports in services.json
. See: Run multiple server instances on one computer.
Configuration
Configure FairCom DB Notify with or without TLS secure communications.
Enable the 'dbnotify' Service for the FairCom DB server by editing the
<faircom>\config\services.json
configuration file and changing the"enabled"
setting fromfalse
totrue
."otherServices": [ { "serviceName": "dbnotify", "serviceLibrary": "./dbnotify/fcdbnotify.dll", "enabled": true },
Edit the
<faircom>\config\dbnotifyconnections.json
file to configure a connection between your FairCom DB server and FairCom MQ. Set the"faircomServerName"
to the name used by FairCom MQ. This defaults to"FCEDGEMQ"
. You can verify this by looking for'SERVER_NAME'
in the<faircom>/config/ctsrvr.cfg
file.{ "logLevel": "development", "directConnectionsToFairComMq": [ { "brokerConnectionName": "brokerCtree", "faircomServerName": "FCEDGEMQ", "brokerHostname": "localhost", "brokerUserName": "ADMIN", "brokerUserPassword": "ADMIN", "reconnectFrequencySeconds": 15, "metadata": {}, "defaultIncludePrimaryKey": "never", "defaultTagChanges": "neverTag", } ] }
Edit the
<faircom>\config\dbnotify\ctreeSQLcustmast.json
file to configure the DB notification criteria.When the DB Server enables the
'dbnotify'
service at startup, it looks in the<faircom>\config\dbnotify
folder for any.json
files. Each.json
file is expected to contain the necessary information related to one table to be monitored. A separate.json
file is needed for each table you want to monitor.Set the "databaseName" to match the database the table to be monitored is in. For this tutorial, it is "ctreeSQL".
Set the "tableName" to match the table you want this file to apply to. In this tutorial, it is "custmast".
Set the "ownerName" to "admin". This should match the user that created the table.
Inside the "publishMqttMessages" array add an entry for each type of message you want to receive. Each entry can use one or more trigger types and include different fields and settings to create a specific type of message.
Set the "topic" to the MQTT topic you want this message type to notify on. For this tutorial, use "topic1".
Set "brokerConnectionName" to the same name used above when defining the connection to FairCom MQ, "brokerCtree".
Set "triggers" to an array of event types you want this message type to notify on.
"QoS" is the quality of service FairCom MQ will use to publish the notification messages. This example uses QoS 0.
"recordFilter" is an optional expression used to identify a subset of records to notify on. If left out it will notify on all records for the given triggers.
"includedFields" is an optional array of field names to include in the notification. If left out all fields will be included.
{ "databaseName": "ctreeSQL", "tableName": "custmast", "ownerName": "admin", "publishMqttMessages": [ { "topic": "topic1", "brokerConnectionName": "brokerCtree", "triggers": [ "delete"], "QoS": 0, "recordFilter": "!stricmp(cm_custstat,\"CA\")", "includedFields": [ "cm_custnumb", "cm_custzipc", "cm_custcity" ], "includePrimaryKey": "never" }, { "topic": "topic1", "brokerConnectionName": "brokerCtree", "triggers": [ "insert", "update"], "QoS": 0, "recordFilter": "atoi(cm_custnumb) >= 1000", "includePrimaryKey": "never", "tagChanges": "tagEachField" } ] }
Restart the DB Server.
Notice
The server is now set up with DB Notify. You can proceed to Test DB notify and perform Troubleshooting if necessary, or you can setup DB Notify with TLS/X509 configuration first.
Complete these steps to secure TCP/IP communications between the DB Notify service and the FairCom DB broker. This involves activating TLS/SSL to encrypt the network communications and activating x509 authorization to secure the user and password used for this connection.
x509 authentication extends SSL support to allow database user authentication based on a client-provided certificate that is trusted by the server, and a user that is registered in the server or is listed in the broker's authorized user database. Typically, users are added using the 'ctadmn'
utility.
TLS is an updated, more secure version of SSL. The terms SSL and TLS are used synonymously throughout this document.
Create certificates
Set up FairCom MQ with SSL and x509
Set up FairCom DB Notify with SSL and X509
SSL communications and x509 authorization require the following certificates and keys.
Names | Description |
---|---|
'ctree_ssl.pem' | The FairCom MQ Broker (server-side) certificate file. This |
'ctsrvr.pem' | The Client-side certificate file. This file is used by the 'dbnotify' service to secure the TCP/IP communication. This is |
'rootca.pem'/'rootca.key' | The Certificate Authority(CA) certificate and key file pair. These files are used to sign both client and server certificates. The |
'admin.pem'/admin.key' | The FairCom ADMIN user certificate and private key pair. Each unique database user, including the FairCom ADMIN user, needs their own certificate and private key pair, signed by a trusted Certificate Authority(CA). This pair is intended for the "admin" account. The |
'JonDoe.pem'/JonDoe.key' | A typical (non-admin) user certificate and private key pair. Each unique database user needs their own certificate and private key pair, signed by a trusted Certificate Authority(CA). This is an example of a "JonDoe" account. Later, you will modify our script to create your own user certificates. |
Sample certificates for use with this demo in addition to a 'createCerts.bat'
script can be found in the <faircom>\config\x509.auth.demo
folder.
The createCerts.bat
script first creates a Certificate Authority(CA) and then uses this CA to sign the other certificates. See the openssl '-subj'
switch in this script where later you should substitute your company information. Near line 75, shows the 'JonDoe' user. Keep this intact to test this demo, then later go back and adjust as you see fit for your company and users.
Use our provided sample certificates, or use the createCerts.bat
script to create your own. The createCerts.bat
script only needs the 'rootca.conf'
onfiguration file. If you desire to create your own certificates, copy these two files to any working directory you choose. If you run createCerts.bat
in the x509.auth.demo
folder, you will get error messages for conflicting with our existing certificate files and keys.
The createCerts.bat
file also requires the OpenSSL.exe utility. Be sure the utility is accessible from your command prompt before running createCerts.bat
.
Execute the createCerts.bat
script from your Windows command prompt. Set a 'PEM pass phrase' when prompted. We used 'FairComQA'
, but you may choose your own phrase.
> createCerts.bat
You will see:
> create root CA cert request and private key > Enter PEM pass phrase: FairComQA
The 'pass phrase' for the Certificate Authority(CA) certificate is set, and the CA certificate files are created. You are prompted when this CA certificate is being used to sign the other certificates.
The certificate files (.pem) and keys (.key) names are displayed as they are created.
ctree_ssl.pem ctsrvr.pem admin.key admin.pem JonDoe.key JonDoe.pem
Copy certificates
Copy the 'ctree_ssl.pem' file (server-side certificate) you generated above into your Broker's server folder:
> copy .\<YourWorkFolder>\ctree_ssl.pem .\<faircom>\server\ctree_ssl.pem
Note: The
'ctree_ssl.key'
file does not need to be specified because it is already embedded in'ctree_ssl.pem'
.Copy the
'ctsrvr.pem'
file (client-side certificate) into your client Apps folder. In this tutorial, we will use our./tools
folder in our Broker's package.> copy .\<YourWorkFolder>\ctsrvr.pem .\<faircom>\tools\ctsrvr.pem
Copy the user certificate and key files you created above. Remember, we made an 'admin' and a 'JonDoe' user. Copy these files into the
./tools
folder.> copy .\<YourWorkFolder>\admin.pem .\<faircom>\tools\admin.pem > copy .\<YourWorkFolder>\admin.key .\<faircom>\tools\admin.key > copy .\<YourWorkFolder>\JonDoe.pem .\<faircom>\tools\JonDoe.pem > copy .\<YourWorkFolder>\JonDoe.key .\<faircom>\tools\JonDoe.key
Disable shared memory if testing the client on the same computer as the server.
Deactivate the Shared Memory Protocol even if you are connecting with TCP/IP, internally. If the Shared Memory protocol is activated, the Shared Memory is automatically switched when possible. To test your TCP/IP TLS/SSL it is best to deactivate Shared Memory by commenting out its keyword in the
ctsrvr.cfg
file.; Communication protocols ; COMM_PROTOCOL FSHAREMM
Deny shared memory for SQL also by adding the following line:
SQL_OPTION NO_SHARED_MEMORY
Activate SSL by editing FairCom MQ Broker's
ctsrvr.cfg
file in the <faircom>\config
folder.Uncomment the SSL and x509 support.
;Here is where you can activate (un-comment) SUBSYSTEM COMM_PROTOCOL SSL { ;This is the file name in your server's directory SERVER_CERTIFICATE_FILE ctree_ssl.pem ;For SSL you can specify (un-comment) a debug log file name DEBUG_LOG ssl.log ;Here you can restrict access to SSL ONLY. SSL_CONNECTIONS_ONLY YES ;Require clients to provide a x509 certificate VERIFY_CLIENT_CERTIFICATE YES ;Use x509 client certificate for database authentication x509_AUTHENTICATION YES ;Use the SUBJECT:CN from the client's certificate as their username x509_PATH CN ;SSL_CIPHERS AES256-SHA256:AES256-GCM-SHA38:DHE-RSA-AES256-SHA256:AES256-GCM-SHA384 }
Update
"stopserver"
in the <faircom>\server
folder to stop the server using SSL.#!/bin/sh echo Stopping the FairCom Database Engine... CTSSL_CLIENT_CERTIFICATE=admin.pem export CTSSL_CLIENT_CERTIFICATE CTSSL_CLIENT_KEY=admin.key export CTSSL_CLIENT_KEY cd ../tools ./ctstop -AUTO none none "FCEDGEMQ@localhost^fssltcp"
Run commandline utilities found in the <faircom>/tools
folder to test that FairCom MQ is enabled with SSL/x509.
First, restart Faircom MQ.
The message
"Initialized SSL support."
appears inCTSTATUS.FCS
.Check that in the server's configuration file (ctsrvr.cfg), only SSL connections are allowed.
SSL_CONNECTIONS_ONLY YES
Set environment variables to set up the SSL user as 'admin'.
set CTSSL_CLIENT_CERTIFICATE=admin.pem set CTSSL_CLIENT_KEY=admin.key
Append
^fssltcp
to the server name to request an SSL-enabled connection for an ISAM connection.ctixmg ADMIN ADMIN "FCEDGEMQ@localhost^fssltcp" ctadmn ADMIN ADMIN "" "FCEDGEMQ@localhost^fssltcp" ctixmg none none "FCEDGEMQ@localhost^fssltcp"
Be sure to use quotes on the command line.
It is not necessary to send user/password because the x509 certificate supplies the necessary user and password credentials.
Run ctadmn to check that the communication is using SSL.
ctadmn none none "" "FCEDGEMQ@localhost^fssltcp" **** FairCom(R) Server Administration Utility **** 1. User Operations 2. Group Definitions 3. File Security 4. Monitor Clients Enter your choice (1-10), or 'q' to quit>> 4
Select 4 to "Monitor Clients".
Monitor Clients: 1. List Attached Clients 2. Kill Client Enter your choice (1-2), or 'q' to return to previous menu>> 1
Select 1 to "List Attached Clients":
You might have to press 'enter' a few times because our 'ctixmg' might be last on the list after other internal MQ threads.
UserID: ADMIN NodeName: Task 19 Communications: FSSLTCP <<<<< SEE SSL Memory: 163K Open Files: 3 Logon Time: 2:51 Tran Time: -- Rqst Time: 2:49 NoRequest Rqst# 196 OPNRFIL
If you see Communications: FSSLTCP as in this example, it means that the TCP/IP connection is using SSL.
Note: The following protocols are some of the more common TCP/IP connections you might observe:
F_TCPIP indicates an unencrypted ISAM TCP/IP connection. FSSLTCP indicates an SSL-enabled ISAM TCP/IP connection. SQL_TCPIP indicates an unencrypted SQL TCP/IP connection. SQL_SSLTCP indicates an SSL-enabled ISAM TCP/IP connection.
Enter “q” to return to the
ctadmn
main menu.Add the 'Jon Doe' user into your Server/Broker's database.
User Operations: 1. Add New User 2. Remove Existing User 3. List Users Enter your choice (1-9), or 'q' to return to previous menu>> 1
Select option 1, “Add New User”.
Enter User Id >> Jon Doe Enter User Description >> Jon Doe
Answer the remaining prompts as you prefer. After the last prompt, press “q” to return to the main menu. Press "q" again to exit
ctadmn
.Set two environment variables to specify the user (JonDoe) with x509 authentication.
set CTSSL_CLIENT_CERTIFICATE=JonDoe.pem set CTSSL_CLIENT_KEY=JonDoe.key
Run your apps, without a user or password.
ctixmg none none "FCEDGEMQ@localhost^fssltcp"
Edit the <faircom>
\config\dbnotifyconnections.json
file to configure a secure connection between your FairCom DB server and the FairCom MQ Broker:Set the "faircomServerName" to "FCEDGEMQ". This is the 'SERVER_NAME' of the MQ broker that is set in the './config/ctsrvr.cfg' file.
Enable "tls" and "x509_authentication".
At "tls", set the "certificateFilename" with the name of the client-side certificate file that you previously created.
At "x509_authentication", set "certificateFilename" and "privateKeyFilename" with the FairCom ADMIN user certificate key and private key you previously created.
{ "logLevel":"development", "directConnectionsToFairComMq":[ { "brokerConnectionName":"brokerCtree", "faircomServerName":"FCEDGEMQ", "brokerHostname":"localhost", "reconnectFrequencySeconds":15, "tls":{ "certificateFilename":"ctsrvr.pem", "enabled":true, "x509_authentication":{ "certificateFilename":"admin.pem", "privateKeyFilename":"admin.key", "enabled":true } } } ] }
Encrypt the
dbnotifyconnections.json
file for secure connection and communication.Rename
dbnotifyconnections.json
todbnotifyconnections.cfg
Execute
"ctcmdset dbnotifyconnections.cfg"
to get an encrypteddbnotifyconnections.set
file.Rename
dbnotifyconnections.set
todbnotifyconnections.json
Edit the <faircom>
\config\dbnotify\ctreeSQLcustmast.json
file to configure the DB notification criteria.When the DB Server enables the 'dbnotify' service at startup, it looks in the
config\dbnotify
folder for any.json
files. Each.json
file is expected to contain the necessary information related to one database file to be monitored. A separate.json
file is needed for each database file you want to monitor.{ "databaseName":"ctreeSQL", "tableName":"custmast", "ownerName":"admin", "publishMqttMessages":[ { "persistenceDatabaseName":"faircom", "persistenceOwnerName":"admin", "persistenceTableName":"mqtt_msg_topic1", "brokerConnectionName":"brokerCtree", "triggers":["delete"], "QoS":0, "recordFilter":"!stricmp(cm_custstat,\"CA\")", "includedFields":[ "cm_custnumb", "cm_custzipc", "cm_custcity" ], "includePrimaryKey":"never" }, { "persistenceDatabaseName":"faircom", "persistenceOwnerName":"admin", "persistenceTableName":"mqtt_msg_topic1", "brokerConnectionName":"brokerCtree", "triggers":["insert","update"], "QoS":0, "recordFilter":"atoi(cm_custnumb) >= 1000", "includePrimaryKey":"never", "tagChanges":"tagEachField" } ] }
The
"topic": "topic1"
on"brokerConnectionName":"brokerCtree"
that you set up when you configured your connection is used for published messages.Copy certificates to the <faircom>
\server
folder.FairCom DB Notify acts as a client of the FairCom MQ Broker, so we need to copy the client-side certificates to the Faircom DB server folder.
Copy the
ctsrvr.pem
file (SSL client-side certificate) into your FairCom DB server folder.> copy .\<YourWorkFolder>\ctsrvr.pem .\<faircom>\server\ctsrvr.pem
Copy 'admin' user certificate and key files into the FairCom DB server folder (x509 authentication):
> copy .\<YourWorkFolder>\admin.pem .\<faircom>\server\admin.pem > copy .\<YourWorkFolder>\admin.key .\<faircom>\server\admin.key
Check if the topics are already there:
If you have already run FairCom DB Notify without x509 in this environment,
"topic1"
and"FairComServer/NotificationPlugin/Errors"
topics are already created and will appear in this list.In API Explorer, in the "Select API" drop-down list choose
MQ API
.In the "JSON Actions" drop-down list select
listTopics
then click "Send request" () to execute the
listTopics
action.The API Response will list all topics that are subscribed to in this broker.
In this example,
"topic1"
is not on the server.
Create topics
"topic1"
and"FairComServer/NotificationPlugin/Errors"
if they do not exist in the FairCom MQ broker.In API Explorer, click "Upload Set of Json files" (
)
Select and upload
createErrorTopic.json
andcreateTopic1.json
found in the <faircom>\config\x509.auth.demo\
folder.In "Uploaded Actions" select and execute the API request for each action.
Execute listTopics again to show those topics.
Note: FairCom DB will support the automatic creation of topics with x509 in the future.
Restart the FairCom DB Server.
Proceed to Test DB Notify.
In FairCom DB, create test files to use with DB Notify:
There are two SQL scripts to assist with this tutorial. The custmast1.sql script creates files to monitor, and custmast2.sql modifies the files. Copy both of the script files to your .\tools directory.
copy \config\x509.auth.demo\custmast1.sql .\tools copy \config\x509.auth.demo\custmast2.sql .\tools
Run the following command from the .\tools directory to create "custmast".
isql -s custmast1.sql -u ADMIN -a ADMIN ctreeSQL INIT DEFINE Create table... MANAGE Add records... 1 record inserted. 1 record inserted. 1 record inserted. 1 record inserted. Display records... NUMBER NAME ------ ---- 1000 Bryan Williams 1001 Michael Jordan 1002 Joshua Brown 1003 Keyon Dooling 4 records selected DONE
The resulting files are saved in <faircom>
\data\ctreeSQL.dbs
admin_custmast.dat
admin_custmast.idx
Subscribe to "topic1".
Leave the "MQTT Management" window open to view "Incoming notification Messages".
Initiate some 'dbnotify' activity in FairCom DB.
From <faircom>/tools, execute the following command:
isql -s custmast2.sql -u ADMIN -a ADMIN ctreeSQL
This runs a script that performs an update, a delete, and an insert command for the 'custmast' file that triggers notification messages.
View the messages in MQ Explorer - MQTT Management.
(Run 'custmast2.sql' multiple times to see more incoming notifications.)
SSL problems
If you have trouble solving errors, you can enable a debug log on both the server-side and client-side.
One of the keywords you uncommented in step 2 in the Broker
ctsrvr.cfg
file wasDEBUG_LOG ssl.log
You can review the server-side
ssl.log
(typically located in the directory where faircom.exe is executed from) for server-side details.You can also enable client-side debugging by setting the following environment variable:
set CTSSL_DEBUG_LOG=ssl_client.log
Where
“ssl_client.log”
is an arbitrary file name. Look for this file to be created where you are executing your client application.Could not initialize c-tree Plus(R) (1104)
Check if
'ctsrvr.pem'
was copied into your Apps folder and'ctree_ssl.pem'
to the/server/
folder.This can be a problem during configuration. For example, FairCom MQ requires a secure connection (SSL) and the client is not using SSL.
X509 environment variables were not set while running a command line program.
NOTE: Enabling SSL client-side debugging may help to figure out the problem. Also, check
ssl.log
in the data folder.Could not logon to server 'FCEDGEMQ@localhost^fssltcp'. Error #162.
The server has broken its connection with the application.
Typically this is because you either do not have your user certificate (admin.pem) or key (admin.key) files in place or you have not set your environment variables.
Executing isql with ssl.
SQL doesn't support x509 authentication yet.
error(-30033): No data
Executing isql with ssl..
SQL doesn't support x509 authentication yet.
error(-30034): Network error on server
Executing isql with ssl..
SQL doesn't support x509 authentication yet.
error(-18104): CT - SSL connection attempt failed
You are executing a client with SSL, trying to connect to a server without SSL or vice-versa.
Could not initialize c-tree Plus(R) (128)
Verify if variables CTSSL_CLIENT_CERTIFICATE and CTSSL_CLIENT_KEY were set and exported (linux).
error in tcp bind 10060 error(-20212): Error in Network Daemon Could not logon to server 'FCEDGEMQ@localhost^fssltcp'. Error #133. Server is not operational. The server is not running.
At CTSTATUS.FCS, there are messages like:
CheckTopics error. Not possible to open table [admin_mqtt_msg_topic1].: 4023 CheckTopics error. Not possible to open error table [admin_mqtt_msg_faircomserver_notificationplugin_errors].: 4023
It is necessary to create topics "topic1" and "FairComServer/NotificationPlugin/Errors" at faircom MQ. See Set up DB Notify with SSL and x509.
DB Notify errors
To fix connection errors or to be sure table configurations were set with success, set "logLevel" to "development" at
dbnotifyconnections.json
.This causes DB Notify to show status messages at
CTSTATUS.FCS
. See Configure connnection.{ "logLevel": "development", "directConnectionsToFairComMq": [ { "brokerConnectionName": "brokerCtree", "faircomServerName": "FCEDGEMQ", "brokerHostname": "localhost", "tls": { "certificateFilename": "ctsrvr.pem", "enabled": true, "x509_authentication": { "certificateFilename": "admin.pem", "privateKeyFilename": "admin.key", "enabled": true } } } ] }