FairCom DB Notify
FairCom DB Notify and FairCom MQ non-secure and secure communications - tutorial
Use non-secure or secure communications between FairCom DB Notify and FairCom MQ tutorial
FairCom DB Notify allows you to define event criteria on any FairCom DB database file. The file is actively monitored and a FairCom DB Notify processing event is triggered when the criteria are satisfied. When triggered, FairCom DB Notify sends a configurable MQTT message to a specified MQTT Broker.
This tutorial describes how to set up these event criteria either with non-secure communication or with secure FairCom DB Notify communications between a FairCom DB Server and a FairCom MQ broker using OpenSSL and x509 authentication.
Ensure that FairCom DB and FairCom MQ are installed and running.
Note
To run both products on the same computer, remap the ports used by one or both of the products. Change the SQL_PORT
configuration setting in ctsrvr.cfg
and the enabled ports in services.json
. See: Run multiple server instances on one computer.
Configure FairCom DB Notify with or without TLS secure communications.
Enable the FairCom DB Notify Service for the FairCom DB server by editing the
<faircom>\config\services.json
configuration file and changing the followingenabled
setting fromfalse
totrue
."otherServices": [ { "serviceName": "dbnotify", "serviceLibrary": "./dbnotify/fcdbnotify.dll", "enabled": true },
Edit the
<faircom>\config\dbnotifyconnections.json
file to configure a connection between your FairCom DB server and FairCom MQ. Set thefaircomServerName
to the name used by FairCom MQ. This defaults to"FCEDGEMQ"
. You can verify this by looking forSERVER_NAME
in the<faircom>/config/ctsrvr.cfg
file of your FairCom MQ server.Note
The
brokerHostname
specifies the machine your FairCom MQ server is running on.{ "logLevel": "development", "directConnectionsToFairComMq": [ { "brokerConnectionName": "brokerCtree", "faircomServerName": "FCEDGEMQ", "brokerHostname": "localhost", "brokerUserName": "ADMIN", "brokerUserPassword": "ADMIN", "reconnectFrequencySeconds": 15, "metadata": {}, "defaultIncludePrimaryKey": "never", "defaultTagChanges": "neverTag", } ] }
Edit the
<faircom>\config\dbnotify\ctreeSQLcustmast.json
file to configure the DB notification criteria.When the DB Server enables the FairCom DB Notify service at startup, it looks in the
<faircom>\config\dbnotify
folder for any.json
files. Each.json
file is expected to contain the necessary information related to one table to be monitored. A separate.json
file is needed for each table you want to monitor.Set the
databaseName
to match the database that contains the table to be monitored. For this tutorial, it isctreeSQL
.Set the
tableName
to match the table you want this file to apply to. In this tutorial, it iscustmast
.Set the
ownerName
toadmin
. This should match the user that created the table.Inside the
publishMqttMessages
array add an entry for each type of MQTT message you want the FairCom DBserver to send to the FairCom MQ broker. Each entry can use one or more trigger types and include different fields and settings to create a specific type of MQTT message.Set the
topic
to the MQTT topic you want this message type to notify on. For this tutorial, usetopic1
.Set
brokerConnectionName
to the same name used above when defining the connection to FairCom MQ,brokerCtree
.Set
triggers
to an array of event types you want this message type to notify on.QoS
is the quality of service FairCom MQ will use to publish the notification messages. This example uses QoS 0.recordFilter
is an optional expression used to identify a subset of records to notify on. If left out it will notify on all records for the given triggers.includedFields
is an optional array of field names to include in the notification. If left out all fields will be included.
{ "databaseName": "ctreeSQL", "tableName": "custmast", "ownerName": "admin", "publishMqttMessages": [ { "topic": "topic1", "brokerConnectionName": "brokerCtree", "triggers": [ "delete"], "QoS": 0, "recordFilter": "!stricmp(cm_custstat,\"CA\")", "includedFields": [ "cm_custnumb", "cm_custzipc", "cm_custcity" ], "includePrimaryKey": "never" }, { "topic": "topic1", "brokerConnectionName": "brokerCtree", "triggers": [ "insert", "update"], "QoS": 0, "recordFilter": "atoi(cm_custnumb) >= 1000", "includePrimaryKey": "never", "tagChanges": "tagEachField" } ] }
Restart the DB Server.
Notice
The server is now set up with DB Notify. You can proceed to Test DB notify and perform Troubleshooting if necessary, or you can setup DB Notify with TLS/X509 configuration first.
Complete these steps to secure TCP/IP communications between the DB Notify service and the FairCom DB broker. This involves activating TLS/SSL to encrypt the network communications and activating x509 authorization to secure the user and password used for this connection.
x509 authentication extends SSL support to allow database user authentication based on a client-provided certificate that is trusted by the server, and a user that is registered in the server or is listed in the broker's authorized user database. Typically, users are added using the ctadmn utility.
TLS is an updated, more secure version of SSL. The terms SSL and TLS are used synonymously throughout this document.
Create certificates
Set up FairCom MQ with SSL and x509
Set up FairCom DB Notify with SSL and X509
SSL communications and x509 authorization require the following certificates and keys.
Names | Description |
---|---|
'rootca.pem'/'rootca.key' | The Certificate Authority(CA) certificate and key file pair. These files are used to sign both client and server certificates. The |
'ctree_ssl.pem' | The FairCom MQ broker (server-side) certificate file. This |
'ctsrvr.pem' | The Client-side certificate file. This file is used by the FairCom DB Notify service to secure the TCP/IP communication. This is |
'admin.pem'/admin.key' | The FairCom ADMIN user certificate and private key pair. Each unique database user, including the FairCom ADMIN user, needs their own certificate and private key pair, signed by a trusted Certificate Authority(CA). This pair is intended for the "admin" account. The |
'JonDoe.pem'/JonDoe.key' | A typical (non-admin) user certificate and private key pair. Each unique database user needs their own certificate and private key pair, signed by a trusted Certificate Authority(CA). This is an example of a "JonDoe" account. Later, you will modify our script to create your own user certificates. The |
Sample certificates for use with this demo in addition to a createCerts.bat
script can be found in the <faircom>\config\x509.auth.demo
folder.
The createCerts.bat
script first creates a Certificate Authority(CA) and then uses this CA to sign the other certificates. See the openssl -subj
switch in this script where later you should substitute your company information. Near line 75, shows the 'JonDoe' user. Keep this intact to test this demo, then later go back and adjust as you see fit for your company and users.
Use our provided sample certificates, or use the createCerts.bat
script to create your own. The createCerts.bat
script only needs the rootca.conf
configuration file. If you desire to create your own certificates, copy these two files to any working directory you choose. If you run createCerts.bat
in the x509.auth.demo
folder, you will get error messages for conflicting with our existing certificate files and keys.
The <faircom>/config/createCerts.bat
file also requires the OpenSSL.exe
utility. Be sure the utility is accessible from your command prompt before running createCerts.bat
.
Execute the createCerts.bat
script from your Windows command prompt. Set a PEM pass phrase when prompted. We used FairComQA
, but you may choose your own phrase.
> createCerts.bat
You will see:
> create root CA cert request and private key > Enter PEM pass phrase: FairComQA
The 'pass phrase' for the Certificate Authority(CA) certificate is set, and the CA certificate files are created. You are prompted when this CA certificate is being used to sign the other certificates.
The certificate files (.pem
) and keys (.key
) names are displayed as they are created.
ctree_ssl.pem ctsrvr.pem admin.key admin.pem JonDoe.key JonDoe.pem
Copy certificates
Copy the
ctree_ssl.pem
file (server-side certificate) you generated above into your broker's server folder. Overwrite it if it already exists in the broker's server folder:> copy .\<YourWorkFolder>\ctree_ssl.pem .\<faircom>\server\ctree_ssl.pem
Note: The
ctree_ssl.key
file does not need to be specified because it is already embedded inctree_ssl.pem
.Copy the
ctsrvr.pem
file (client-side certificate) into your client Apps folder. In this tutorial, we use the./tools
folder in the Broker's package.> copy .\<YourWorkFolder>\ctsrvr.pem .\<faircom>\tools\ctsrvr.pem
Copy the user certificate and key files you created above into your client Apps folder. Remember, we made an 'admin' and a 'JonDoe' user.
> copy .\<YourWorkFolder>\admin.pem .\<faircom>\tools\admin.pem > copy .\<YourWorkFolder>\admin.key .\<faircom>\tools\admin.key > copy .\<YourWorkFolder>\JonDoe.pem .\<faircom>\tools\JonDoe.pem > copy .\<YourWorkFolder>\JonDoe.key .\<faircom>\tools\JonDoe.key
Disable shared memory in the broker.
Deactivate the Shared Memory Protocol even if you are connecting with TCP/IP, internally. If the Shared Memory protocol is activated, the Shared Memory is automatically used when possible. To test your TCP/IP TLS/SSL deactivate Shared Memory by commenting out its keyword in the
ctsrvr.cfg
file.; Communication protocols ; COMM_PROTOCOL FSHAREMM
Deny shared memory for SQL also by adding the following line:
SQL_OPTION NO_SHARED_MEMORY
Activate SSL by editing FairCom MQ Broker's
ctsrvr.cfg
file in the<faircom>\config
folder.Uncomment the SSL and x509 support.
;Here is where you can activate (un-comment)SSL SUBSYSTEM COMM_PROTOCOL SSL { ;This is the file name in your server's directory SERVER_CERTIFICATE_FILE ctree_ssl.pem ;For SSL you can specify (un-comment) a debug log file name DEBUG_LOG ssl.log ;Here you can restrict access to SSL ONLY. SSL_CONNECTIONS_ONLY YES ;Require clients to provide a x509 certificate VERIFY_CLIENT_CERTIFICATE YES ;Use x509 client certificate for database authentication x509_AUTHENTICATION YES ;Use the SUBJECT:CN from the client's certificate as their username x509_PATH CN ;SSL_CIPHERS AES256-SHA256:AES256-GCM-SHA38:DHE-RSA-AES256-SHA256:AES256-GCM-SHA384 }
Save your changes to
ctsrvr.cfg
.If using Linux, Unix, or macOS, update
stopserver
in the<faircom>\server
folder to stop the server using SSL.#!/bin/sh echo Stopping the FairCom Database Engine... CTSSL_CLIENT_CERTIFICATE=admin.pem export CTSSL_CLIENT_CERTIFICATE CTSSL_CLIENT_KEY=admin.key export CTSSL_CLIENT_KEY cd ../tools ./ctstop -AUTO none none "FCEDGEMQ@localhost^fssltcp"
Run command-line utilities found in the <faircom>/tools
folder to test that FairCom MQ is SSL-enabled with an x509 ISAM connection.
Restart FairCom MQ.
The log message
"Initialized SSL support."
appears inCTSTATUS.FCS
.Check that the server's configuration file (
ctsrvr.cfg
) allows only SSL connections.SSL_CONNECTIONS_ONLY YES
Launch two different CMD shells and run these two commands in each of the CMD shells to set environment variables in both.
set CTSSL_CLIENT_CERTIFICATE=admin.pem set CTSSL_CLIENT_KEY=admin.key
Verify that x509 connections are working correctly for the ctixmg, ctadmn, and ctixmb tools, and check each tool's connection type.
Verify connections
In the first CMD shell, from the FairCom MQ
tools
folder, run each of the following command lines one at a time. If you get an error such as "Could not initialize c-tree Plus" or "Could not logon to server", the configuration is not configured correctly. If you get a menu or are asked a question, the x509 connection is working.Append
^fssltcp
to the server name to request an SSL-enabled connection for an ISAM connection. In these examples, you may need to changeFCEDGEMQ
to match thectsrvr.cfg
SERVER_NAME
value of your MQ server.Be sure to use quotes on the command line.
It is not necessary to send user/password because the x509 certificate supplies the necessary user and password credentials.
ctixmg ADMIN ADMIN "FCEDGEMQ@localhost^fssltcp" ctadmn ADMIN ADMIN "" "FCEDGEMQ@localhost^fssltcp" ctixmg none none "FCEDGEMQ@localhost^fssltcp"
Check the connection type
While one tool is running in the first CMD shell, run ctadmn in the second CMD shell from the FairCom MQ
tools
folder.ctadmn none none "" "FCEDGEMQ@localhost^fssltcp" **** FairCom(R) Server Administration Utility **** 1. User Operations 2. Group Definitions 3. File Security 4. Monitor Clients Enter your choice (1-10), or 'q' to quit>> 4
Enter 4 to choose
Monitor Clients
.Monitor Clients: 1. List Attached Clients 2. Kill Client Enter your choice (1-2), or 'q' to return to previous menu>> 1
Enter 1 to choose
List Attached Clients
:Look at the
NodeName
of each client that is listed to find the ctixmg, ctadmn, or ctixmb tool you are currently running in the first CMD shell.You might have to press enter a few times because our tool might be last on the list after other internal MQ threads.
The following examples are outputs for the ctixmg and ctadmn tools, with the items of interest bolded.
UserID: ADMIN NodeName: ctixmg Task 20 Communications: FSSLTCP Memory: 235K Open Files: 3 Logon Time: 0:06 Tran Time: -- Rqst Time: 0:06 NoRequest Rqst# 196 OPNRFIL UserID: ADMIN NodeName: ctadmn Task 34 Communications: FSSLTCP Memory: 288K Open Files: 9 Logon Time: 7:38 Tran Time: -- Rqst Time: 7:38 NoRequest Rqst# 196 OPNRFIL
Communications: FSSLTCP
as in these examples indicates the TCP/IP connection is using SSL.Note: The following protocols are some of the more common TCP/IP connections you might observe:
F_TCPIP indicates an unencrypted ISAM TCP/IP connection. FSSLTCP indicates an SSL-enabled ISAM TCP/IP connection. SQL_TCPIP indicates an unencrypted SQL TCP/IP connection. SQL_SSLTCP indicates an SSL-enabled ISAM TCP/IP connection.
Enter q to return to the ctadmn main menu.
Kill the tool in the first CMD shell by pressing ctrl-c, launch the next tool, and check its connection type until done with all three tools.
In the second CMD shell running the ctadmn tool, add the
JonDoe
user into your Server/Broker's database.User Operations: 1. Add New User 2. Remove Existing User 3. List Users Enter your choice (1-9), or 'q' to return to previous menu>> 1
Enter 1,
Add New User
.Enter User Id >> Jon Doe Enter User Description >> Jon Doe
Answer the remaining prompts as you prefer. After the last prompt, press q to return to the main menu. Press q again to exit ctadmn.
Set two environment variables to specify the user
JonDoe
with x509 authentication.set CTSSL_CLIENT_CERTIFICATE=JonDoe.pem set CTSSL_CLIENT_KEY=JonDoe.key
Run the ctixmg tool again, without a user or password, to verify that the
JonDoe
user is correctly configured with an SSL connection. This should launch without a "Could not initialize c-tree Plus" error. You may need to changeFCEDGEMQ
to match yourSERVER_NAME
value.ctixmg none none "FCEDGEMQ@localhost^fssltcp"
With the FairCom DB installation, edit the
<faircom>\config\dbnotifyconnections.json
file to configure a secure connection between your FairCom DB server and the FairCom MQ Broker:Set the
faircomServerName
toFCEDGEMQ
. This is theSERVER_NAME
of the MQ broker that is set in the/config/ctsrvr.cfg
file in the FairCom MQ installation.Enable
tls
andx509_authentication
.At
tls
, set thecertificateFilename
with the name of the client-side certificate file that you previously created (ctsrvr.pem
in this tutorial).At
x509_authentication
, setcertificateFilename
andprivateKeyFilename
with the FairCom ADMIN user certificate key and private key you previously created (admin.pem
andadmin.key
in this tutorial).{ "logLevel":"development", "directConnectionsToFairComMq":[ { "brokerConnectionName":"brokerCtree", "faircomServerName":"FCEDGEMQ", "brokerHostname":"localhost", "reconnectFrequencySeconds":15, "tls":{ "certificateFilename":"ctsrvr.pem", "enabled":true, "x509_authentication":{ "certificateFilename":"admin.pem", "privateKeyFilename":"admin.key", "enabled":true } } } ] }
Encrypt the
dbnotifyconnections.json
file for secure connection and communication.Rename
dbnotifyconnections.json
todbnotifyconnections.cfg
Execute ctcmdset dbnotifyconnections.cfg to get an encrypted
dbnotifyconnections.set
file. (While in theconfig
folder, this command can be run using:..\tools\ctcmdset dbnotifyconnections.cfg
)Rename
dbnotifyconnections.set
todbnotifyconnections.json
Edit the
<faircom>\config\dbnotify\ctreeSQLcustmast.json
file to configure the DB notification criteria.When the DB Server enables the FairCom DB Notify service at startup, it looks in the
config\dbnotify
folder for any.json
files. Each.json
file is expected to contain the necessary information related to one database file to be monitored. A separate.json
file is needed for each database file you want to monitor.{ "databaseName":"ctreeSQL", "tableName":"custmast", "ownerName":"admin", "publishMqttMessages":[ { "persistenceDatabaseName":"faircom", "persistenceOwnerName":"admin", "persistenceTableName":"mqtt_msg_topic1", "brokerConnectionName":"brokerCtree", "triggers":["delete"], "QoS":0, "recordFilter":"!stricmp(cm_custstat,\"CA\")", "includedFields":[ "cm_custnumb", "cm_custzipc", "cm_custcity" ], "includePrimaryKey":"never" }, { "persistenceDatabaseName":"faircom", "persistenceOwnerName":"admin", "persistenceTableName":"mqtt_msg_topic1", "brokerConnectionName":"brokerCtree", "triggers":["insert","update"], "QoS":0, "recordFilter":"atoi(cm_custnumb) >= 1000", "includePrimaryKey":"never", "tagChanges":"tagEachField" } ] }
The
"topic": "topic1"
on"brokerConnectionName":"brokerCtree"
that you set up when you configured your connection is used for published messages.Copy certificates to the
<faircom>\server folder
.FairCom DB Notify acts as a client of the FairCom MQ Broker, so we need to copy the client-side certificates to the FairCom DB server folder.
Copy the
ctsrvr.pem
file (SSL client-side certificate) into your FairCom DB server folder.> copy .\<YourWorkFolder>\ctsrvr.pem .\<faircom>\server\ctsrvr.pem
Copy
admin user certificate
andkey
files into the FairCom DBserver
folder (x509 authentication):> copy .\<YourWorkFolder>\admin.pem .\<faircom>\server\admin.pem > copy .\<YourWorkFolder>\admin.key .\<faircom>\server\admin.key
Switch to the machine that FairCom MQ is running on and use MQ Explorer to check if the topics are already there:
If you have already run FairCom DB Notify without x509 in this environment,
"topic1"
and"FairComServer/NotificationPlugin/Errors"
topics are already created and will appear in this list.In API Explorer, in the drop-down list choose MQ API.
In the listTopics then click Send request (
drop-down list select) to execute the
listTopics
action.The API Response will list all topics that are subscribed to in this broker.
In this example,
"topic1"
is not on the server.
Create topics
"topic1"
and"FairComServer/NotificationPlugin/Errors"
if they do not exist in the FairCom MQ broker.In API Explorer, click Upload Set of Json files (
)
Select and upload
createErrorTopic.json
andcreateTopic1.json
found in the<faircom>\config\x509.auth.demo\
folder.In API request for each action.
select and execute theExecute listTopics again to show those topics.
Note: FairCom DB will support the automatic creation of topics with x509 in the future.
On the machine that FairCom DB is installed on, restart the FairCom DB Server.
Proceed to Test FairCom DB Notify using secure communication and x509 authentication.
Both FairCom DB and FairCom MQ should be running at this point. If you modified any config files in the previous steps, be sure to restart those servers so the changes take effect.
In FairCom DB, create test files to use with DB Notify:
There are two SQL scripts to assist with this tutorial. The
custmast1.sql
script creates files to monitor, andcustmast2.sql
modifies the files. Copy both of the script files to your.\tools
directory.copy config\x509.auth.demo\custmast1.sql .\tools copy config\x509.auth.demo\custmast2.sql .\tools
Run the following command from the
.\tools
directory to createcustmast
.isql -s custmast1.sql -u ADMIN -a ADMIN ctreeSQL INIT DEFINE Create table... MANAGE Add records... 1 record inserted. 1 record inserted. 1 record inserted. 1 record inserted. Display records... NUMBER NAME ------ ---- 1000 Bryan Williams 1001 Michael Jordan 1002 Joshua Brown 1003 Keyon Dooling 4 records selected DONE
The resulting files are saved in
<faircom>\data\ctreeSQL.dbs
admin_custmast.dat
admin_custmast.idx
In MQ Explorer, on the FairCom MQ machine Subscribe to
topic1
.Leave the "MQTT Management" window open to view "Incoming Messages - streaming".
On the FairCom DB machine, initiate some FairCom DB Notify activity in FairCom DB.
From
<faircom>/tools
, execute the following command:isql -s custmast2.sql -u ADMIN -a ADMIN ctreeSQL
This runs a script that performs an update, a delete, and an insert command for the
custmast
file that triggers notification messages.In MQ Explorer, on the FairCom MQ machine, view the messages in MQ Explorer - MQTT Management.
(Run
custmast2.sql
multiple times to see more incoming notifications.)
SSL problems
If you have trouble solving errors, you can enable a debug log on both the server-side and client-side.
One of the keywords you uncommented in step 2 in the Broker
ctsrvr.cfg
file was DEBUG_LOG ssl.logYou can review the server-side
ssl.log
(typically located in the directory where faircom.exe is executed from) for server-side details.You can also enable client-side debugging by setting the following environment variable:
set CTSSL_DEBUG_LOG=ssl_client.log
Where
ssl_client.log
is an arbitrary file name. Look for this file to be created where you are executing your client application.Could not initialize c-tree Plus(R) (1104)
Check if
ctsrvr.pem
was copied into your Apps folder andctree_ssl.pem
to the/server/
folder.This can be a problem during configuration. For example, FairCom MQ requires a secure connection (SSL) and the client is not using SSL.
X509 environment variables were not set while running a command line program.
NOTE: Enabling SSL client-side debugging may help to figure out the problem. Also, check
ssl.log
in the data folder.Could not logon to server 'FCEDGEMQ@localhost^fssltcp'. Error #162.
The server has broken its connection with the application.
Typically this is because you either do not have your user certificate (
admin.pem
) or key (admin.key
) files in place or you have not set your environment variables.Executing isql with ssl.
SQL doesn't support x509 authentication yet.
error(-30033): No data
Executing isql with ssl..
SQL doesn't support x509 authentication yet.
error(-30034): Network error on server
Executing isql with ssl..
SQL doesn't support x509 authentication yet.
error(-18104): CT - SSL connection attempt failed
You are executing a client with SSL, trying to connect to a server without SSL or vice-versa.
Could not initialize c-tree Plus(R) (128)
Verify if variables
CTSSL_CLIENT_CERTIFICATE
andCTSSL_CLIENT_KEY
were set and exported (linux).error in tcp bind 10060 error(-20212): Error in Network Daemon Could not logon to server 'FCEDGEMQ@localhost^fssltcp'. Error #133. Server is not operational. The server is not running.
At
CTSTATUS.FCS
, there are messages like:CheckTopics error. Not possible to open table [admin_mqtt_msg_topic1].: 4023 CheckTopics error. Not possible to open error table [admin_mqtt_msg_faircomserver_notificationplugin_errors].: 4023
It is necessary to create topics
topic1
andFairComServer/NotificationPlugin/Errors
at FairCom MQ. See Set up DB Notify with SSL and x509.DB Notify errors
To fix connection errors or to be sure table configurations were set with success, set
logLevel
todevelopment
atdbnotifyconnections.json
.This causes DB Notify to show status messages at
CTSTATUS.FCS
. See Configure connnection.{ "logLevel": "development", "directConnectionsToFairComMq": [ { "brokerConnectionName": "brokerCtree", "faircomServerName": "FCEDGEMQ", "brokerHostname": "localhost", "tls": { "certificateFilename": "ctsrvr.pem", "enabled": true, "x509_authentication": { "certificateFilename": "admin.pem", "privateKeyFilename": "admin.key", "enabled": true } } } ] }