Upgrade notes for developers
This information is intended as a quick reference to demonstrate how easy it is to move from previous versions of FairCom DB database technology to the latest FairCom Database Engine. Because the FairCom Database Engine is a flexible tool that has enjoyed great longevity, it is impossible to anticipate every situation that will be encountered when upgrading. As always, our outstanding engineering team is ready to assist at every opportunity. Feel free to contact your nearest FairCom office for advice and guidance before migrating your application. We have helped successfully migrate many original FairCom DB applications to our very latest versions, sometimes within minutes.
The next sub-sections provide notes about upgrading from previous releases.
Upgrade notes for developers
FairCom DB includes mtmake.exe
(mtmake
on Linux/Unix) located in \<your installation folder>\build_sdk\build
for building the Standalone libraries. Once you follow the prompts to build your library of choice, execute mk.bat
(mk on Linux/Unix) to build your project. For more details, see mtmake.
Include Files
The include files needed for linking your client applications have been consolidated into the following folders:
\<your installation folder>\build_sdk\include
\<your installation folder>\build_sdk\build\ <your arbitrary directory specified in mtmake.exe>
Libraries
The FairCom DB Standalone library to link with your application will be located in \<your installation folder>\build_sdk\build\ <your arbitrary directory specified in mtmake.exe>\obj\release
(or it will be located in debug if you chose the default value when executing mk.bat
)
Compiling Examples
For examples of how to compile your client application, consult the ctree.mak
file created when you execute mtmake.exe
.
Required Libraries
The following libraries are required when building C/C++ applications:
Linux:
Library | Capability |
---|---|
OpenSSL 1.0.2t | SSL support needs to be installed on most Linux distros. The correct version of OpenSSL is included in our package and our make file points to it. To include the proper encryption system libraries, you will likely need to add the following link switches:
You can review |
Libz | Compression support. Needs to be installed on CentOS. |
libncurses5 | Text based interface is included on most Linux distros. |
gcc, make | Compiling tutorials are included on most Linux distros. |
glibc | Included in most Linux distros. If your glibc is before 2.17 you will also have to include the rt library (-lrt) for |
Windows:
Library | Capability |
---|---|
MS Visual Studio Runtime |
|
There are a few minor compatibility changes with this release. See chapter "12. Compatibility" in the FairCom DB V13 Change Log for details.
Search for "compatibility" in the V13 Change Log, to see where all compatibility changes are discussed.
File Camouflage has been restricted in V13. Calling ctSETENCRYPT()
with options that would enable File Camouflage now fails with error CAMO_NSUP_ERR(1210). Existing application files migrated from an earlier version may continue to use Camouflage, but operations that recreate these files (such as Compact, Alter Table, etc) will create a new copy of the file without Camouflage. If file encryption is desired, it is suggested to enable the ADVANCED_ENCRYPTION YES
configuration option and convert these files to use AES encryption, which can be done using a faircom utility (ctcmpcif
or ctcv67
).
Several files used by faircom may have Camouflage enabled, depending on server configuration when the file was first created: FAIRCOM.FCS
, SYSLOG*.FCS
, SQL database dictionaries (e.g. ctreeSQL.fdd
), and possibly Transaction logs (L*.FCS
), Log Templates (L*.FCT
), or Log Archives (L*.FCA
). Transaction logs using Camouflage are no longer supported and must be removed before updating to V13. The other files will be checked at startup and converted to remove the Camouflage. Password hashes stored in FAIRCOM.FCS
remain in a secure form.
The behavior of several related server configuration keywords has changed:
SYSLOG ENCRYPT
ADMIN_ENCRYPT
LOG_ENCRYPT
These keywords no longer produce Camouflaged files:
LOG_ENCRYPT
and SYSLOG ENCRYPT
are only allowed when ADVANCED_ENCRYPTION YES
is enabled.
ADMIN_ENCRYPT
is ignored unless ADVANCED_ENCRYPTION YES
is enabled.
SQL database dictionaries are only encrypted if ADVANCED_ENCRYPTION YES
is enabled.
With V12, FairCom has strengthened it's license file checks. Therefore, before rolling out a V12 production license (ctsrvr*.lic
file), we recommend thorough testing on your production machine to ensure your license files are properly sized for your production hardware. Details on CPU counting and licensing can be found here.
FAIRCOM.FCS V9 and older must be recreated:
FairCom DB V12 and FairCom RTG V3 have deprecated an older algorithm that prevents usage of FAIRCOM.FCS
files from FairCom DB V9 and older Server lines (and any customers using the prev10logon switch within their V10 and newer ctsrvr*.lic
files). The solution is to recreate the FAIRCOM.FCS
file by not moving this file forward.
Note
Existing user IDs and passwords will need to be recreated with V12 and V3
Adjusting page size is a complex topic. To learn more details about it, see PAGE_SIZE Conversions.
Warning
Warning: Changing the PAGE_SIZE is a maintenance task that should be done carefully and with a full reliable backup. Practice on a copy of your data and other folders until you are confident with the results. FairCom DB V12 has a default PAGE_SIZE 32K (V11 defaulted to 8K), and it will automatically attempt to convert the FairCom Internal files (FAIRCOM.FCS
, *.FSD
, *.FDD
) during the initial startup. The automatic conversion requires the file ctscmp.exe
(ctscmp
on Linux/Unix) to be present in the working directory where the FairCom Server binary (faircom.exe
for V12) is located (the \server
directory for V12).
Notice that a file created with a larger PAGE_SIZE
cannot be opened by a system with a smaller PAGE_SIZE
.
Follow these procedures to rebuild the affected files:
Shut down the FairCom Database Engine, start it again, then shut it down again to be sure all logs are flushed and a clean final checkpoint has been written.
Make a good backup of your databases and your server folder.
Move this copy to a different machine from your live system. Keep a clean copy in case you need to start over.
Edit
ctsrvr.cfg
and add thePAGE_SIZE
keyword if it is not already there:PAGE_SIZE 32768
The size should be a power of 2. The maximum page size is 64k (
65536
).The current default in V12 and later is
32768
.
Go to the FairCom
data
directory.Delete all log and start files (i.e., all files that start with
"L00…"
,"S00…"
,"D00…"
). With a clean shutdown you should not have any"I00…"
files in the directory.For V12.0.2 and greater, FairCom DB attempts to convert its internal control files to the larger
PAGE_SIZE
specified inctsrvr.cfg
, if needed. See FairCom Internal File Conversion below for important details.For V12.0.1 and earlier, rebuild the three superfiles used by the FairCom Database Engine using the
ctscmp
utility including the-sect
parameter:Remember that the sect size is the page size/128. For example, if your page size is
32768
, use a sect size of256
.ctscmp.exe <server directory>\data\ctdbdict.fsd 256
ctscmp.exe <server directory>\data\ctreeSQL.dbs\SQL_SYS\ctreeSQL.fdd 256
ctscmp.exe <server directory>\data\FAIRCOM.FCS 256
Delete the current index files for your data files.
Change the indexes for your data files using the
ctcmpcif
orctcmpcif.standalone
utility including the-sect
parameter (which is-256
in the example below):./ctcmpcif (server directory)/data/ctreeSQL.dbs/MyData.dat ADMIN ADMIN FAIRCOMS -256
./ctcmpcif.standalone (server directory)/data/ctreeSQL.dbs/MyData.dat -256
If using
ctcmpcif
, restart the server first.Fixed-length files:
Rebuild the indexes for your fixed-length data files using the
ctrbldif
utility, including the-sect
parameter (which is-256
in the example below):./ctrbldif (server directory)/data/ctreeSQL.dbs/MyData.dat -256
Variable-length files:
Rebuild the indexes using
ctcmpcif.exe
as follows:ctcmpcif.exe <your application data directory>\your_file.dat 256
Note
It is acceptable to use
ctcmpcif.exe
to rebuild both fixed and variable-length files. Thectcmpcif.exe
utility creates a new copy of the data file by copying all the active records into a new file, and then callingctrbldif.exe
to generate a new index. This step is required to ensure the deleted space management index is properly built within the variable length data file. Fixed-length files don’t have any internal indexes; therefore, callingctrlbdif.exe
is sufficient.Restart the server and verify no errors were logged in
<faircom>/data/CTSTATUS.FCS
.Connect to your data files and verify no errors were returned.
FairCom Internal File Conversion
When the PAGE_SIZE
server configuration option is changed, FairCom V12.0.2 and later attempts to apply this change to all FairCom server-controlled files. At server startup, we check the page size of the following FairCom files:
ctdbdict.fsd
Database dictionaries
FAIRCOM.FCS
SEQUENCEDT.FCS
DFRKSTATEDT.FCS
SYSLOGDT.FCS
REPLFFCHGDT.FCS
REPLOGDT.FCS
REPLOGSHIPDT.FCS
REPLSTATEDT.FCS
REPLSYNCDT1.FCS
REPLSYNCDT2.FCS
If any do not match the newly specified PAGE_SIZE
, a backup copy of the existing file is made with an .FCB
extension. The server process then attempts to invoke the FairCom ctscmp.exe
(ctscmp
on Linux/Unix) superfile rebuild utility with the new configured PAGE_SIZE
.
Superfiles require that the PAGE_SIZE
at open matches the PAGE_SIZE
at file creation time, or a KSIZ_ERR (40) or SPAG_ERR (417) error occurs at file open. For FAIRCOM.FCS
, this prevents the server from starting when operating with a different configured PAGE_SIZE
setting.
A new configuration keyword has been added to control the automatic page size conversion at startup:
PAGE_SIZE_CONVERSION {YES|NO}Default: YES
Limitations:
We expect
ctscmp.exe
(ctscmp
on Linux/Unix) to exist in the process working directory and have permission to be executable by the server process. If this utility does not exist, the server fails to start with the following messages likely logged toCTSTATUS.FCS
:- User# 00001 Wrong PAGESIZE for FAIRCOM.FCS, attempting to convert file from PAGESIZE 8192 to 32768 - User# 00001 Did not find ctscmp in working directory for Superfile conversion: 2 - User# 00001 Could not process User/Group Information in FAIRCOM.FCS: 417 - User# 00001 Could not initialize server. Error: 417 - User# 00001 O1 M0 L74 F417 P0x (recur #1) (uerr_cod=417
The
LOCAL_DIRECTORY
keyword must be set and must be different from the working directory. Otherwise,ctscmp.exe
will encounter a TCOL_ERR (537) and fail.Superfile conversion occurs only after auto-recovery. Recovery is likely to fail if run with a different
PAGE_SIZE
setting. We expect any needed recovery should occur before making a major configuration impacting the physical attributes of critical operational files such as authentication information and SQL database dictionaries (system tables).NO FURTHER ATTEMPT IS MADE to convert any other existing indexes. All other application created indexes will require manual rebuilding to convert to a new page size. See the steps below for the best practice procedures.
The following messages may be found logged to
CTSTATUS.FCS
after a successful conversion:- User# 00001 Wrong PAGESIZE for FAIRCOM.FCS, attempting to convert file from PAGESIZE 8192 to 32768 - User# 00001 Backup created ../data/\FAIRCOM.FCS => ../data/\FAIRCOM.FCS.1621883226.FCB - User# 00001 Superfile conversion successful - User# 00001 Wrong PAGESIZE for ctdbdict.fsd, attempting to convert file from PAGESIZE 8192 to 32768 - User# 00001 Backup created ../data/\ctdbdict.fsd => ../data/\ctdbdict.fsd.1621883232.FCB - User# 00001 Superfile conversion successful - User# 00001 Wrong PAGESIZE for .\ctreeSQL.dbs\SQL_SYS\ctreeSQL.fdd, attempting to convert file from PAGESIZE 8192 to 32768 - User# 00001 Backup created ../data/\.\ctreeSQL.dbs\SQL_SYS\ctreeSQL.fdd => ../data/\.\ctreeSQL.dbs\SQL_SYS\ctreeSQL.fdd.1621883237.FCB - User# 00001 Superfile conversion successful
Once correct server operations are confirmed after conversion, the *.FCB
files can be removed and deleted.
Additional server directory housekeeping files:
DFRKSTATEDT.FCS
DFRKSTATEIX.FCS
Deferred Key pending add files
Updated Configuration Defaults
New V11 Config file defaults:
; Maximum status log size of 32 MB, keeping one prior copy
CTSTATUS.FCS
defaults to -32000000
These options are new to V11, and these are their defaults:
; Flush updates for transaction files to file system as soon as possible in the background
TRAN_DATA_FLUSH_SEC 60
TRAN_INDEX_FLUSH_SEC 60
; Flush updates for non-tran files to file system as soon as possible in the background
NONTRAN_DATA_FLUSH_SEC IMMEDIATE
NONTRAN_INDEX_FLUSH_SEC IMMEDIATE
The following options have been added as convenience options. Be sure to review them as they may change behavior in your particular installations in subtle ways:
; Limit JVM memory
SETENV DH_JVM_OPTION_STRINGS=-Xms100m -Xmx300m
; Suppress dynamic dump logging of backed up file names to CTSTATUS.FCS
CTSTATUS_MASK DYNAMIC_DUM_FILES
; Log final SNAPSHOT stats to SNAPSHOT.FCS on shutdown for baseline metrics
DIAGNOSTICS SNAPSHOT_SHUTDOWN
Other compatibility changes:
Check the V11 FairCom DB Update Guide for the latest changes with the V11 server.
Critical updates:
Check the V11 FairCom DB Update Guide for all pertinent critical update information.
A few notes about the V10 changes:
FairCom DB V10.0 bring about major security updates and as such, all prior client/server connectivity is now incompatible. Note that this impacts previous c-tree Plus ODBC drivers. See V10 Client/Server Compatibility.
The
SERVER_DIRECTORY
keyword is now deprecated in favor ofLOCAL_DIRECTORY
. See SERVER_DIRECTORY deprecated.Check the Compatibility section of the V10 Update Guide for other minor release changes. See V10 Compatibility Notes.
V9 introduced several minor compatibility issues of its own that you should be aware of when moving forward:
Extended headers are now included by default on all new files created by FairCom DB.
Commit Read Lock support is default and requires record locks on any update.
Files created by FairCom DB SQL are
HUGE
by default.
V8 introduced memory files, queues, and notification and performance monitoring tools.
Cleanly shut down any existing V8 Servers and remove remaining transaction logs (see Steps for a Clean Upgrade). V9 introduces a new transaction log format.
Store a copy of the V8 dynamic dump restore utility (
ctrdmp
) with any remaining backup files in case of future recovery. Consider validating a fresh V9 dump and restore and review your backup procedures at this time.
V7 introduced HUGE
, segmented and partitioned file support, and many new cache options for performance. V7.12 increased the PAGE_SIZE
default from 2048
bytes to 8192
bytes. Keep this in mind with indexes from previous versions, as the page size is used as the index node size.
Cleanly shut down any existing V7 Servers and remove remaining transaction logs (see Steps for a Clean Upgrade). V9 introduces a new transaction log format.
Store a copy of the V7 dynamic dump restore utility (
ctrdmp
) with any remaining backup files in case of future recovery. Consider validating a fresh V9 dump and restoring and reviewing your backup procedures at this time.
V6 introduced updated c-tree Plus file format with extended headers for advanced feature support.
Convert your existing V6 data files with the
ctcv67
conversion utility. This utility converts existing c-tree Plus data files to the extended format introduced in V7. Extended header support offersHUGE
file support and segmented files.Use the
Xtd8
file creation functions to create new Extended files.For data files supporting
HUGE
file support, remember to update your duplicate index keys in theIFIL
definitions to include an additional 4 bytes for the associated record position.Note that the
ctreep.h
header file automatically includes thectv6v7.h
header.Cleanly shut down any existing V6 Servers and remove remaining transaction logs (Steps for a Clean Upgrade). V9 introduces a new transaction log format.
Store a copy of the V6 dynamic dump restore utility (
ctrdmp
) with any remaining backup files in case of future recovery. Consider validating a fresh V9 dump and restore and review your backup procedures at this time.
As always, the FairCom DB is designed for immediate use right out of the box.
Applications written for c-tree Plus V6 that include ctreep.h
as required will compile and link properly right out of the box. Your application will function exactly as before using the latest FairCom DB library. ctreep.h
automatically includes the ctv6v7.h
header (described in #defines).
Developers wanting to use HUGE
, segmented files or other functionality made available with the Xtd8
file format (see File Formats in the FairCom Programmer's Reference Guide) first introduced in V7 must also:
Use the 8-byte file creation functions to create new extended files.
Change
IFIL
definitions for indexes, allowing duplicates to store 8-byte offsets for indexes referencingHUGE
files.Convert existing data files using the
ctcv67
utility.
Note that the ctoptn.h
file is now created in a custom.xxx
directory where .xxx
represents the operational model. This file was previously created in the custom directory.
For indexes associated with Standard data files, a duplicate key length includes 4 bytes for the associated record position which is used to break ties. If an index is created for a HUGE
data file, then the key length must include 8 bytes for the associated record position.
The FairCom DB #define
constants were modified to use a uniform style of ctWXYZ
by appending ct
to the beginning of all FairCom DB defines. For example, TRNLOG
is changed to ctTRNLOG
.
Two header files, ctv6v7.h
and ctv7v6.h
, provide compatibility.
Add the ctv6v7.h
compatibility header file to applications written for c-tree Plus V6.x
to automatically change the old defines to the new defines to allow the application to be compiled with the latest version of FairCom DB.
Add the ctv7v6.h
compatibility header file to applications written for the latest version of FairCom DB to automatically change the new defines to the old defines to allow the application to be compiled with c-tree Plus V6.x
. This header provides “rename #defines”
to set the symbols back to their V6 convention.
The following #defines
were changed:
ADMOPEN
AG_LOCK
AUTOSAVE
AUTOTRN
CHECKLOCK
CHECKREAD
CIPFASE
DEFERCP
DELUPDT
DISABLERES
DUPCHANEL
ENABLE
ENABLE_BLK
EXCLUSIVE
FREE
GETLKISAM
LK_BLOCK
LKSTATE
LOGFIL
LOGIDX
MIRROR_SKP
NONEXCLUS IVE
OPENCRPT
OVRFASE
PENDERR
PENDREN
PENDRNU
PERMANENT
PREIMG
READFIL
READREC
READREC_BLK
RESET
RESTORE
RESTORE_BLK
RESTRED
RESTRED_BLK
RESTRSV
SAVECTREE
SAVENV
SHARED
SS_LOCK
SUPERFILE
SUSPEND
TRANDEP_CRE
TRANDEP_DEL
TRANDEP_SFM
TRNBEGLK
TRNLOG
TWOFASE
VIRTUAL
VLENGTH
WRITETHRU
(V4.1F-V4.3C)
Existing c-tree V4 files require conversion to the updated FairCom DB format. The following steps outline this process.
Make a BACKUP of ALL OF YOUR FILES before you begin!
Convert your existing data files with the
ctcv43
file conversion utility and set these converted files aside.Use the
ctexmc
parameter file create utility to create empty data and index files from your existing parameter files. (We do this as theIFIL
conversion will require a valid index file to be in place, even if empty).Copy in your newly converted data files, overwriting the empty data files just created by
ctexmc
. (Leave the new empty index files in place).Convert your existing parameter files to
IFIL
structures withctptoi
parameter conversion utility. This utility will place anIFIL
structure directly into the converted data files.Rebuild your index files with the
ctrbldif
rebuild utility.(Optional) Stamp a
DODA
structure (data schema) based on the'C'
structure definitions into your data file with the FairCom DBPutDODA()
API call. TheDODA
makes available many higher level interfaces such as ODBC, FairCom DB API, .NET and FairCom DB SQL.Include the single FairCom DB
ctreep.h
header file in your application to replace the previous numerous c-tree headers:#include "ctreep.h"