Upgrade notes for developers

This information is intended as a quick reference to demonstrate how easy it is to move from previous versions of FairCom DB database technology to the latest FairCom Database Engine. Because the FairCom Database Engine is a flexible tool that has enjoyed great longevity, it is impossible to anticipate every situation that will be encountered when upgrading. As always, our outstanding engineering team is ready to assist at every opportunity. Feel free to contact your nearest FairCom office for advice and guidance before migrating your application. We have helped successfully migrate many original FairCom DB applications to our very latest versions, sometimes within minutes.

The next sub-sections provide notes about upgrading from previous releases.

Upgrade notes for developers

upgrade notesnotes for developershow to upgradecompile your application

Compiling Your Application

FairCom DB includes mtmake.exe (mtmake on Linux/Unix) located in \<your installation folder>\build_sdk\build for building the Standalone libraries. Once you follow the prompts to build your library of choice, execute mk.bat (mk on Linux/Unix) to build your project. For more details, see mtmake.

Include Files

The include files needed for linking your client applications have been consolidated into the following folders:

\<your installation folder>\build_sdk\include
\<your installation folder>\build_sdk\build\ <your arbitrary directory specified in mtmake.exe>

Libraries

The FairCom DB Standalone library to link with your application will be located in \<your installation folder>\build_sdk\build\ <your arbitrary directory specified in mtmake.exe>\obj\release (or it will be located in debug if you chose the default value when executing mk.bat)

Compiling Examples

For examples of how to compile your client application, consult the ctree.mak file created when you execute mtmake.exe.

Required Libraries

The following libraries are required when building C/C++ applications:

Linux:

Library	Capability
OpenSSL 1.0.2t	SSL support needs to be installed on most Linux distros. The correct version of OpenSSL is included in our package and our make file points to it. To include the proper encryption system libraries, you will likely need to add the following link switches: `-lssl` `-lcrypto` You can review `/drivers/c.isam/tutorials/cmdline/Makefile` for a working example of building C and C++ applications.
Libz	Compression support. Needs to be installed on CentOS.
libncurses5	Text based interface is included on most Linux distros.
gcc, make	Compiling tutorials are included on most Linux distros.
glibc	Included in most Linux distros. If your glibc is before 2.17 you will also have to include the rt library (-lrt) for `"clock_gettime"`.

Windows:

Library	Capability
MS Visual Studio Runtime	user32.lib gdi32.lib winspool.lib comdlg32.lib advapi32.lib shell32.lib kernel32.lib oldnames.lib crypt32.lib

V13 Changes

There are a few minor compatibility changes with this release. See chapter "12. Compatibility" in the FairCom DB V13 Change Log for details.

Search for "compatibility" in the V13 Change Log, to see where all compatibility changes are discussed.

Camouflage compatibility

File Camouflage has been restricted in V13.1.0. Calling ctSETENCRYPT() with options that would enable File Camouflage now fails with error CAMO_NSUP_ERR(1210). Existing application files migrated from an earlier version may continue to use Camouflage by using keyword ENABLE_CAMO YES. If file encryption is desired, it is suggested to enable the ADVANCED_ENCRYPTION YES configuration option and convert these files to use AES encryption, which can be done using a faircom utility (ctcmpcif or ctcv67).

Several files used by faircom may have Camouflage enabled, depending on server configuration when the file was first created: FAIRCOM.FCS, SYSLOG*.FCS, SQL database dictionaries (e.g. ctreeSQL.fdd), and possibly Transaction logs (L*.FCS), Log Templates (L*.FCT), or Log Archives (L*.FCA). Transaction logs using Camouflage are no longer supported and must be removed before updating to V14. The other files will be checked at startup and converted to remove the Camouflage. FAIRCOM.FCS and SQL dictionaries require the ctscmp utility to be present in the process working directory for conversion. Password hashes stored in FAIRCOM.FCS remain in a secure form.

The behavior of several related server configuration keywords has changed:

SYSLOG ENCRYPT
ADMIN_ENCRYPT
LOG_ENCRYPT

These keywords no longer produce Camouflaged files:

LOG_ENCRYPT and SYSLOG ENCRYPT are only allowed when ADVANCED_ENCRYPTION YES is enabled.

ADMIN_ENCRYPT is ignored unless ADVANCED_ENCRYPTION YES is enabled.

SQL database dictionaries are only encrypted if ADVANCED_ENCRYPTION YES is enabled.

If AES encryption is desired for these files, they must be either recreated or converted using the ctscmp or ctcmpcif utilities, which have encryption-specific options. Transaction logs, Log templates, or Log Archives don't have an option to change the encryption of existing logs. These files must be removed.

The format of the tamper-resistant configuration file (ctsrvr.set) has changed. Existing ctsrvr.set files must be recreated using a current version of ctcfgset. Additionally, the following keywords are not allowed in ctsrvr.set: LOCAL_DIRECTORY, FIPS_ENCRYPTION, and MULTILINE_STATUS_LOG_MESSAGE.

To remove camouflage from a superfile, use the superfile compact utility ctscmp with the -cleartext option: ctscmp <myfile> -cleartext

To remove camouflage from a regular file, use ctcmpcif <myfile> -encrypt=none

If using the server for conversions, set ENABLE_CAMO and CHANGE_ENCRYPTION_ON_COMPACT to YES. These can be removed following conversion.

V12 Changes

With V12, FairCom has strengthened it's license file checks. Therefore, before rolling out a V12 production license (ctsrvr*.lic file), we recommend thorough testing on your production machine to ensure your license files are properly sized for your production hardware. Details on CPU counting and licensing can be found here.

FAIRCOM.FCS V9 and older must be recreated:

FairCom DB V12 and FairCom RTG V3 have deprecated an older algorithm that prevents usage of FAIRCOM.FCS files from FairCom DB V9 and older Server lines (and any customers using the prev10logon switch within their V10 and newer ctsrvr*.lic files). The solution is to recreate the FAIRCOM.FCS file by not moving this file forward.

Note

Existing user IDs and passwords will need to be recreated with V12 and V3

See Adjusting PAGE_SIZE

Adjusting PAGE_SIZE

Adjusting page size is a complex topic. To learn more details about it, see PAGE_SIZE Conversions.

Warning

Warning: Changing the PAGE_SIZE is a maintenance task that should be done carefully and with a full reliable backup. Practice on a copy of your data and other folders until you are confident with the results. FairCom DB V12 has a default PAGE_SIZE 32K (V11 defaulted to 8K), and it will automatically attempt to convert the FairCom Internal files (FAIRCOM.FCS, *.FSD, *.FDD) during the initial startup. The automatic conversion requires the file ctscmp.exe (ctscmp on Linux/Unix) to be present in the working directory where the FairCom Server binary (faircom.exe for V12) is located (the \server directory for V12).

Notice that a file created with a larger PAGE_SIZE cannot be opened by a system with a smaller PAGE_SIZE.

Follow these procedures to rebuild the affected files:

Shut down the FairCom Database Engine, start it again, then shut it down again to be sure all logs are flushed and a clean final checkpoint has been written.
Make a good backup of your databases and your server folder.
Move this copy to a different machine from your live system. Keep a clean copy in case you need to start over.
Edit ctsrvr.cfg and add the PAGE_SIZE keyword if it is not already there:
PAGE_SIZE 32768
- The size should be a power of 2. The maximum page size is 64k (65536).
- The current default in V12 and later is 32768.
Go to the FairCom data directory.
Delete all log and start files (i.e., all files that start with "L00…", "S00…", "D00…"). With a clean shutdown you should not have any "I00…" files in the directory.
For V12.0.2 and greater, FairCom DB attempts to convert its internal control files to the larger PAGE_SIZE specified in ctsrvr.cfg, if needed. See FairCom Internal File Conversion below for important details.
For V12.0.1 and earlier, rebuild the three superfiles used by the FairCom Database Engine using the ctscmp utility including the -sect parameter:
- Remember that the sect size is the page size/128. For example, if your page size is 32768, use a sect size of 256.
```
ctscmp.exe <server directory>\data\ctdbdict.fsd 256
```
```
ctscmp.exe <server directory>\data\ctreeSQL.dbs\SQL_SYS\ctreeSQL.fdd 256  
```
```
ctscmp.exe <server directory>\data\FAIRCOM.FCS 256
```
Delete the current index files for your data files.
Change the indexes for your data files using the ctcmpcif or ctcmpcif.standalone utility including the -sect parameter (which is -256 in the example below):
```
./ctcmpcif (server directory)/data/ctreeSQL.dbs/MyData.dat ADMIN ADMIN FAIRCOMS -256
```
```
./ctcmpcif.standalone (server directory)/data/ctreeSQL.dbs/MyData.dat -256
```
If using ctcmpcif, restart the server first.
Fixed-length files:
Rebuild the indexes for your fixed-length data files using the ctrbldif utility, including the -sect parameter (which is -256 in the example below):
```
./ctrbldif (server directory)/data/ctreeSQL.dbs/MyData.dat -256
```
Variable-length files:
Rebuild the indexes using ctcmpcif.exe as follows:
```
ctcmpcif.exe <your application data directory>\your_file.dat 256
```
Note
It is acceptable to use ctcmpcif.exe to rebuild both fixed and variable-length files. The ctcmpcif.exe utility creates a new copy of the data file by copying all the active records into a new file, and then calling ctrbldif.exe to generate a new index. This step is required to ensure the deleted space management index is properly built within the variable length data file. Fixed-length files don’t have any internal indexes; therefore, calling ctrlbdif.exe is sufficient.
Restart the server and verify no errors were logged in <faircom>/data/CTSTATUS.FCS.
Connect to your data files and verify no errors were returned.

FairCom Internal File Conversion

When the PAGE_SIZE server configuration option is changed, FairCom V12.0.2 and later attempts to apply this change to all FairCom server-controlled files. At server startup, we check the page size of the following FairCom files:

ctdbdict.fsd
Database dictionaries
FAIRCOM.FCS
SEQUENCEDT.FCS
DFRKSTATEDT.FCS
SYSLOGDT.FCS
REPLFFCHGDT.FCS
REPLOGDT.FCS
REPLOGSHIPDT.FCS
REPLSTATEDT.FCS
REPLSYNCDT1.FCS
REPLSYNCDT2.FCS

If any do not match the newly specified PAGE_SIZE, a backup copy of the existing file is made with an .FCB extension. The server process then attempts to invoke the FairCom ctscmp.exe (ctscmp on Linux/Unix) superfile rebuild utility with the new configured PAGE_SIZE.

Superfiles require that the PAGE_SIZE at open matches the PAGE_SIZE at file creation time, or a KSIZ_ERR (40) or SPAG_ERR (417) error occurs at file open. For FAIRCOM.FCS, this prevents the server from starting when operating with a different configured PAGE_SIZE setting.

A new configuration keyword has been added to control the automatic page size conversion at startup:

PAGE_SIZE_CONVERSION {YES|NO}Default: YES

Limitations:

We expect ctscmp.exe (ctscmp on Linux/Unix) to exist in the process working directory and have permission to be executable by the server process. If this utility does not exist, the server fails to start with the following messages likely logged to CTSTATUS.FCS:

- User# 00001 Wrong PAGESIZE for FAIRCOM.FCS, attempting to convert file from PAGESIZE 8192 to 32768

- User# 00001 Did not find ctscmp in working directory for Superfile conversion: 2

- User# 00001 Could not process User/Group Information in FAIRCOM.FCS: 417

- User# 00001 Could not initialize server. Error: 417

- User# 00001 O1 M0 L74 F417 P0x (recur #1) (uerr_cod=417

The LOCAL_DIRECTORY keyword must be set and must be different from the working directory. Otherwise, ctscmp.exe will encounter a TCOL_ERR (537) and fail.
Superfile conversion occurs only after auto-recovery. Recovery is likely to fail if run with a different PAGE_SIZE setting. We expect any needed recovery should occur before making a major configuration impacting the physical attributes of critical operational files such as authentication information and SQL database dictionaries (system tables).

NO FURTHER ATTEMPT IS MADE to convert any other existing indexes. All other application created indexes will require manual rebuilding to convert to a new page size. See the steps below for the best practice procedures.

The following messages may be found logged to CTSTATUS.FCS after a successful conversion:

- User# 00001 Wrong PAGESIZE for FAIRCOM.FCS, attempting to convert file from PAGESIZE 8192 to 32768

- User# 00001 Backup created ../data/\FAIRCOM.FCS => ../data/\FAIRCOM.FCS.1621883226.FCB - User# 00001 Superfile conversion successful

- User# 00001 Wrong PAGESIZE for ctdbdict.fsd, attempting to convert file from PAGESIZE 8192 to 32768

- User# 00001 Backup created ../data/\ctdbdict.fsd => ../data/\ctdbdict.fsd.1621883232.FCB

- User# 00001 Superfile conversion successful

- User# 00001 Wrong PAGESIZE for .\ctreeSQL.dbs\SQL_SYS\ctreeSQL.fdd, attempting to convert file from PAGESIZE 8192 to 32768

- User# 00001 Backup created ../data/\.\ctreeSQL.dbs\SQL_SYS\ctreeSQL.fdd => ../data/\.\ctreeSQL.dbs\SQL_SYS\ctreeSQL.fdd.1621883237.FCB

- User# 00001 Superfile conversion successful

Once correct server operations are confirmed after conversion, the *.FCB files can be removed and deleted.

V11 Changes

Additional server directory housekeeping files:

DFRKSTATEDT.FCS
DFRKSTATEIX.FCS

Deferred Key pending add files

Updated Configuration Defaults

New V11 Config file defaults:

; Maximum status log size of 32 MB, keeping one prior copy

CTSTATUS.FCS defaults to -32000000

These options are new to V11, and these are their defaults:

; Flush updates for transaction files to file system as soon as possible in the background

TRAN_DATA_FLUSH_SEC     60

TRAN_INDEX_FLUSH_SEC    60

; Flush updates for non-tran files to file system as soon as possible in the background

NONTRAN_DATA_FLUSH_SEC  IMMEDIATE

NONTRAN_INDEX_FLUSH_SEC IMMEDIATE

The following options have been added as convenience options. Be sure to review them as they may change behavior in your particular installations in subtle ways:

; Limit JVM memory

SETENV                  DH_JVM_OPTION_STRINGS=-Xms100m -Xmx300m

; Suppress dynamic dump logging of backed up file names to CTSTATUS.FCS

CTSTATUS_MASK           DYNAMIC_DUM_FILES

; Log final SNAPSHOT stats to SNAPSHOT.FCS on shutdown for baseline metrics

DIAGNOSTICS             SNAPSHOT_SHUTDOWN

Other compatibility changes:

Check the V11 FairCom DB Update Guide for the latest changes with the V11 server.

Critical updates:

Check the V11 FairCom DB Update Guide for all pertinent critical update information.

V10 Changes

A few notes about the V10 changes:

FairCom DB V10.0 bring about major security updates and as such, all prior client/server connectivity is now incompatible. Note that this impacts previous c-tree Plus ODBC drivers. See V10 Client/Server Compatibility.
The SERVER_DIRECTORY keyword is now deprecated in favor of LOCAL_DIRECTORY. See SERVER_DIRECTORY deprecated.
Check the Compatibility section of the V10 Update Guide for other minor release changes. See V10 Compatibility Notes.

V9 Changes

V9 introduced several minor compatibility issues of its own that you should be aware of when moving forward:

Extended headers are now included by default on all new files created by FairCom DB.
Commit Read Lock support is default and requires record locks on any update.
Files created by FairCom DB SQL are HUGE by default.

V8 Changes

V8 introduced memory files, queues, and notification and performance monitoring tools.

Cleanly shut down any existing V8 Servers and remove remaining transaction logs (see Steps for a Clean Upgrade). V9 introduces a new transaction log format.
Store a copy of the V8 dynamic dump restore utility (ctrdmp) with any remaining backup files in case of future recovery. Consider validating a fresh V9 dump and restore and review your backup procedures at this time.

V7 Changes

V7 introduced HUGE, segmented and partitioned file support, and many new cache options for performance. V7.12 increased the PAGE_SIZE default from 2048 bytes to 8192 bytes. Keep this in mind with indexes from previous versions, as the page size is used as the index node size.

Cleanly shut down any existing V7 Servers and remove remaining transaction logs (see Steps for a Clean Upgrade). V9 introduces a new transaction log format.
Store a copy of the V7 dynamic dump restore utility (ctrdmp) with any remaining backup files in case of future recovery. Consider validating a fresh V9 dump and restoring and reviewing your backup procedures at this time.

V6 Changes

V6 introduced updated c-tree Plus file format with extended headers for advanced feature support.

Convert your existing V6 data files with the ctcv67 conversion utility. This utility converts existing c-tree Plus data files to the extended format introduced in V7. Extended header support offers HUGE file support and segmented files.
Use the Xtd8 file creation functions to create new Extended files.
For data files supporting HUGE file support, remember to update your duplicate index keys in the IFIL definitions to include an additional 4 bytes for the associated record position.
Note that the ctreep.h header file automatically includes the ctv6v7.h header.
Cleanly shut down any existing V6 Servers and remove remaining transaction logs (Steps for a Clean Upgrade). V9 introduces a new transaction log format.
Store a copy of the V6 dynamic dump restore utility (ctrdmp) with any remaining backup files in case of future recovery. Consider validating a fresh V9 dump and restore and review your backup procedures at this time.

Upgrading V6 applications

As always, the FairCom DB is designed for immediate use right out of the box.

Applications written for c-tree Plus V6 that include ctreep.h as required will compile and link properly right out of the box. Your application will function exactly as before using the latest FairCom DB library. ctreep.h automatically includes the ctv6v7.h header (described in #defines).

Developers wanting to use HUGE, segmented files or other functionality made available with the Xtd8 file format (see File Formats in the FairCom Programmer's Reference Guide) first introduced in V7 must also:

Use the 8-byte file creation functions to create new extended files.
Change IFIL definitions for indexes, allowing duplicates to store 8-byte offsets for indexes referencing HUGE files.
Convert existing data files using the ctcv67 utility.

Note that the ctoptn.h file is now created in a custom.xxx directory where .xxx represents the operational model. This file was previously created in the custom directory.

Duplicate keys

For indexes associated with Standard data files, a duplicate key length includes 4 bytes for the associated record position which is used to break ties. If an index is created for a HUGE data file, then the key length must include 8 bytes for the associated record position.

#defines

The FairCom DB #define constants were modified to use a uniform style of ctWXYZ by appending ct to the beginning of all FairCom DB defines. For example, TRNLOG is changed to ctTRNLOG.

Two header files, ctv6v7.h and ctv7v6.h, provide compatibility.

Add the ctv6v7.h compatibility header file to applications written for c-tree Plus V6.x to automatically change the old defines to the new defines to allow the application to be compiled with the latest version of FairCom DB.

Add the ctv7v6.h compatibility header file to applications written for the latest version of FairCom DB to automatically change the new defines to the old defines to allow the application to be compiled with c-tree Plus V6.x. This header provides “rename #defines” to set the symbols back to their V6 convention.

The following #defines were changed:

ADMOPEN
AG_LOCK
AUTOSAVE
AUTOTRN
CHECKLOCK
CHECKREAD
CIPFASE
DEFERCP
DELUPDT
DISABLERES
DUPCHANEL
ENABLE
ENABLE_BLK
EXCLUSIVE
FREE
GETLKISAM
LK_BLOCK
LKSTATE
LOGFIL
LOGIDX
MIRROR_SKP
NONEXCLUS IVE
OPENCRPT
OVRFASE
PENDERR
PENDREN
PENDRNU
PERMANENT
PREIMG
READFIL
READREC
READREC_BLK
RESET
RESTORE
RESTORE_BLK
RESTRED
RESTRED_BLK
RESTRSV
SAVECTREE
SAVENV
SHARED
SS_LOCK
SUPERFILE
SUSPEND
TRANDEP_CRE
TRANDEP_DEL
TRANDEP_SFM
TRNBEGLK
TRNLOG
TWOFASE
VIRTUAL
VLENGTH
WRITETHRU

V4 Changes

(V4.1F-V4.3C)

Existing c-tree V4 files require conversion to the updated FairCom DB format. The following steps outline this process.

Make a BACKUP of ALL OF YOUR FILES before you begin!
Convert your existing data files with the ctcv43 file conversion utility and set these converted files aside.
Use the ctexmc parameter file create utility to create empty data and index files from your existing parameter files. (We do this as the IFIL conversion will require a valid index file to be in place, even if empty).
Copy in your newly converted data files, overwriting the empty data files just created by ctexmc. (Leave the new empty index files in place).
Convert your existing parameter files to IFIL structures with ctptoi parameter conversion utility. This utility will place an IFIL structure directly into the converted data files.
Rebuild your index files with the ctrbldif rebuild utility.
(Optional) Stamp a DODA structure (data schema) based on the 'C' structure definitions into your data file with the FairCom DB PutDODA() API call. The DODA makes available many higher level interfaces such as ODBC, FairCom DB API, .NET and FairCom DB SQL.
Include the single FairCom DB ctreep.h header file in your application to replace the previous numerous c-tree headers:
```
#include "ctreep.h"
```