FairCom ISAM for C

Incremental ISAM Structures

Incremental ISAM structures permit your application to create (CreateIFile()), open (OpenIFile(), OpenFileWithResource()), close (CloseIFile(), CloseRFile()) and rebuild (RebuildIFile()) individual ISAM data file / index file combinations. Three structures work together to enable the incremental ISAM capability: IFIL defines the data file, IIDX defines the indexes, and ISEG defines the key segments. These structures are defined in the ctifil.h header file.

In This Section

Incremental ISAM Example

IFIL Structure

All the incremental ISAM functions require a pointer to an IFIL structure as their input parameter. The IFIL structure defines the characteristics of the data file and includes a pointer to a structure that defines the associated indexes.

The formal type definition for IFIL follows:

typedef struct ifil {

pTEXT pfilnam, /* file name (w/o ext) */

FILNO dfilno, /* data file number */

UCOUNT dreclen, /* data record length */

UCOUNT dxtdsiz, /* data file ext size */

COUNT dfilmod, /* data file mode */

COUNT dnumidx, /* number of indexes */

UCOUNT ixtdsiz, /* index file ext size */

COUNT ifilmod, /* index file mode */

pIIDX ix, /* index information */

pTEXT rfstfld, /* r-tree 1st fld name */

pTEXT rlstfld, /* r-tree last fld name */

FILNO tfilno /* temporary file number */

} IFIL;

Note: pTEXT is a c-tree data type that equates to TEXT *. For example, pTEXT pfilnam is the same as TEXT *pfilnam. The same holds true for all ‘p’ data types, i.e., pISEG = ISEG *.

In V12 the file number typedef was formally changed from COUNT, a two-byte value to FILNO, a four-byte value. Refer to this link for compatibility details. Four Byte File Numbering

Corresponding IFIL Member	c-tree Parameter	Interpretation
pfilnam	filnam	The file name pointed to by pfilnam is shared by both the data file and index file. The name should not include the file extension. .dat and .idx (as defined in ctifil.h) are added to the name for the data file and index file. The extensions can be changed using extended function calls. The file name must agree with the operating system’s file naming conventions.
dfilno	datno	The file numbers assigned to the data file and any associated indexes run consecutively. If dfilno is negative, c-tree finds the first available block of numbers. If dfilno is positive, c-tree assigns dfilno as the data file number. In either case, tfilno is set equal to the data file number.
dreclen	datlen	Record length for fixed-length records or the fixed-length portion of variable-length records.
dxtdsiz	xtdsiz	Data file extension size. When the data file needs to be extended, this is the number of bytes the file grows.
dfilmod	filmod	Data file mode indicator. See Data Management.
dnumidx	N/A	The number of associated indexes (which may be zero).
ixtdsiz	xtdsiz	Index file extension size. Same as dxtdsiz, but for the index file.
ifilmod	filmod	Index file mode indicator. See Data Management.
ix	N/A	Pointer to IIDX index structures defined below.
rfstfld	N/A	An optional pointer to the symbolic name of the first data field in the record, as specified in the r-tree DODA. This structure entry and the next one are required for r-tree and d‑tree users only.
rlstfld	N/A	An optional pointer to the symbolic name of the last data field in the record.
tfilno	N/A	Output parameter set equal to the file number assigned to the data file. For greatest flexibility, use this argument for functions that request the data file number (datno or dfilno).

File Names up to 4K

With V12 and prior, c-tree limited file name lengths to 255 byte. Starting with V13, this optional support allows for file name lengths up to 4K in size. This limit is controlled by a compile-time macro, MAX_NAME. Modern operating systems support file names over 255 bytes and in this revision will allow MAX_NAME to be increased, supporting file names up to 4096 bytes.

Note: When enabled, these changes break compatibility with existing c-tree binaries and dynamic dump backups.

Windows Notes:

Historically, Windows API functions have limited file names passed to its file and directory functions to MAX_PATH (260) bytes. Starting with Windows 10, version 1607, this limit has been removed from common Windows API file and directory functions. For information about Windows file naming conventions, see: Microsoft Windows 10 - Naming Files, Paths, and Namespaces.

As noted on that page, you must opt in to the new behavior. To enable this support:

Create the registry key HKLM\SYSTEM\CurrentControlSet\Control\FileSystem LongPathsEnabled (Type: REG_DWORD) and set it to 1. Then reboot the system.
Create a manifest file named longPath.manifest containing the following text:
<?xml version="1.0" encoding="UTF-8" standalone="yes"?>

<assembly xmlns="urn:schemas-microsoft-com:asm.v1" manifestVersion="1.0">

<application xmlns="urn:schemas-microsoft-com:asm.v3">

<windowsSettings xmlns:ws2="http://schemas.microsoft.com/SMI/2016/WindowsSettings">

<ws2:longPathAware>true</ws2:longPathAware>

</windowsSettings>

</application>

</assembly>

In the Visual Studio project settings for each executable that uses Windows API functions with a file name over 260 bytes (such as ctsrvr.exe and ctrdmp.exe), specify the name of this manifest file under Properties | Manifest Tool | Input and Output | Additional Manifest Files. Note this has already been done in the makefiles created by FairCom’s mtmake make builder.

mtmake Parameter Adds Manifest for Long Paths

A new mtmake 4Kfilepaths parameter has been added to enable the extra manifest for longPathAware builds.

{"4Kfilepaths", "Add manifest to support 'longPathAware' 4K file paths.",MTOPTN_NORMAL},

IIDX Structure

Each index is described by an IIDX structure. If a data file has associated indexes, the ix member of the IFIL structure must point to an array of IIDX structures. In particular, there must be dnumidx elements (i.e., IIDX structures) in the array.

Each IIDX structure defined over a given data file receives a unique file number assigned sequentially from the data file number tfilno. For example, if tfilno is file number 5, and this data file has 3 indexes, they receive file numbers 6, 7 and 8. 6 is assigned to the IIDX structure pointed to by the IFIL element ix, 7 is assigned to the next IIDX structure, and 8 to the last IIDX structure for this particular data file. Since each IIDX structure gets its own file number, there is no difference in performance between placing all IIDX definitions for a particular data file in the same physical file or in individual physical files.

The formal type definition for IIDX follows:

typedef struct iidx {

COUNT ikeylen, /* key length */

ikeytyp, /* key type */

ikeydup, /* duplicate flag */

inulkey, /* NULL key flag */

iempchr, /* empty character */

inumseg, /* number of segments */

pISEG seg, /* segment information */

pTEXT ridxnam, /* r-tree symbolic name */

pTEXT aidxnam, /* optional index file name */

pCOUNT altseq, /* optional alternate sequence */

pUTEXT pvbyte /* optional pointer to pad byte */

} IIDX;

Corresponding IIDX Members	c-tree Parameter	Interpretation
ikeylen	keylen	Key length.
ikeytyp	keytyp	Key type. See the table of ikeytyp values below.
ikeydup	keydup	Duplicate flag.
inulkey	N/A	Null key flag.
iempchr	N/A	Empty character.
inumseg	N/A	Number of key segments.
seg	N/A	Pointer to ISEG key segment structures.
ridxnam	N/A	Pointer to optional symbolic index name for use with r-tree.
aidxnam	filnam	Pointer to an optional index file name.
altseq	N/A	If not NULL, this points to an array of 256 short integers that specify an alternate collating sequence.
pvbyte	N/A	If not NULL, this points to a byte value to be used for trailing padding compression. An ASCII space (0x20) is the default padding character.

The following text provides additional information about select IIDX elements:

ikeylen - The sum of the segment lengths (defined below). If duplicates are allowed, the last 4 bytes of the key value are reserved for the 4 bytes of the associated data record offset. For example, a key length of 12 bytes allowing duplicates has 8 bytes of actual key value and 4 bytes for the tie-breaking data record offset. DropIndex() marks a transaction-controlled index for deletion by setting keylen to -keylen until the drop is committed. A GetIFile() call after the DropIndex() call, but before the Commit() or Abort() will return this interim setting.

ikeytyp - Valid key types are listed below. For additional information, consult Alternative Key Types.

Value	Mnemonic	Explanation
0		fixed-length key
0x4	COL_PREFIX	leading-character compression - legacy
0x8	COL_SUFFIX	padding compression - legacy
0xC	COL_BOTH	leading/padding compression combined (key types 4 and 8) - legacy
0x80	KTYP_NOISMKEYUPD	do not allow an ISAM update to change the key value
0x608	KTYP_VLENGTH	variable-length keys - keys over variable length fields
0xE00	KTYP_VLENGTH_SRLE	variable-length keys - simple RLE key compression for keys over variable length fields

Value	Explanation
0	Unique index. Duplicate keys not allowed.
1	Supports duplicate key values. Key values in the index contain additional bytes associated with the record offset value (either 4, or 8 bytes).
2	Unique key values within a file partition. (Partitioned file indexes only)
3	Supports duplicate key values with statistical information on distinct key counts; whole and partial keys. (Applies to partial keys in the case of multiple segments.)
4	Unique index with statistical information on distinct key counts for partial keys (in the multiple segment case).
5	Supports duplicate key values with statistical information on distinct key counts for the entire key.

Value	Explanation
0	No check is made.
1	If every byte of the resulting key value is the empty character, iempchr, don't add the key to the index file.
2	If the full key value matches the null key value set for the index, don’t add the key to the index file.
3	If any segment of the key value matches its corresponding portion of the null key value set for the index, don’t add the key to the index file.

ISEG Structure

Each key value is composed of one or more segments. A segment is simply a sequence of bytes from a specified offset within the data record. Each key segment is described by an ISEG structure. The seg member of each index IIDX structure must point to an array of ISEG structures. In particular, there must be inumseg elements (i.e., ISEG structures) in the array. The formal type definition for ISEG follows:

typedef struct iseg {

COUNT soffset, /* segment position (offset) */

slength, /* segment length */

segmode /* segment mode */

} ISEG;

Segment Position

The segment position specifies the location of the key segment within the data record. This can be specified in one of three ways, as determined by the segment mode value (see Key Segment Modes):

Absolute byte offset. The number of bytes from the beginning of the data record to the first byte of the key segment. The smallest offset, zero, means that the segment begins with the first byte of the record. If a segment begins with the nth byte of the record, its offset value should be n minus one.
This option works with fixed length fields, which have known offsets.
Relative field number. The segment position is treated as a field number, not a byte offset. This applies only to variable-length data records. With variable-length records, all of the fields in the variable-length portion are variable-length fields terminated with a delimiter. This delimiter is typically the NULL character, unless the default is changed by SetVariableBytes(). A segment position of zero corresponds to the first variable-length field, beginning at the first byte beyond the variable-length file’s fixed-length portion. When a variable-length file is created, the record length parameter is treated as the minimum record length, which corresponds to the fixed-length portion of the record.
This option works with variable-length string types using NULL terminators.
The following schematic demonstrates how varying-length fields are numbered. The delimiter is shown as a NULL byte, but can be any delimiter you set:
Record Schema field number. When using a Record Schema, (Record Schemas), use a schema segment type and specify the segment position by the field number in the record schema. The first field in the schema is field zero.
This option works with any type of data in any part of the record, allowing key segments on non-variable-length data types in the variable-length portion of the record. All fields in the variable-length portion must be packed as opposed to aligned. No padding bytes are allowed between fields.

Though segoffset is a COUNT field, since a fixed length file or the fixed length portion of a variable-length file may exceed 32767 bytes, segment offsets exceeding 32767 bytes, but less than 65535 bytes, are supported. The internal logic deals with segment offsets that “go negative”. A minus one for a segment offset (equivalent to 65535 for a UCOUNT) still signifies the “end of segments” so that segment offsets can now range from 0 to 65534. If a value greater than 32767 is stored in a COUNT field, it appears as a negative value; but its unsigned integer value correctly corresponds to the original value. For example, storing 65534 into a COUNT results in a value of -2 when expressed as a COUNT, but -2 cast to a UCOUNT corresponds to 65534.

Segment Length

The segment length specifies the number of bytes that this key segment will take in the key itself. The sum of the segment lengths for an index file must equal the key length unless duplicate keys are allowed. In this case, the segments should sum to the key length less 4 bytes. If the sum of the segment lengths does not match the key length, CreateISAM(), OpenISAM(), CreateIFile(), or OpenIFile() return error ISLN_ERR (115).

For variable-length fields, segment modes 4 and 5, c-tree automatically pads a key segment to the full segment length using the padding byte. The padding byte defaults to an ASCII space (0x20), but can be changed using SetVariableBytes(). When using a field other than a variable-length field, make the segment length less than or equal to the actual size of the field.

Segment Mode

The segment mode determines what transformations, if any, must be made to the key segment. In addition, it determines which of the three methods of determining the segment position will be used.

Detailed information on key segment modes is presented in the Key Segment Modes section. When using a Record Schema to manage the building of keys, refer to Record Schemas for more information.

Incremental ISAM Example

The following C declarations show how the variable-length data base example shown before can be expressed in terms of initialized incremental ISAM structures. Because of the nested relationship of IFIL, IIDX, and ISEG, we define the segment information, then the index information, and then the data file information. For the sake of completeness, we added the optional r-tree symbolic names.

Note: (pTEXT) 0 is the same as (TEXT *) 0 and likewise for pCOUNT and pUTEXT.

ISEG vc_seg[] = {

{0,15,5}, /* 1st segment, 1st index */

{4, 5,2}, /* 2nd segment, 1st index */

{0, 4,0}, /* 1st segment, 2nd index */

{4, 9,2}, /* 1st segment, 3rd index */

{0, 9,5} /* 2nd segment, 3rd index */

};

IIDX vc_idx[] = {

{24, /* key length, 1st index */

12, /* key type */

1, /* duplicate flag on */

1, /* NULL key flag on */

32, /* empty char (decimal) */

2, /* number of key segments */

&vc_seg[0], /* pointer to segment array */

"name_key", /* r-tree symbolic index name */

(pTEXT) 0, /* optional index name */

(pCOUNT) 0, /* alternate collating sequence */

(pUTEXT) 0 /* optional pointer to pad byte */

{ 4, /* key length, 2nd index */

1, /* key type */

0, /* duplicate flag off */

0, /* NULL key flag off */

0, /* empty char ignored */

1, /* number of key segments */

&vc_seg[2], /* pointer to segment array */

"numb_key", /* index name */

(pTEXT) 0, /* optional index name */

(pCOUNT) 0, /* alternate collating sequence */

(pUTEXT) 0 /* optional pointer to pad byte */

{22, /* key length, 3rd index */

12, /* key type */

1, /* duplicate flag on */

1, /* NULL key flag on */

32, /* empty char (decimal) */

2, /* number of key segments */

&vc_seg[3], /* pointer to segment array */

"zipc_key", /* index name */

(pTEXT) 0, /* optional index name */

(pCOUNT) 0, /* alternate collating sequence */

(pUTEXT) 0 /* optional pointer to pad byte */

}

};

IFIL vc_dat = {

"vcust", /* root of file name */

1, /* data file number */

15, /* minimum record length */

4096, /* data file extension size */

5, /* data file mode */

3, /* number of indexes */

4096, /* index file extension size */

1, /* index file mode */

vc_idx, /* pointer to index array */

"vcust_1st", /* 1st data field */

"vcust_last" /* last data field */

/* NULL: tfilno */

};

With the above structures, the incremental ISAM functions, (CreateIFile(), OpenIFile(), CloseIFile(), RebuildIFile()), are called with a single input parameter: &vc_dat. &vc_dat is a pointer to an IFIL structure that points to the array of index structures and segment structures.

It is not necessary to put all the segment values in one array. You could use separate arrays of ISEG structures for each index. However, all indexes for a single data file must be in a single array of IIDX structures.