FairCom ISAM for C

RebuildIFile

Incremental ISAM rebuild.

Short Name

RBLIFIL()

Type

ISAM function

Declaration

COUNT RebuildIFile(pIFIL ifilptr)

Description

RebuildIFile() rebuilds the data file and index files referenced by the IFIL structure pointed to by ifilptr. The file(s) to be rebuilt MUST be closed. RebuildIFile() opens the file(s), performs the rebuild, and closes the files upon completion.

RebuildIFile() does the following:

Resets the update corrupt flag updflg in the header of the data file.
Rebuilds the internal delete stack chain in fixed-length files and rebuilds the internal delete management index in variable-length files.
Removes the existing index file and builds an index, optimized for both size and speed, over the existing data file.
If a SRLSEG exists, RebuildIFile() finds the highest serial number in use and updates the file header with that value.

The user data is not affected in any way.

If you desire the files to be rebuilt in directories that are not specified until run-time, then ifilptr->pfilnam should not be finalized until run-time.

If RESOURCES are defined, and if ifilptr->tfilno is set to updateIFIL, RebuildIFile() automatically places the current IFIL array into a resource record in the data file - even if the data file was created without resources. After writing the IFIL structure, RebuildIFile() checks to see if the IIDX or ISEG arrays have changed. If changes are detected, the new structures replace any existing structures in the data file or will be added if no structures were previously present.

Note: Do not use RebuildIFile() to add indexes to an existing index, since the member headers are all at the front of the physical file with the key values afterward, and there is no slack space to add new member headers without recreating the physical index file. Therefore, first delete the physical index file and then do the rebuild if additional index members are added. You may also use PermIIndex() or TempIIndexXtd().

Rebuild times can be optimized by increasing the number of temporary sort files used. Increase the #define MAX_HANDLES found in ctsort.h from its default of 50 to a larger number up to the maximum of 255. The value to use is a function of the size of the index to be rebuilt and the number of available file handles and is best determined by performing timing tests on the target system. The temporary files created during rebuild are c-tree files. After changing ctsort.h, recompile the entire c-tree library, including removing the object files and library, and the rebuild application.

The path and/or drive of the temporary sort files can be modified in non-server mode with minor changes to ctsort.h and ctsort.c as follows:

Change approximately line 32 of ctsort.h from:

#define WORK_DRIVE_OR_PATH ""

to:

pTEXT WORK_DRIVE_OR_PATH;

Now declare WORK_DRIVE_OR_PATH as an ‘extern’ in the rebuild application and assign the desired path. See also the GetCtTempFileName() function description.

A second method for decreasing rebuild times is to link the rebuild application with a single-user c-tree library so that index caching will be used. To increase the size of the index cache, increase the number of index buffers (i.e., bufs parameter to InitISAM() or the first number of the parameter file initialization record). The index cache uses a hashing algorithm, therefore the larger the index cache the faster the rebuild will be. The size of the index cache can be calculated as follows:

memory(in bytes) = bufs *(sect * 128 + MAXLEN + 128)

MAXLEN, the maximum key length defined in ctopt2.h, defaults to 1024 bytes.

Option to skip initial data file scan

Normally the rebuild function starts by scanning the data file, checking for valid record marks if it is a variable-length data file, and checking that the logical and physical end of file values are correct. If the data file is known to be in a good state, it can be beneficial to skip this scan. OR the skipdatascanIFILoption bit into the tfilno field of the IFIL structure that you pass to the rebuild function. When using this and other options, remember to negate the tfilno value after you OR in the options. For example:

myifil.tfilno = -(redosrlIFILoption | skipdatascanIFILoption);

RBLIFIL(&myifil);

Automatic Duplicate Purge Logic

Prior to c-tree V6.7, rebuilding a file with duplicate key values on an index for which duplicates were not allowed terminated the rebuild completely or left the indexes in an inconsistent state. The index not supporting duplicates would not have an entry for the record while the other indexes would have an entry for the record. For consistency, this is still the default behavior, but details concerning the duplicate keys are sent to a temporary text file instead of the console. The name of the temporary file is sent to the console and to CTSTATUS.FCS. This information is not subject to the RB_OUTPUT define. It happens automatically and may not be disabled.

Records containing the unwanted duplicates are marked deleted, and none of the indexes will have an entry for this record. A copy of each deleted duplicate record is stored in a temporary file, which notes the position, length, and content of the record. CTSTATUS.FCS contains entries indicating how many duplicate keys were rejected, the name of the data file, the name of the temporary file, and a message indicating successful completion of the duplicate purge.

To implement the automatic duplicate purge behavior, where records with duplicate keys are purged from the data file and copied into a temporary file, one of the following two approaches is used:

Incremental ISAM Structure Support (IFIL): Rebuilding files defined by an Incremental ISAM Structure is typically accomplished with RebuildIFile(). Setting ifilptr->tfilno to purgeIFIL, which is defined in ctport.h, instigates the new auto-purge logic. It is permissible to set ifilptr->tfilno to purgeIFIL + updateIFIL so both behaviors can be achieved.
Parameter Files: For parameter files rebuilt with ctrbld.c, add #define ctRBLD_AUTO_PURGE when compiling ctrbld.c. This causes the parameter file rebuild to attempt automatic record purge if duplicate keys are rejected.

Note: If the record is longer than the operating system maximum unsigned integer (MUI), only MUI bytes are stored in the temporary file for this record. Typically, this will only be an issue with 16-bit systems where MUI will be 64K bytes.

To re-instate a deleted record, open the temporary file using c-tree open calls, retrieve the desired record, and insert it back into the data file.

If purgeIFIL is used and a binary stream file can be opened, duplicate keys and bad serial numbers are automatically purged from the indexes and the data records are deleted from the file. RebuildIFile() returns DUPJ_ERR (650) and the temporary stream file contains the deleted data records. This is an informational, not fatal, error.

If purgeIFIL is NOT used and an ASCII stream file can be opened, the duplicate keys and bad serial numbers are not added to the indexes, but the data records remain in the file. RebuildIFile() returns DUPL_ERR (652), and lists the offending keys and data records in the stream file. Before V6.7, RebuildIFile() returned NO_ERROR (0) under this condition. DUPL_ERR is an informational, not fatal, error.

Option Values

Prior to FairCom DB V9.5, several option values were defined such as updateIFIL, purgeIFIL, and badpartIFIL, that can be specified. These values are specified by adding them together. For example: myifil.tfilno = updateIFIL + purgeIFIL. A new approach simplifies checking of these options. The following option values are now specified in ctport.h:

#define updateIFILoption 0x0002

#define purgeIFILoption 0x0004

#define badpartIFILoption 0x0008

#define redosrlIFILoption 0x0010

These values can now be OR-ed together and, using the setIFILoptions() macro, assigned to the tfilno field of the IFIL structure passed to the compact or rebuild function. For example, to indicate that you want to assign new serial numbers, do the following:

myifil.tfilno = setIFILoptions(redosrlIFILoption);

RBLIFIL(&myifil);

If you want to also use more than one option when compacting or rebuilding a file, OR them in to the value. For example:

/* assign new serial numbers and update IFIL resource. */

myifil.tfilno = setIFILoptions(redosrlIFILoption | updateIFILoption);

RBLIFIL(&myifil);

If a stream file cannot be opened, RebuildIFile() returns either KDUP_ERR (2) or KSRL_ERR (605) when duplicate keys and/or bad serial numbers are encountered. Further, no more indexes are processed. That is, the rebuild does not run to completion as it does in the above two cases.

To rebuild a mirrored file, ensure the mirror is specified in ifilptr. RebuildIFile() copies the primary file to the mirrored name on successful completion.

badpartIFILoption (badpartIFIL)

The rebuild should reference the partition host. If the partition host can be opened, we attempt to open each member and rebuild only the members that get an error on open. If the partition host itself fails to open, we rebuild the host and all members.

Online Rebuild - new for V13

An online index rebuild has been implemented. It requires the target file to have the ctTRNLOG file mode (full transaction control).

To invoke an online rebuild, use onlineIFILoption:

myifil.tfilno = setIFILoptions(onlineIFILoption);

RBLIFIL(&myifil);

The file must be closed by the calling user, as for a normal rebuild, but may be open by OTHER users for update. The IFIL resource embedded in the file is used, not the IFIL definition provided by the caller. Behavior is undefined if the IFIL resource embedded in the file doesn't match the definition used to create the indexes.

NOTES: Online rebuild uses the dynamic dump and immediate restore, so it requires the ctrdmp binary exists in the server working directory. Only one dynamic dump may occur at a time, so the online rebuild will wait to begin if backups or other online rebuilds are in process. When the new index is ready, access to the file will be suspended while the index is replaced. Any active transactions that have updated this file will be aborted at this point. If an error occurs after this point, the index will need to be rebuilt in exclusive mode. The online rebuild is not supported for memory files, partition files, segmented files, or superfiles. Files with deferred indexes are not currently supported.

Return

Value	Symbolic Constant	Explanation
0	NO_ERROR	Successful rebuild of ISAM files.
12	FNOP_ERR	Could not open file(s). Check isam_fil for the specific file number.
20	KMIN_ERR	Key length too large for node size.
22	FNUM_ERR	File number is out of range.
45	KLEN_ERR	Key length exceeds MAXLEN parameter in ctoptn.h.
46	FUSE_ERR	If ifilptr->dfilno >= 0, file number (range) already in use. If ifilptr->dfilno < 0, no block of numbers available.
107	IDRK_ERR	Too many index files for one data file. Increase MAX_DAT_KEY parameter in ctoptn.h.
109	IKRS_ERR	Too many key segments. Increase MAX_KEY_SEG parameter in ctoptn.h.
115	ISLN_ERR	Key segments do not equal key length.
120	RRLN_ERR	Not enough dynamic memory for record buffer.
122	RMOD_ERR	Attempt to change between fixed and variable-length records.
123	RVHD_ERR	A variable-length record is not preceded by a valid record mark.
454	NSUP_ERR	Mirrored file rebuilds are not supported. Rebuild the primary if necessary and then copy to the mirrored file name.
484 485		Error opening sortwork file. Increase the number of file handles requested at c-tree initialization time (InitISAM(), InitISAMXtd() or second value in parameter file Initialization Record).
605	KSRL_ERR	Bad serial number.
650	DUPJ_ERR	Informational: Duplicate records successfully purged.
652	DUPL_ERR	Duplicates found and listed in stream file.

See c-tree Error Codes for a complete listing of valid c-tree error values.

Example

extern IFIL customer;

COUNT retval;

main() {

if (retval = InitISAM(10,10,16))

printf("\nInitISAM error = %d",retval);

if (!(RebuildIFile(&customer)))

printf("\nSuccessful rebuild");

else

printf("\nRebuildIFile isam_err = %d",isam_err);

CloseISAM();

}

Limitations

RebuildIFile() can be called from applications linked with the Standalone Multi-user (FPUTFGET) mode. RebuildIFile() opens the files in ctEXCLUSIVE mode, so no processes may access the file during rebuild. For performance reasons, single-user rebuilds are recommended.

To rebuild superfiles, call SuperfilePrepassXtd() prior to RebuildIFile().