Product Documentation

V11.5 Update Guide

Previous Topic

Next Topic

NO_FIXED_OFFSET_CHECK - Prevent reads/writes to arbitrary offsets for fixed-length files

In V11.5 and later, a feature prevents corruption and errors when an attempt is made to write to arbitrary offsets in a fixed-length file. Checks were added to ensure that a read/write/lock call on a fixed-length file is properly aligned. If an incorrect alignment is detected, the call will fail with FALG_ERR (767).

This feature can be disabled by keyword COMPATIBILITY NO_FIXED_OFFSET_CHECK.

Note: This is a Compatibility Change.

Previous Topic

Next Topic

Preventing Possible Data Loss with Compact & Rebuild Operations

In most cases, compact and/or rebuild operations perform as expected: scanning your data file for potential invalid record headers, fixing those it finds invalid, and finally writing out a new compressed file and rebuilding indexes. Recently, a case was investigated where our default behavior should not be used due to potential data file corruption.

In a few rare and select cases, FairCom identified situations where compacting a variable-length and (typically) non-transaction enabled (non-TRNLOG) file, using the ctcmpcif compact utility or any of the CompactIFile functions, can fail, potentially, and unexpectedly, also altering your original file being compacted such that further attempts result in data loss.

Typically, it is only expected to see this situation occur with non-TRNLOG enabled files. However, in case #2 discussed below, a valid TRNLOG enabled file data file was found to exhibit this very unusual behavior. This raises an important point – if you encounter situations where you believe you must rebuild TRNLOG enabled files, please contact FairCom, as this is a highly unexpected event and we want to fully understand the context and nature of your data integrity situation. The recoverable nature of TRNLOG files should always maintain full data integrity through nearly any circumstance.

We have modified prior behavior to take a much more conservative approach protecting your original data state. Compact and rebuild (which can produce similar results) facilities listed below have been altered such that, when any data file integrity issue is detected, default behavior is now to stop further operations. In addition, no changes are made to existing original data records.

Note: This is a compatibility Change.

Full List of Functions Changed

The following utilities and function calls now default to stopping if an error is encountered:

ctcmpcif – Compact utility
ctrbldif – Rebuild utility
GUI tool - Dr. ctree has compact and rebuild functionality

Long function name

Short function name

CompactIFile()

CMPIFIL()

CompactIFileXtd()

CMPIFILX()

CompactIFileXtd8()

CMPIFILX8()

RebuildIFil()

RBLIFIL()

RebuildIFileXtd()

RBLIFILX()

RebuildIFileXtd8()

RBLIFILX8()

Default Behavior Changes the Original File

Prior to c-treeACE V11.5, the default behavior for compact and rebuild operations over variable-length record data files was to scan for invalid record headers before compacting a file. If the logic found invalid headers, it attempted to fix the headers in the original file. If the operation failed at this point, the original file could become corrupted. Two significant classes of situations are known to arise:

  1. In the vast majority of cases, this can only occur if the original data file is already corrupted due to invalid and irrecoverable record headers. Because the original data file is corrupted, restoring a backup is recommended.
  2. In some rare cases, it is possible the operation could exhibit vulnerability caused by mistaking old marks or binary record data as valid record header marks. In a very specific case, compact operations can treat space management nodes with inuse length 0, while logically a correct value, as invalid. This case could lead to data loss in a valid file, making it necessary to restore from a backup.

To ensure the second case does not cause data loss, observe our recommendations below and always back up all concerned data files before compacting or rebuilding.

Compact and Rebuild Best Practice

To avoid potential data loss, FairCom strongly recommends only using our latest c-treeACE V11.5 compact and rebuild logic that default to stopping if a corrupt data record header is encountered, and returns error DCPT_ERR (1107) rather than attempting to repair the data link in the record header. This allows an administrator to know with greater certainty that serious damage has occurred and they may prefer to recover the file from a backup.

If an administrator prefers allowing c-treeACE to attempt to recover a file, as done prior to V11.5, the following options are available to revert to prior behavior:

  • -repairdata command-line switch - Use this switch with the ctcmpcif and ctcmpcif compact utilities, e.g., ctcmpcif -rdata
  • -repairCorruptIFILoption option - The repairCorruptIFILoption option should be OR-ed into the tfilno of the rebuild (RebuildIFIL()) and compact (CompactIFile()) functions using the setIFILoptions() macro.

    For example:
    myIFIL.tfilno = setIFILoptions(repairCorruptIFILoption | updateIFILoption);

Best Practices:
1.) It is always important to back up concerned data files before you attempt to compact or rebuild them.
2.) Ensure sufficient disk space is available before starting a compact or rebuild. You will need up to at least 3X the size of the data file plus index files (original files + new files + copy) for a successful compact or rebuild.
3.) Consider keeping a copy of the data file for sufficient time and ensure all expected data is available before removing the backup.

TOCIndex