|
|
|
Opening and closing files is an expensive FairCom Server I/O activity. Specific application classes stress this area more than others. For example, c-treeRTG COBOL applications frequently open and close many files. Analysis of these operations has identified specific areas for improvement, notably with high concurrent operations as these application increase connections.
Performance of FairCom Server concurrent file open and close operations has improved up to 77% in specific test cases. These improvements were especially noted when scaling on large systems, such as those with many CPU cores and concurrent database connections.
Scalability Test Results
Test programs were run on Windows 10, Solaris 11, and Linux CentOS 6, with 128 connections repeatedly opening and closing an ISAM file set consisting of one data file and three index files (a host index plus two members). Note: 2000 iterations were run on Windows and CentOS 6; only 200 iterations were run on Solaris (due to the slower CPU speed of the Solaris test box).
The number of files was varied between 1 and 128 as shown in the table below. Note that all connections opening the same file is the best case because much of the time the file is already open. Each connection opening its own file is the worst case because each open and close physically opens and closes that connection's file.
Original:
Files |
Windows 10 |
Solaris 11 |
Centos6 |
|---|---|---|---|
1 |
2.022 |
2.837 |
3.067 |
2 |
2.231 |
3.654 |
2.902 |
4 |
2.558 |
14.189 |
2.915 |
8 |
6.439 |
11.581 |
2.698 |
16 |
18.607 |
22.002 |
3.139 |
32 |
27.337 |
32.489 |
6.589 |
64 |
60.170 |
50.911 |
11.589 |
128 |
113.519 |
78.803 |
21.287 |
With file open/close optimizations:
Files |
Windows 10 |
Solaris 11 |
Centos6 |
|---|---|---|---|
1 |
1.804 |
1.894 |
2.713 |
2 |
1.795 |
1.771 |
2.687 |
4 |
1.940 |
2.149 |
2.597 |
8 |
2.036 |
2.706 |
2.575 |
16 |
2.357 |
3.686 |
2.492 |
32 |
5.241 |
7.010 |
5.355 |
64 |
12.379 |
13.462 |
10.452 |
128 |
25.849 |
27.322 |
18.778 |
For the worst case (128 connections on 128 files), the improvements are:
Windows 10: 77% reduction in time (from 113.5 sec. to 25.8 sec.)
Solaris 11: 65% reduction in time (from 78.8 sec. to 27.3 sec.)
Centos6: 12% reduction in time (from 21.3 sec. to 18.8 sec.)
These modifications made a smaller relative difference on CentOS 6 than on other systems as the original performance on CentOS 6 (21.3 sec.) was already close to the time of the optimized performance on the other systems (25.8 sec. and 27.3 sec.).
File Open / Close Configuration Options
PENDING_FILE_OPEN_RETRY_LIMIT <limit>
The pending open retry limit is set by the configuration option PENDING_FILE_OPEN_RETRY_LIMIT, which defaults to 0 (wait forever). Normally, we do not expect a file open or create to exceed the pending file open retry limit, but this limit is provided to ensure the calling function returns an error after a limited number of retries in the unexpected case that a file remains in a pending open state for a long period of time.
OPTIMIZE_FILE_OPEN <value>
OPTIMIZE_FILE_CLOSE <value>
These configuration options accept values of YES (to enable the optimization) and NO (to disable the optimization). Both of these optimizations default to YES. These options can only be set in the configuration file. They cannot be changed at run time.
The following messages in CTSTATUS.FCS indicate that these optimizations are on:
File open optimization is enabled.
File close optimization is enabled.
The following messages in CTSTATUS.FCS indicate that these optimizations are off:
File open optimization is off.
File close optimization is off.
Note: This change is a server‑side only change and it is not mandatory to relink your client applications to benefit. However, it is FairCom's best practice recommendation to always relink all client‑side applications such that the client version always matches the FairCom Server process whenever possible.
PENDING_FILE_OPEN_RETRY_LIMIT is Configurable at Runtime
It is possible to change the PENDING_FILE_OPEN_RETRY_LIMIT server setting at runtime in the usual ways (ctadmn and the ctSETCFG() API function).
Error POPN_ERR (1130) is not typically expected during normal operation. If this error occurs, increasing the PENDING_FILE_OPEN_RETRY_LIMIT setting to a larger value might prevent or reduce occurrences of that error; setting it to 0 definitively prevents the error but open requests otherwise failing may wait for long time.
See Also
POPN Error 1130
The new c-tree error code POPN_ERR (1130) is returned by a file open or create function when the operation cannot be completed because the file unexpectedly remains in a pending open state for the maximum number of pending open retries.
System File ID Lists
To prevent possible errors if two connections open the same physical file using two different aliases at the same time when the c-tree Server's file open optimization enhancement is effective, the c-tree Server now maintains a list of the system file IDs of the files that it opens. This list makes it possible to determine that a file is already open, or pending open, under a different alias. The following keywords control this new system file ID list:
SYSTEM_FILE_ID_LIST YES | NO - Defaults to YES. Enables the system file ID list. Can be turned on or off at runtime when the server is in a quiesced state.
DIAGNOSTICS SYSTEM_FILE_ID_LIST - Enables logging of adds/deletes to the system file ID list to the file SYSIDHASH.FCS in the server's LOCAL_DIRECTORY directory. Can be turned on or off at runtime.
Note: The file open optimizations (OPTIMIZE_FILE_OPEN configuration option) can also be turned off at runtime when the server is in a quiesced state.
To allow an administrator to change server configuration options that can only be changed when the server is in a quiesced state, we added an option to the ctquiet utility. The -m option can be specified one or more times in a call to ctquiet. This option quiesces the server, changes the specified configuration options, and resumes normal server activity. This process avoids additional calls to the Server, which could increase the risk of ending up with an abandoned quiesced server (meaning the process that quiesced the server has failed, leaving the FairCom Server in a quiesced (quiet) state).
See the ctquiet documentation for complete utility options
Example:
C:\> ctquiet -p ADMIN -m "optimize_file_open no" -m "system_file_id_list no"
Note: The system file ID list requires the file open optimization, so if a request is made to turn on the system file ID list, we also turn on the file open optimization, and if a request is made to turn off the file open optimization, we turn off the system file ID list.
Faster File Open and Close with Large Numbers of Open Files
To speed file‑open checks, FairCom Server stores names of open files in a hash table. As long as filenames are evenly distributed over the hash bins, each bin should contain relatively few entries. An updated hash function now more evenly distributes filenames for scalability.
Faster Return for Files Not Found on Open
Profiling found unnecessary time spent in open attempts on files that don't exist. Open operations now explicitly check if the file exists, and if not, immediately returns an error. Previously, the open functionality continued processing before returning with the final error condition.
For a file that does not exist, the file open function returns error code of FNOP_ERR with system error code (sysiocod) ERROR_FILE_NOT_FOUND on Windows and system error code ENOENT on Unix systems. Applications should always check this sysiocod value for full FNOP_ERR error context.
