Doubling FairCom DB workload on an IBM AIX system has been observed to more than double CPU utilization or generate very high CPU utilization. An AIX environment variable can be changed and the server restarted to compare performance.
The following environment variable is used:
We found that 32-bit c-treeSQL Server on AIX requires EXTSHM ON to be set in the environment before it is started, otherwise attaching to a shared memory region may fail with a "too many files" error (EMFILE, system error code 24). With FairCom Servers V12 and later a message is also logged for 32-bit AIX environments with this tip included. The following message is now logged to CTSTATUS.FCS when this error occurs:
Wed Oct 21 08:21:05 2020
User# 00024 FSHAREMM: at_SessionReallocCommBuffer: shmat for comm mem resize
failed: 24 Verify that EXTSHM ON is set in the environment before starting the process.
When running a 64-bit FairCom DB Server on AIX, it was suspected memory pages were being "stolen" by another process or the file system cache. The key diagnostic indicator in this case was to monitor RSS memory of the FairCom DB process (ps v PID) and see if it drops at times of poor performance.
It was found that the FairCom DB executable and/or the system needed to be set for large page support.
The instructions found on the following pages were followed:
The following commands were then run as root:
vmo -p -o v_pinshm=1 -o lru_file_repage=0
vmo -p -o lgpg_size=16777216 -o lgpg_regions=16
chuser capabilities=CAP_BYPASS_RAC_VMM,CAP_PROPAGATE fctech
ldedit -b lpdata ctreesql
There have been sporadic reports of poor FairCom DB performance when running on AIX operating systems. Symptoms reported include:
It may be possible to improve performance, in some cases, by adjusting the AIX environment variables SPINLOOPTIME and YIELDLOOPTIME, which change mutex behavior.
The above link says that SPINLOOPTIME defaults to 40 and YIELDLOOPTIME defaults to 0.
Try setting SPINLOOPTIME=650 and YIELDLOOPTIME=10, restart the server and see if this improves the situation. Further tuning may give greater benefits.
A previously activated server was found to not be activated with a newly provided activation key. The fcactvat utility failed with system error 26 "Text file busy or in use".
AIX 6 can cache shared objects, in this case ctreedbs.so, and fcactvat can then not stamp the binary. An AIX 6 utility, slibclean, is available on that platform that releases the object from the cache allowing successful stamping.