To analyze where it is spending its time, the Replication Agent now supports tracking its function call times. This feature involved modifications to the Replication Agent and the repadm utility.
The repadm utility
The repadm utility's getfuncstats option reads and displays the function timings. By default, the cumulative figures are displayed, in order of highest to lowest total call time. Example:
repadm -c getfuncstats -u ADMIN -p ADMIN -s FAIRAGENT
Wed Jan 06 10:45:12 2016
function name %tot time(usec) calls avg(usec)
ADDREC 65 92147487 71237 1293
TRANBEG 20 28112621 71277 394
TRANEND 3 5067465 61559 82
ctReplGetNextChange 2 4183921 3638 1150
RWTVREC 2 4043569 61559 65
Internal 2 3279513 1 3279513
TRANRST 1 1782020 9718 183
TRANABTX 0 1029898 9718 105
ctReplPersistMinLog 0 64001 39 1641
Idle 0 18704 1 18704
When the -d option is used, the differences in stats between the previous and current time interval are displayed. For example:
repadm -c getfuncstats -d -u ADMIN -p ADMIN -s FAIRAGENT
Wed Jan 06 10:46:17 2016
function name %tot time(usec) calls avg(usec)
ADDREC 83 4244397 2799 1516
TRANEND 4 212929 2491 85
ctReplGetNextChange 3 169176 143 1183
RWTVREC 3 167174 2492 67
Internal 2 137212 1 137212
TRANBEG 1 83824 2800 29
TRANRST 1 57137 309 184
TRANABTX 0 34180 309 110
ctReplPersistMinLog 0 2741 2 1370
-c resetfuncstats resets the function timings to zero. Example:
repadm -c resetfuncstats -u ADMIN -p ADMIN -s FAIRAGENT
Successfully reset function statistics.
Notes to Aid Analyzing Function Timings
Generally, the ctRepl* function calls are made on the source server, and other function calls, such as TRANBEG, ADDREC, RWTVREC, and TRANEND are made on the target server.
The "Internal" function indicates time spent in the Replication Agent's internal operations, rather than in calls to the source and target servers. For example, the internal time includes writes to the Replication Agent's exception log.
The "Idle" function is a calculated value: When reading cumulative function times, "Idle" is the time since the statistics were last reset minus the total time used by all functions. When reading delta function times, "Idle" is the time since the last time this repadm connection read the statistics minus the time used by all functions in that time interval.
The %tot value is calculated by the repadm utility using the total time and individual function call times returned by the replication agent.
Calls to ctReplGetNextChange that time out are not included in the timings. So when the Replication Agent is caught up and waiting for additional changes from the source server, the ctReplGetNextChange function times are expected to be low, and the time is counted as idle time. Here is an example showing a Replication Agent that is caught up and waiting for more changes from the source server (using the -d option):
Wed Jan 06 10:51:17 2016
function name %tot time(usec) calls avg(usec)
Idle 99 5002207 1 5002207
Internal 0 37 1 37
ctReplGetNextChange 0 0 4 0
Replication Agent Configuration Options
Note: By default, the Replication Agent does not collect function timings. This can be changed at Replication Agent startup by adding the option function_timing on in ctreplagent.cfg. It can be changed at runtime by using the repadm utility's fnctim=on command. Likewise, the repadm fnctim=off command turns off function timing collection. The repadm utility automatically enables function timing collection when the getfuncstats option is used.
The Replication Agent now also supports an option to disable exception logging. Use the option exception_logging off in ctreplagent.cfg, or at runtime use the repadm utility's exceptionlog=off option.