c-treeACE V10.0 Update Guide

Shared Memory Client-Server Communication for Unix/Linux

The FairCom Server for Unix supports shared memory connections. Shared memory communication between clients and servers residing on the same machine generally provides much better performance for locally running applications. Local shared memory connections are supported across the board including ISAM and SQL connections, and this includes JDBC and Windows ADO.NET Data providers.

Configuration

Include the following server configuration in ctsrvr.cfg to enable this support:

COMM_PROTOCOL FSHAREMM

FairCom client libraries are compiled with this feature enabled by default.

Note: The COMM_PROTOCOL option specifies the protocol used for ISAM connections. By default, local SQL connections use shared memory unless the SQL_OPTION NO_SHARED_MEMORY keyword is specified. See the c-treeSQL Server Operations and Utilities Guide for more information about the communication protocol for SQL connections.

System Files, Permissions and Ownership

The FairCom shared memory communication protocol creates a file used by clients to find the shared memory identifier for its shared memory logon region, and creates a Unix domain socket as a file for initial communication between a client and server.

The FairCom Server creates the directory /tmp/ctreedbs and the file /tmp/ctreedbs/<servername>.logon. This file name is determined by the value specified with the SERVER_NAME configuration option (but see important note below). This file contains an identifier of a shared-memory region used for clients to connect. The following configuration option allows this directory to be directly specified:

SHMEM_DIRECTORY <directory_name>

IMPORTANT: SERVER_PORT applies to the TCP/IP protocol and overrides SERVER_NAME if both are used together.

If your server combines shared memory and TCP/IP usage, here are a few tips:

If you are content with the TCP/IP port resulting from the SERVER_NAME option, then use that option and you can connect using the name with either protocol.
If you wish to explicitly set the TCP/IP port, use SERVER_PORT to set that port (then connect with #port to use TCP/IP on that port) and SERVER_NAME to set the name used by the shared memory protocol. Note that this approach means that a connection attempt will not be able to 'fall back' to using TCP/IP if the shared memory connection fails, unless you choose your server name so that it matches your SERVER_PORT setting. For example, consider the following set of options:

SERVER_PORT 7000

SERVER_NAME #7000

Then connect with a server name of #7000. The client will attempt to connect using shared memory first and if that fails it will connect with TCP/IP on port 7000.

Note: on some Linux/Unix systems, be sure to place double quotes around the connection string (for example when using the bash shell): For example: ./ctstat -vas -u admin -p ADMIN -s "#5598ocalhost"
Without the double quotes, bash treats # as a comment, so it ignores the remainder of the connection string.

c-treeACE must have sufficient read, write, create, and delete permissions with this directory. The following server keyword sets the shared memory resource permissions:

SHMEM_PERMISSIONS <permissions>

The default is 660. 666 will allow access to c-treeACE by any user account.

Note: Use caution when increasing access permissions to shared memory resources. For example, shared memory permission of 666 allows any user to attach to a shared memory segment and read or write to it. This means that any process can make a request to a FairCom Server or could read the request data of another process through such a shared memory region.

By default, a client application must belong to the server owner’s primary group to use shared memory. This is configurable with the SHMEM_GROUP keyword.

SHMEM_GROUP <group>

Possible errors indicating problems:

FSHAREMM: Could not get group ID for group <group> for shared memory

FSHAREMM: Failed to set group for LQMSG shared memory region: X

Shared Memory Keys

When more than one FairCom process is run on a Unix system, the shared memory key used by the servers might hash to the same value, causing problems connecting to the servers. This happens as the ftok() system call is used by default to generate the shared memory keys, and ftok() is not guaranteed to return unique values. Another possibility is that another unrelated process might happen to use the same shared memory key as generated by the FairCom Server.

An administrator can specify a specific shared memory key for ISAM and SQL shared memory communication protocols to ensure that keys do not match keys already in use on the system. This is specified with the following FairCom configuration options:

SHMEM_KEY_ISAM <isam_shared_memory_key>

SHMEM_KEY_SQL <sql_shared_memory_key>

The shared memory key values can be specified in either decimal or hexadecimal format. For example:

; Set shared memory key for ISAM connections to the specified decimal value:

SHMEM_KEY_ISAM 12345

; Set shared memory key for ISAM connections to the specified hexadecimal value:

SHMEM_KEY_ISAM 0xabcd

These server configuration options support specifying an environment variable, whose value is substituted for the configuration option value when the server starts up. For example, if the environment variable MY_ISAM_KEY is set to a numeric value such as 12345 or 0xabcdef before starting the server process, then the following option can be specified in the server configuration file to use this environment variable value for the SHMEM_KEY_ISAM configuration option value:

SHMEM_KEY_ISAM %MY_ISAM_KEY%

Client Configuration

From the client side, either set the global variable ctshmemdir to the directory name before connecting, or set the CTREE_SHMEM_DIRECTORY environment variable. The environment variable takes precedence over the ctshmemdir setting. This allows the directory to be dynamically overridden without having to recompile client code.

Errors with Shared Memory Protocol

The FairCom Server logs error messages to CTSTATUS.FCS when a shared-memory connection attempt fails. The message is of the form:

FSHAREMM: <error message>

Adjusting System Limits

When running the FairCom Server with more than 128 shared-memory connections, you may encounter one of the following errors:

FSHAREMM: Connect named pipe failure: 13

FSHAREMM: Connect named pipe failure: 28

Many Unix/Linux implementations have a default limit of 128 system semaphores, which are used by FairCom shared memory connections. However, this value applies system-wide among all processes.

FSHAREMM: Failed to create system semaphore: check system semaphore limits such as SEMMNI

The following error can be reported as well:

FSHAREMM: Failed to create shared memory segment: check shared memory limits such as SHMMNI

These are typically kernel configurations. The FairCom Server requires (2 + # shared memory CONNECTIONS) shared memory segments (SHMMNI) and semaphores (SEMMNI).

The ipcs command displays current limits:

#ipcs -l

------ Semaphore Limits --------

max number of arrays = 128

------ Shared Memory Limits --------

max number of segments = 128

To increase limits to allow up to 1024 shared memory segments and semaphores, consider adding the following to your local /etc/sysctl.conf file.

kernel.shmmni = 1024

kernel.sem = 250 256000 32 1024

Run this command to then enable support:

/sbin/sysctl -p

Note: In general, you will require root superuser access to make these changes. Consult your specific Unix/Linux documentation for the actual file location and parameters of this configuration.

Usage

To take advantage of this feature, check the following:

Shut down your FairCom Server and add the following keyword to your ctsrvr.cfg file:
COMM_PROTOCOL FSHAREMM
Restart the FairCom Server.
Execute any FairCom utility you've linked with V9.5 or later of the FairCom Server you have on the same machine as the FairCom Server process. Even if you are linked with a c-tree TCP/IP library, it will automatically detect if you are running on the same machine and try to connect via shared memory. This way you don't need multiple versions of your application and utilities.
You can monitor your connections by listing the clients from the ctadmn command-line utility on Linux, or by using the c-treeACE Monitor program from Windows (it is shown in the Comm Info column on the far right).

Usage Notes

Unix and Windows client libraries are built with shared-memory support by default.
When the FairCom Server detects a request to connect from the same machine as the client, it first attempts to connect using shared memory. If that succeeds, the connection uses the shared-memory protocol. Otherwise, the connection is made using TCP/IP.
32-bit clients can connect to 64-bit servers (and vice versa).
By default, a client application must belong to the server owner’s primary group to use shared memory. This is configurable with the SHMEM_GROUP keyword.
Shared memory uses a Unix domain (file system) socket for transferring data between the client and server. The Unix domain socket exists as a file named /tmp/ctreedbs/<servername>.logon.
The COMPATIBILITY SHMEM_PIPE option has been deprecated and no longer has any effect.
A Unix/Linux server using shared-memory communications will create a directory /tmp/ctreedbs. If this directory already exists (for example, if a different user had started the server, even from a previous run) and the server does not have write permission to this directory, startup will fail, most likely reporting a DSRV_ERR error (509, duplicate server), even if no other server is currently running.
pthread mutex shared-memory support is expected for Unix systems. However, if a client and the server are compiled with incompatible options (for example, the client uses System V semaphores but the server uses pthread mutexes) the connection attempt will fail and the FairCom Server will log one of the following messages to CTSTATUS.FCS:
If client is using System V semaphore and server is using pthread mutex:

A shared-memory connection attempt by an incompatible client failed: pthread mutex required

If server is using System V semaphore and client is using pthread mutex:

A shared-memory connection attempt by an incompatible client failed: SYSV sema required

System Tools

The Unix/Linux ipcs utility is useful for listing the shared-memory regions and semaphore sets that currently exist on a system. ipcs can also be used to remove shared-memory regions and semaphore sets. Under normal circumstances, the FairCom Server removes shared-memory regions and semaphore sets for connections that have been closed. However, if the FairCom Server process terminates abnormally, it may be necessary to manually remove the shared-memory regions and semaphore sets belonging to this process.