|
|
|
The FairCom Server for Unix supports shared memory connections. Shared memory communication between clients and servers residing on the same machine generally provides much better performance for locally running applications. Local shared memory connections are supported across the board including ISAM and SQL connections, and this includes JDBC and Windows ADO.NET Data providers.
Configuration
Include the following server configuration in ctsrvr.cfg to enable this support:
COMM_PROTOCOL FSHAREMM
FairCom client libraries are compiled with this feature enabled by default.
Note: The COMM_PROTOCOL (COMM_PROTOCOL, /doc/ctedge_admin/27910.htm) option specifies the protocol used for ISAM connections. By default, local SQL connections use shared memory unless the SQL_OPTION NO_SHARED_MEMORY keyword is specified. See the c-treeSQL Server Operations and Utilities Guide for more information about the communication protocol for SQL connections.
System Files, Permissions and Ownership
The FairCom shared memory communication protocol creates a file used by clients to find the shared memory identifier for its shared memory logon region, and creates a Unix domain socket as a file for initial communication between a client and server.
The FairCom Server creates the directory /tmp/ctreedbs and the file /tmp/ctreedbs/<servername>.logon. This file name is determined by the value specified with the SERVER_NAME configuration option (but see important note below). This file contains an identifier of a shared-memory region used for clients to connect. The following configuration option allows this directory to be directly specified:
SHMEM_DIRECTORY <directory_name>
IMPORTANT: SERVER_PORT applies to the TCP/IP protocol and overrides SERVER_NAME if both are used together.
If your server combines shared memory and TCP/IP usage, here are a few tips:
SERVER_PORT 7000
SERVER_NAME #7000
Then connect with a server name of #7000. The client will attempt to connect using shared memory first and if that fails it will connect with TCP/IP on port 7000.
Note: on some Linux/Unix systems, be sure to place double quotes around the connection string (for example when using the bash shell): For example: ./ctstat -vas -u admin -p ADMIN -s "#5598ocalhost"
Without the double quotes, bash treats # as a comment, so it ignores the remainder of the connection string.
FairCom DB must have sufficient read, write, create, and delete permissions with this directory. The following server keyword sets the shared memory resource permissions:
SHMEM_PERMISSIONS <permissions>
The default is 660. 666 will allow access to FairCom DB by any user account.
Note: Use caution when increasing access permissions to shared memory resources. For example, shared memory permission of 666 allows any user to attach to a shared memory segment and read or write to it. This means that any process can make a request to a FairCom Server or could read the request data of another process through such a shared memory region.
By default, a client application must belong to the server owner’s primary group to use shared memory. This is configurable with the SHMEM_GROUP keyword.
SHMEM_GROUP <group>
Possible errors indicating problems:
FSHAREMM: Could not get group ID for group <group> for shared memory
FSHAREMM: Failed to set group for LQMSG shared memory region: X
Shared Memory Keys
When more than one FairCom process is run on a Unix system, the shared memory key used by the servers might hash to the same value, causing problems connecting to the servers. This happens as the ftok() system call is used by default to generate the shared memory keys, and ftok() is not guaranteed to return unique values. Another possibility is that another unrelated process might happen to use the same shared memory key as generated by the FairCom Server.
An administrator can specify a specific shared memory key for ISAM and SQL shared memory communication protocols to ensure that keys do not match keys already in use on the system. This is specified with the following FairCom configuration options:
SHMEM_KEY_ISAM <isam_shared_memory_key>
SHMEM_KEY_SQL <sql_shared_memory_key>
The shared memory key values can be specified in either decimal or hexadecimal format. For example:
; Set shared memory key for ISAM connections to the specified decimal value:
SHMEM_KEY_ISAM 12345
; Set shared memory key for ISAM connections to the specified hexadecimal value:
SHMEM_KEY_ISAM 0xabcd
These server configuration options support specifying an environment variable, whose value is substituted for the configuration option value when the server starts up. For example, if the environment variable MY_ISAM_KEY is set to a numeric value such as 12345 or 0xabcdef before starting the server process, then the following option can be specified in the server configuration file to use this environment variable value for the SHMEM_KEY_ISAM configuration option value:
SHMEM_KEY_ISAM %MY_ISAM_KEY%
Client Configuration
From the client side, either set the global variable ctshmemdir to the directory name before connecting, or set the CTREE_SHMEM_DIRECTORY environment variable. The environment variable takes precedence over the ctshmemdir setting. This allows the directory to be dynamically overridden without having to recompile client code.
Errors with Shared Memory Protocol
The FairCom Server logs error messages to CTSTATUS.FCS when a shared-memory connection attempt fails. The message is of the form:
FSHAREMM: <error message>
Adjusting System Limits
When running the FairCom Server with more than 128 shared-memory connections, you may encounter one of the following errors:
FSHAREMM: Connect named pipe failure: 13
FSHAREMM: Connect named pipe failure: 28
Many Unix/Linux implementations have a default limit of 128 system semaphores, which are used by FairCom shared memory connections. However, this value applies system-wide among all processes.
FSHAREMM: Failed to create system semaphore: check system semaphore limits such as SEMMNI
The following error can be reported as well:
FSHAREMM: Failed to create shared memory segment: check shared memory limits such as SHMMNI
These are typically kernel configurations. The FairCom Server requires (2 + # shared memory CONNECTIONS) shared memory segments (SHMMNI) and semaphores (SEMMNI).
The ipcs command displays current limits:
#ipcs -l
------ Semaphore Limits --------
max number of arrays = 128
------ Shared Memory Limits --------
max number of segments = 128
To increase limits to allow up to 1024 shared memory segments and semaphores, consider adding the following to your local /etc/sysctl.conf file.
kernel.shmmni = 1024
kernel.sem = 250 256000 32 1024
Run this command to then enable support:
/sbin/sysctl -p
Note: In general, you will require root superuser access to make these changes. Consult your specific Unix/Linux documentation for the actual file location and parameters of this configuration.
Usage
To take advantage of this feature, check the following:
COMM_PROTOCOL FSHAREMM
Usage Notes
The COMPATIBILITY SHMEM_PIPE option has been deprecated and no longer has any effect.
If client is using System V semaphore and server is using pthread mutex:
A shared-memory connection attempt by an incompatible client failed: pthread mutex required
If server is using System V semaphore and client is using pthread mutex:
A shared-memory connection attempt by an incompatible client failed: SYSV sema required
System Tools
The Unix/Linux ipcs utility is useful for listing the shared-memory regions and semaphore sets that currently exist on a system. ipcs can also be used to remove shared-memory regions and semaphore sets. Under normal circumstances, the FairCom Server removes shared-memory regions and semaphore sets for connections that have been closed. However, if the FairCom Server process terminates abnormally, it may be necessary to manually remove the shared-memory regions and semaphore sets belonging to this process.
|
|
|
On Unix systems, the FairCom Server logs a message to CTSTATUS.FCS when it fails to create a system semaphore or shared memory segment due to a resource limit setting. This helps the server administrator understand the cause of the error. Example message:
Fri Aug 22 15:14:44 2014
- User# 00013 FSHAREMM: Failed to create system semaphore: check system semaphore limits such as SEMMNI
Fri Aug 22 15:14:44 2014
- User# 00013 NewUser: Unable to create an instance of a named pipe
|
|
|
FairCom ISAM and SQL ports are independent of each other. In general, there is a shared memory connection protocol enabled for each, in addition to TCP/IP ports. Keep in mind that's a total of four (4) connection protocols, with configuration options for each.
Possible SQL shared memory connection errors can appear such as the following. Analyze and correct these as you would for the ISAM errors previously mentioned as the same parameters should be examined.
- User# 00012 sqlshmlisten: shared memory protocol initialization failed: -1
- User# 00012 sqlshmlisten: shared memory protocol accept failed: -2
- User# 00012 sqlshmlisten: Failed to get shared memory environment: XX
Named pipe creation failed with error ERROR_PIPE_BUSY. The operation will be retried
- User# 00012 sqlshmlisten: shared memory protocol listen failed: -4
Failed permissions on the temporary directory needed.
- User# 00012 SQLSHAREMM: Failed to open /tmp/ctreedbs/CTSQL_6597 for shared memory: 13
|
|
|
When clients used the shared memory protocol on Unix systems and the client and server processes were run under different user accounts, shared memory segments could be left behind after the connections were closed. The ipcs -m listing showed shared memory segments with no processes attached. In V10.3 and later, the logic has been modified to correct this.
Note: These changes add a field to the client and server logon data structures to pass the user ID between the client and server processes, resulting in the following compatibility considerations:
An old client can connect to a new server and it will behave as it did before these changes.
A new client cannot connect to an old server using shared memory. It will receive error 133 if the server is configured to only use shared memory. It will connect with TCP/IP if the server is using both shared memory and TCP/IP. The old server will log the following message to CTSTATUS.FCS:
FSHAREMM: The client's shared memory version (2) is not compatible with the server's shared memory version (1)
At startup, the FairCom Server now logs messages to CTSTATUS.FCS to indicate the shared memory directory used for logon purposes and the shared memory protocol version that it is using:
FSHAREMM: SHMEM_DIRECTORY=/tmp/ctreedbs/
FSHAREMM: Protocol version=2
|
|
|
When more than one FairCom Server was run on a Unix system, the shared memory keys used by different servers could have the same value, which prevented connections to the servers. In addition, it was possible for unrelated applications to collide with default keys generated by FairCom DB servers.
To address this key collision, it is now possible for an administrator to specify specific shared memory keys for ISAM and SQL shared memory communication protocols ensuring the keys do not match existing keys already in use on the system.
New FairCom Server configuration options are available to directly specify a shared memory key. SQL and ISAM each require separate shared memory support
SHMEM_KEY_ISAM <isam_shared_memory_key>
SHMEM_KEY_SQL <sql_shared_memory_key>
Shared memory key values can be specified in either decimal or hexadecimal format. For example:
; Set shared memory key for ISAM connections to the specified decimal value:
SHMEM_KEY_ISAM 12345
; Set shared memory key for ISAM connections to the specified hexadecimal value:
SHMEM_KEY_ISAM 0xabcd
These server configuration options support specifying an environment variable, whose value is substituted for the configuration option value when the server starts up. For example, if the environment variable MY_ISAM_KEY is set to a numeric value such as 12345 or 0xabcdef before starting the server process, then the following option can be specified in the server configuration file to use this environment variable value for the SHMEM_KEY_ISAM configuration option value:
SHMEM_KEY_ISAM %MY_ISAM_KEY%
Compatibility Notes:
When these configuration options are not used, FairCom Server uses the old method of assigning shared memory keys so its shared memory communication protocol is compatible with old clients. FairCom Server now writes the shared memory key to the shared memory resource file and new clients know to read the shared memory key from this file. If a new client finds no shared memory key in the file, it uses the old method to assign a shared memory key so it is compatible with an old server.
The shared memory resource file is named /tmp/ctreedbs/<server_name> for ISAM, and /tmp/ctreedbs/CTSQL_<sql_port> for SQL, where /tmp/ctreedbs is the default shared memory directory. It can be changed using the SHMEM_DIRECTORY configuration option.
An old client will not be able to connect to a new server using shared memory if the server uses the SHMEM_KEY_ISAM or SHMEM_KEY_SQL configuration option to specify a shared memory key that differs from the shared memory key that the old method would generate.
|
|
|
In release V11 and later, Unix and Linux FairCom DB Servers use a Unix domain socket instead of named pipes for the initial shared memory protocol communication between the client and server. (Prior to this revision, all Unix and Linux systems except AIX used named pipes for the initial shared memory connection.)
The following shared memory protocol changes were enacted:
|
|
|
On Unix/Linux systems, a user can belong to more than one group of which one group is the primary group, and all other groups are secondary groups. When the SHMEM_PERMISSIONS option is used to only enable user and group permissions on shared memory resources, the resources created for shared memory connections (files, semaphores, shared memory regions) are assigned with the user's current primary group by default.
To address this situation, a new configuration option, SHMEM_GROUP, has been added preventing a user account that shares a secondary group with the user account under which the FairCom Server process is running failing to connect using shared memory.
This option causes the FairCom Server to assign group membership to the specified group. This option applies to the resources for both the ISAM and the SQL shared memory protocol.
As an example, consider two user accounts:
If the user1 account runs the FairCom Server with SHMEM_PERMISSIONS 660 in ctsrvr.cfg, a client program run by the user2 account will fail to connect using shared memory.
To allow the client program run by user2 to connect, add the following configuration option to ctsrvr.cfg and restart FairCom Server:
SHMEM_GROUP group2
This causes the shared memory resources to be assigned to group group2, which allows the user2 client program to connect.
|
|
|
Orphan shared memory segments and semaphores contribute to the system limit. Their presence can affect the number of user connections you will be able to achieve. There is no way of ensuring these are removed if the server process exits unexpectedly, so they must be removed manually. To remove them, follow these procedures:
ipcs -m -o
This shows all the shared memory segments on the system. Ones with no attached processes might be safe to remove.
Note: Be careful not to delete the shared memory segments created by other processes. Deleting a shared memory segment that is in use does not cause an immediate problem for c-tree. This is similar to deleting a file that is open: any process using it can keep using it, but no one can start using it.
For Solaris 5.10, see List Interprocess Semaphores on Solaris 5.10 below.
Solaris does not provide a way to see which interprocess semaphores belong to a specific process. To get an idea of whether a semaphore might be in use, use ipcs -s -t to see the time of last use. You may be able to infer which semaphores belong to c-tree by the time of last use (especially if you can shut down c-tree cleanly and observe the change in semaphores).
Note: Deleting an interprocess semaphore if it is actively in use by c-tree causes the connection to immediately fail. Deleting an interprocess semaphore that is in use by another process may cause that process to fail.
Once you have removed the items shown above, you can start the server and connect to it.
Raising the Limits on Solaris 5.10 and Later
The following parameters are useful for managing resources:
To temporarily raise the limits set for these parameters, run the following as root, where SHELL_PID is the PID of the shell that will be starting the c-tree server process:
prctl -n project.max-shm-ids -r -v 1024 -i project default
prctl -n project.max-sem-ids -r -v 1024 -i project default
prctl -n process.max-sem-nsems -r -v 1024 SHELL_PID
You can then start FairCom Server from the shell with PID = $SHELL_PID and connect users.
The above settings will be reset to the default when the machine is rebooted. (Also, remember that max-sem-nsems is only increased for that shell process and its children).
To make these changes permanent so they are effective after every reboot, execute the following command:
/usr/sbin/projmod -sK "project.max-shm-ids=(privileged,1024,deny)" default
After changing the shared memory parameters, you may need to delete orphaned shared memory segments as described above.
List Interprocess Semaphores on Solaris 5.10
Solaris 5.10 (and later) has a different method for capturing the current number of semaphores:
prctl -n project.max-shm-ids $$
Example output:
/export/home/fctech$ prctl -n project.max-shm-ids $$
process: 3636: -tcsh
NAME PRIVILEGE VALUE FLAG ACTION RECIPIENT
project.max-shm-ids
privileged 128 - deny -
system 16.8M max deny
prctl -n process.max-sem-nsems $$
Example output:
/export/home/fctech$ prctl -n process.max-sem-nsems $$
process: 3636: -tcsh
NAME PRIVILEGE VALUE FLAG ACTION RECIPIENT
process.max-sem-nsems
privileged 512 - deny -
system 32.8K max deny -
|
|
|
Problem: Client cannot connect via shared memory. CTSTATUS message: FSHAREMM: Read of socket failed: 11
Solution: The client may not have sufficient permissions to connect. Check SHMEM_PERMISSIONS and SHMEM_GROUP server configuration. May need to add the user to the system level group identified by SHMEM_GROUP.
On Unix: Client side shared memory debugging may be enabled by setting the environment variable CTCLIENT_DEBUG_LOG=<filename> for the client process(s), which will write additional messages to <filename>.
