Linux Pacemaker

A Cluster is a collection of nodes and resources (usually services) managed as a set for high availability.

Pacemaker is a high-availability cluster resource manager based on the Open Cluster Framework (OCF). Pacemaker provides detection of and recovery from node-level and service-level failures. Faulty nodes can be fenced ensuring data integrity. Pacemaker configurations define clusters, nodes, resources, alerts, and STONITH fencing agents. 

Pacemaker cluster management is included with many major Linux distributions. It is a cluster resource manager and executes as a daemon called CRMd on each node. All cluster changes are routed through it, such as node instantiations, moves, starts, stops, and information queries. Pacemaker can script and cluster any resource, such as FairCom DB. 

Pacemaker uses quorum voting (votequorum service) to determine when a cluster node should be fenced or isolated to prevent split-brain phenomena. More than 50% of nodes must agree on cluster operations. If more than 50% of the nodes in a cluster are offline, clustered services are stopped. Pacemaker utilizes Corosync as its high-availability framework and a specific Corosync option provides for two node clusters where only one node is used as a quorum.

Corosync is a distributed messaging and quorum platform for cluster-aware components. Cluster communications go through the Corosync daemon utilizing an in-memory database. It manages cluster membership and messaging communications as well as quorum rules, and cluster state transfer between nodes. Corosync uses the kronosnet library.

A Node is an RHEL host participating in the cluster. Each cluster node runs a local resource manager daemon (LRMd) which is the interface between CRMd and local cluster resources on the node. 

One node in the cluster is the Designated Coordinator (DC) which stores and distributes cluster state to all cluster nodes. This cluster state is maintained in the Cluster Information Base (CIB).

A cluster resource is an application or data managed by the cluster — for example, a database instance running in an RHEL cluster can be configured to be a cluster resource. In this definition, FairCom DB is defined as a resource that requires cluster management to run its services in a highly available configuration.

Resource agent is an abstraction that allows Pacemaker to manage services it knows nothing about. This abstraction takes the form of an executable script that contains logic for starting, stopping, and checking the health of a defined resource such as a database, or other service provided by the cluster.

Fencing agents regularly check the status of a resource and, if Pacemaker determines that a resource has failed, one or more agents ensure the resource stays down. Fencing agents implement Shoot The Other Node In The Head (STONITH) which is usually targeted to the hardware layer ensuring the complete removal of a node from the cluster.

Fencing is an important component of any successful clustering as it ensures that a failed node is unable to run or provide any services at all. This prevents scenarios such as split brain which, in the case of a database, allows data to be written to both servers in the cluster. This incorrectly puts some data on one server and some data on the other causing the breaking of data integrity, errors, and making it very difficult to recover correct data.

Groups are created to manage multiple cluster resources as a unit.

The files in this section are required to configure the server and its embedded replication agent to run in a pacemaker cluster.

It is important to configure replication on both nodes correctly. Once a failover node has recovered, it now becomes the secondary node and the replication direction is now reversed between the nodes. If replication is not correctly configured, the secondary node will not be in sync after recovery.

; c-tree Replication Agent Configuration File

;file filter must specify the names of files to include/exclude for replication and the purposes, including sync_commit
file_filter             <../config/replSync.xml

;replication agent unique id; must be same on both systems
unique_id               agent1

check_update            yes

replicate_data_definitions      yes

;enable synchronous commit processing for replication agent
syncagent               yes

parallel_apply          yes
num_analyzer_threads    1
num_apply_threads       4
sync_log_writes         yes

; Source server connection info
source_authfile         ../config/replication/source_auth.set
source_server           FAIRCOMS@node1
source_nodeid           10.0.0.1

; Target server connection info
target_authfile         ../config/replication/target_auth.set
target_server           FAIRCOMS@localhost
target_nodeid           10.0.0.2

socket_timeout          5

lock_retry_count        10
lock_retry_sleep        1000

; Read 8 KB batches from source server
batch_size              8192

; Use a 1-second timeout when reading from source server
read_timeout_ms         1000

exception_mode          transaction

; The master server will remember the replication agent's log position
; even when the replication agent disconnects.
remember_log_pos        yes

; Log file name 
log_file_name  ../data/ctreplagent.log

; Source server connection info
source_server           FAIRCOMS@node2
source_nodeid           10.0.0.2

; Target server connection info
target_server           FAIRCOMS@localhost
target_nodeid           10.0.0.1

<?xml version="1.0" encoding="us-ascii"?>
<replfilefilter version="1" persistent="y">
        <file status="include">mark.dat</file>
        <file status="include">admin_test.dat</file>
        <purpose>create_file</purpose>
        <purpose>read_log</purpose>
        <purpose>open_file</purpose>
        <purpose>sync_commit</purpose>
</replfilefilter>

The server configuration file (ctsrvr.cfg) requires additional options specific to participating in a Pacemaker cluster.

;Required keywords for pacemaker-based replication
CHECK_CLUSTER_ROLE              YES
REPL_NODEID                     10.0.0.1

;Optional keywords related to pacemaker replication
SECONDARY_STARTUP_TIMEOUT_SEC 	20
SYNC_COMMIT_TIMEOUT             10
MAX_REPL_LOGS                   100
REPLICATE                       *.dat

The server sets a global cluster attribute name promotestate to indicate which is the current primary server node and whether it is running in synchronous or asynchronous replication mode.

The server runs the getsyncstate shell script to read the cluster attribute and the setsyncstate shell script to set its value.

Ensure that these scripts have been put into a directory that is included in the server’s PATH environment variable and is readable and executable by the user ID under which the server process is run.

The FairCom server runs setsyncstate automatically when configured to work under Pacemaker. When the primary server calls the setsyncstate script it sets its replication mode to synchronous or asynchronous using the following options:

If the FairCom server sets the replication mode to asynchronous the cluster is at risk in case of failure.

To notify the administrator of changes to the replication mode, edit the setsyncstate script to use a specified notification method. The commented lines (#) in the script show where to insert notification methods. Use the echo command to send notifications to a file in the tmp directory. To specify another notification method, such as sending an email or running a Python script, use any command that can be run from a shell.

 setsyncstate script:

#!/bin/bash

case $1 in
"y")
  crm_attribute -G --name promotestate --update "{ \"node\":\"$(hostname)\", \"sync\":\"y\" }"
  rc=$?
	
  # You can add a notification that the server is entering synchronous mode here.
  # For example:
  # echo "`date` Node $(hostname) entered synchronous mode." >> /tmp/faircomdbSyncState.log
	
  echo rc=$rc
  exit $rc
  ;;
"n")
  if [ "$2" == "" ]; then
    crm_attribute -G --name promotestate --update "{ \"node\":\"$(hostname)\", \"sync\":\"n\" }"
  else
    crm_attribute -G --name promotestate --update "{ \"node\":\"$(hostname)\", \"sync\":\"n\", \"logposition\":\"$2\" }"
  fi
  rc=$?

  # You can add a notification that the server is entering asynchronous mode here.
  # For example:
  # echo "`date` Node $(hostname) entered asynchronous mode." >> /tmp/faircomdbSyncState.log

  echo rc=$rc
  exit $rc
  ;;
"c")
  # option c checks that the script can be executed
  echo rc=0
  exit 0
  ;;
*)
  echo error: must specify y, n, or c
  rc=1
  echo rc=$rc
  exit $rc
  ;;
esac

This function in the FairCom DB API enables background broadcast detection.

NINT ctDECL ctSetClientLibraryOption(NINT option, pVOID value);

ctSetClientLibraryOption(ctCLIOPT_BROADCAST_READ, "YES");

This API call checks your current failover state.

TEXT ctDECL ctGetFailOverState();

if (ctGetFailOverState() == 1)

This API call can reset failoverstate upon successful failover.

ctResetFailOverState()

sudo pcs resource describe ocf:heartbeat:faircomdb

ocf:heartbeat:faircomdb - FairCo DB resource agent

Resource agent script for FairComDB Database Server

Resource options:
 binary: Full path to the FairComDB binary
  config: Full path to a FairComDB configuration directory or configuration file
  pid: File to read running process PID
  user: User name or id FairComDB will run under
  group: Group name or id FairComDB will run under
  faircomdb_adminpass (required): FairComDB ADMIN password
  faircomdb_monitor_user: FairComDB user name to connect for monitoring
  faircomdb_monitor_userpass: FairComDB password to connect for monitoring
  faircomdb_servername (required): FairComDB server name to connect for
                                   monitoringi
  faircomdb_host: FairComDB host name to connect for monitoring
  faircomdb_working_directory (required): FairComDB server working directory.
                                          This must be writable for FairComDB to
                                          start and operate
  faircomdb_data_directory (required): FairComDB server data directory. This
                                       must be writable for FairComDB to start
                                       and operate
 faircomdb_heartbeat_tcpip: FairComDB heartbeat option that forces TCPIP 
connections when enabled. Setting false  will use a shared
memory heartbeat that will fallback to TCPIP
  ctsmon: Path to FairComDB ctsmon monitor utility. Assume located in $PATH if
          not provided
  repadm: Path to FairComDB Replication Agent Administrator utility. Assume
          located in $PATH if not provided

Default operations:
  start: interval=0s timeout=30s
  stop: interval=0s timeout=30s
  monitor: interval=10s timeout=20s
  notify: interval=0s timeout=30s
  promote: interval=0s timeout=20s
  demote: interval=0s timeout=20s

pcs cluster destroy --all

pcs cluster setup faircom cluster1.company.com cluster2.company.com

pcs cluster start --all

pcs cluster cib clcfg

Options should be configured for your specific system.

pcs -f clcfg resource create faircomdb ocf:heartbeat:faircomdb binary=/opt/faircomdb/server/faircom config=/opt/faircomdb/server/ctsrvr.cfg ctsmon=/opt/faircomdb/tools/ctsmon ctstop=/opt/faircomdb/tools/ctstop faircomdb_adminpass=ADMIN faircomdb_servername=FAIRCOMS faircomdb_working_directory=/opt/faircomdb/server faircomdb_data_directory=/opt/faircomdb/data user=fctech repadm=/opt/faircomdb/tools/repadm pid=/opt/faircomdb/server/faircomdb.pid

pcs -f clcfg resource promotable faircomdb master-max=1 master-node-max=1 clone-max=2 clone-node-max=1 notify=true

pcs -f clcfg resource op add faircomdb monitor timeout=20 interval=11 role=Master

pcs -f clcfg resource update faircomdb op promote timeout=120s
pcs -f clcfg resource update faircomdb op start timeout=120s
pcs -f clcfg resource update faircomdb op stop timeout=120s

pcs -f clcfg resource create VIP ocf:heartbeat:IPaddr2 ip=YOUR_IP cidr_netmask=22 op monitor interval=30

pcs -f clcfg constraint colocation add VIP with master faircomdb-clone score=INFINITY

pcs -f clcfg constraint order faircomdb-clone then start VIP

This must be customized for each system.

pcs -f clcfg stonith create vmfence fence_vmware_soap pcmk_host_map="cluster1.company.com:RHEL8-Node3;cluster2.company.com:RHEL8-Node4" ipaddr=vmware-host.company.com ssl_insecure=1 login=root passwd=mysecretpassword
pcs -f clcfg property set stonith-enabled=true

pcs -f clcfg property set no-quorum-policy=ignore

pcs -f clcfg resource defaults update resource-stickiness=INFINITY

pcs -f clcfg resource defaults update migration-threshold=1

pcs -f clcfg resource defaults update failure-timeout=60

pcs cluster cib-push clcfg

Pacemaker provides examples in /etc/corosync/corosync.conf.

totem {
    version: 2
    cluster_name: my_cluster 
    transport: knet
    crypto_cipher: aes256
    crypto_hash: sha256
}

nodelist {
    node {
        ring0_addr: node1.company.com 
        name: node1.company.com 
        nodeid: 1
    }

    node {
        ring0_addr: node2.company.com 
        name: node2.company.com 
        nodeid: 2
    }
}

quorum {
    provider: corosync_votequorum
    two_node: 1 
}

logging {
    to_logfile: yes
    logfile: /var/log/cluster/corosync.log
    to_syslog: yes
    timestamp: on
}

Example scripts provided by Pacemaker can be found in /usr/share/pacemaker/alerts.

pcs alert create  
   id=failover
   path=/usr/local/bin/failover.shpath=/var/lib/pacemaker/failover.sh
   description=”Script to run at failover"

./failover.sh
#!/bin/bash


##USEFUL DEBUG##
#printf "$CRM_alert_node $CRM_alert_rsc $CRM_alert_task $CRM_alert_kind $CRM_alert_desc $CRM_alert_attribute_name $CRM_alert_attribute_value" | ncat -u 255.255.255.255 5595
#printf "FairCom FailOver Event" | ncat -u 255.255.255.255 5595
##END OF USEFUL DEBUG##


if [[ $CRM_alert_rsc == "VIP" && $CRM_alert_task == "monitor" && $CRM_alert_desc == "Cancelled" ]]; then
  printf "FairCom FailOver Event" | ncat -u 255.255.255.255 5595
fi
if [[ $CRM_alert_task == "node" && $CRM_alert_kind == "lost" ]]; then
  printf "FairCom FailOver Event" | ncat -u 255.255.255.255 5595
fi

When the server sets its role to the primary or secondary server after receiving notification of its role from Pacemaker, it writes its current role to the file serverrole.ini in the server’s data directory (which is set using the LOCAL_DIRECTORY server configuration option). This file contains the value 1 for the primary server and 2 for the secondary server. The server uses this file when it starts up to know its role the previous time it ran.