How to Build a PostgreSQL High Availability Cluster with repmgr and a Witness Node

A step-by-step architectural guide to deploying zero-data-loss automated failover for production databases.

PostgreSQL provides a robust foundation for standard enterprise database operations. Out of the box, it offers strong physical streaming replication, write-ahead log (WAL) archiving, and point-in-time recovery (PITR).

However, PostgreSQL lacks an automated, native solution for end-to-end High Availability (HA) and automatic failover. If your primary node goes down, manual intervention is required to promote a standby — leading to unwanted downtime.

To solve this, database administrators rely on ecosystem tools. One of the most mature, reliable, and trusted open-source utilities for managing cluster replication and automated failovers is repmgr (Replication Manager).

In this guide, we will walk through how to architect, configure, and validate a highly available 3-node PostgreSQL 15 cluster using a primary server, a standby clone, and a dedicated witness node to completely eliminate split-brain scenarios.

The 3-Node Quorum Architecture

To implement a reliable automated failover structure, our architecture requires three distinct nodes to achieve a consensus quorum:

1. Node01 (Primary / Master): Handles all active read/write traffic and generates the upstream WAL stream.
2. Node02 (Standby / Secondary): Stream-replicates data from the primary in hot standby mode, ready to handle traffic if Node01 fails.
3. Node03 (Witness Node): A lightweight PostgreSQL instance that does not host production data, but acts as a tie-breaking voter during failover elections to ensure an absolute majority decision.

Prerequisites

Ensure that all three servers are running matching major versions of your operating system, PostgreSQL 15, and repmgr 15.

Phase 1: Preparing Node Environments

Before configuring replication settings, we must ensure directory paths, permissions, and system user accounts are uniformly initialized across all target servers.

1. Data Directory and Ownership Initializations

Run these commands as a privileged root or sudo user on all servers to map out data paths and establish the core system account:

Bash

# Create the targeted PostgreSQL data storage path
mkdir -p /var/lib/pgsql/data

# Assign systemic user account controls
useradd postgres
chown -R postgres:postgres /usr/local/pgsql/
chown -R postgres:postgres /var/lib/pgsql/15/data/

2. Installing repmgr

With the base system prepped, install the specific repository packages matching your major PostgreSQL engine version:

Bash

# Execute using an account with sudo or root privileges
sudo yum install repmgr_15*

Phase 2: Primary Node Configuration

1. Tune postgresql.conf

Access the primary instance configuration file (/var/lib/pgsql/15/data/postgresql.conf) and update the parameters below to accommodate active streaming replication and logging:

Ini, TOML

max_wal_senders = 10
max_replication_slots = 10
wal_level = 'replica'       # Can also be set to 'logical' if required
hot_standby = on
archive_mode = on
wal_log_hints = on
shared_preload_libraries = 'repmgr'

2. Establish repmgr Database Objects

Log into your primary instance using psql to create the administrative user account and a dedicated schema tracking database:

SQL

CREATE USER repmgr;
CREATE DATABASE repmgr WITH OWNER repmgr;

3. Update HBA Access Controls (pg_hba.conf)

Ensure your nodes can talk to one another cleanly. If your enterprise policies mandate SSL communication, configure hostssl. Otherwise, adjust to standard host protocols. Add the following records to authorize your cluster network topology:

Plaintext

# --- SSL Setup (Optional but recommended) ---
# Ensure "ssl = on" is defined in postgresql.conf alongside your cert paths.
hostssl replication repmgr <Primary_Server_IP>/32 trust
hostssl replication repmgr <Secondary_Server_IP>/32 trust
hostssl repmgr      repmgr <Witness_Server_IP>/32   trust

# --- Standard Clear-Text Traffic Fallbacks ---
host repmgr repmgr <Primary_Server_IP>/32   trust
host repmgr repmgr <Secondary_Server_IP>/32 trust

Note: After saving these changes, restart the PostgreSQL instance via systemctl to apply them (sudo systemctl restart postgresql-15.service).

4. Create the Primary repmgr.conf File

Under /var/lib/pgsql/repmgr/repmgr.conf, instantiate your primary node identity. Ensure ownership of this file belongs to the postgres system user:

Ini, TOML

node_id=1
node_name=Node01
conninfo='host=<Primary_Server_IP> user=repmgr dbname=repmgr connect_timeout=2 password=<your_secure_password>'
data_directory='/var/lib/pgsql/data'
failover=automatic

# Cluster orchestration behaviors
promote_command='/usr/pgsql-15/bin/repmgr standby promote -f /var/lib/pgsql/repmgr/repmgr.conf --log-to-file'
follow_command='/usr/pgsql-15/bin/repmgr standby follow -f /var/lib/pgsql/repmgr/repmgr.conf --log-to-file --upstream-node-id=%n'

log_level=INFO
log_file='/var/log/repmgr/repmgr.log'
use_replication_slots=true
pg_bindir='/usr/pgsql-15/bin'

# Service management mappings
repmgrd_service_start_command='sudo /usr/bin/systemctl start repmgrd.service'
repmgrd_service_stop_command='sudo /usr/bin/systemctl stop repmgrd.service'
service_start_command='sudo /usr/bin/systemctl start postgresql-15.service'
service_stop_command='sudo /usr/bin/systemctl stop postgresql-15.service'
service_restart_command='sudo /usr/bin/systemctl restart postgresql-15.service'
service_reload_command='sudo /usr/bin/systemctl reload postgresql-15.service'

5. Register the Primary Node

With your configuration defined, register Node01 into your replication matrix:

Bash

/usr/pgsql-15/bin/repmgr -f /var/lib/pgsql/repmgr/repmgr.conf primary register

Verify it is successfully initialized using the cluster management utility:

Bash

/usr/pgsql-15/bin/repmgr -f /var/lib/pgsql/repmgr/repmgr.conf cluster show

Phase 3: Building & Registering the Standby Node

With our primary node up and running, we can provision our active secondary standby instance.

1. Build the Secondary repmgr.conf

Configure the settings on the standby server under /var/lib/pgsql/repmgr/repmgr.conf. Note the updated unique node_id (set to 2) and matching network connection parameters:

Ini, TOML

node_id=2
node_name=Node02
conninfo='host=<Secondary_Server_IP> user=repmgr dbname=repmgr connect_timeout=2 password=<your_secure_password>'
data_directory='/var/lib/pgsql/data'
failover=automatic

promote_command='/usr/pgsql-15/bin/repmgr standby promote -f /var/lib/pgsql/repmgr/repmgr.conf --log-to-file'
follow_command='/usr/pgsql-15/bin/repmgr standby follow -f /var/lib/pgsql/repmgr/repmgr.conf --log-to-file --upstream-node-id=%n'

log_level=INFO
log_file='/var/log/repmgr/repmgr.log'
use_replication_slots=true
pg_bindir='/usr/pgsql-15/bin'

repmgrd_service_start_command='sudo /usr/bin/systemctl start repmgrd.service'
repmgrd_service_stop_command='sudo /usr/bin/systemctl stop repmgrd.service'
service_start_command='sudo /usr/bin/systemctl start postgresql-15.service'
service_stop_command='sudo /usr/bin/systemctl stop postgresql-15.service'
service_restart_command='sudo /usr/bin/systemctl restart postgresql-15.service'
service_reload_command='sudo /usr/bin/systemctl reload postgresql-15.service'

2. Execute Standby Cloning

Before performing a full data sync, run a --dry-run operation to verify that firewall access, network routing, and authentication credentials are clear:

Bash

/usr/pgsql-15/bin/repmgr -h <Primary_Server_IP> -U repmgr -d repmgr -f /var/lib/pgsql/repmgr/repmgr.conf standby clone -c -F --dry-run

If the pre-flight check finishes cleanly without errors, execute the live command to clone the database structure over to the standby server:

Bash

/usr/pgsql-15/bin/repmgr -h <Primary_Server_IP> -U repmgr -d repmgr -f /var/lib/pgsql/repmgr/repmgr.conf standby clone -c -F

3. Register Node02

Start your secondary database instance engine via systemctl and register it with the cluster schema:

Bash

/usr/pgsql-15/bin/repmgr -f /var/lib/pgsql/repmgr/repmgr.conf standby register -F

Phase 4: Witness Node Integration

A witness node evaluates split-brain scenarios when network connectivity drops between data nodes. It ensures an absolute mathematical majority during democratic automatic failover calculations.

1. Build Witness repmgr.conf

Populate the dedicated config settings on your third isolated node (Node03):

Ini, TOML

node_id=3
node_name=Witness
conninfo='host=<Witness_Server_IP> user=repmgr dbname=repmgr connect_timeout=2 password=<your_secure_password>'
data_directory='/var/lib/pgsql/data'

2. Register the Witness Server

Run the target command directly from Node03, mapping back to your active primary node to establish cluster membership:

Bash

/usr/pgsql-15/bin/repmgr -f /var/lib/pgsql/repmgr/repmgr.conf witness register -h <Primary_Server_IP> -F

Phase 5: Verification & Inter-Node Communication

1. Cluster Mapping Checks

To confirm all three nodes are communicating correctly, run this status mapping command from any server in your environment:

Bash

/usr/pgsql-15/bin/repmgr -f /var/lib/pgsql/repmgr/repmgr.conf cluster show

2. Replication Stream Validation

Run the following SQL snippet on the primary server to review WAL differences, active connection addresses, and replication lag metrics across your targets:

SQL

SELECT 
    client_addr AS client, 
    usename AS user, 
    application_name AS name,  
    state, 
    sync_state AS mode,  
    (pg_wal_lsn_diff(pg_current_wal_lsn(), sent_lsn) / 1024)::bigint AS pending_kb,  
    (pg_wal_lsn_diff(sent_lsn, write_lsn) / 1024)::bigint AS write_kb,  
    (pg_wal_lsn_diff(write_lsn, flush_lsn) / 1024)::bigint AS flush_kb,  
    (pg_wal_lsn_diff(flush_lsn, replay_lsn) / 1024)::bigint AS replay_kb,  
    (pg_wal_lsn_diff(pg_current_wal_lsn(), replay_lsn) / 1024)::bigint AS total_lag_kb  
FROM pg_stat_replication;

To confirm the secondary node is operating in the correct passive state, execute this recovery query on the standby server:

SQL

SELECT pg_is_in_recovery(); -- Expected return value: t (true)

3. Passwordless SSH Setup

For automated management scripts and seamless cross-node operations, establish secure public key-based SSH access between all three endpoints:

Bash

# Generate localized environment RSA keys
ssh-keygen -t rsa

# Disperse public keys across cluster endpoints
ssh-copy-id postgres@<Primary_Server_IP>
ssh-copy-id postgres@<Secondary_Server_IP>
ssh-copy-id postgres@<Witness_Server_IP>

Phase 6: Configuring the repmgrd Daemon

The automatic orchestration of split-second promotions requires configuring the background monitoring daemon (repmgrd). Complete these steps on all three servers:

• Open and define a system service file under /etc/systemd/system/repmgrd.service:

Ini, TOML

[Unit]
Description=Replication Manager Daemon
Documentation=[https://repmgr.org](https://repmgr.org)
After=network.target postgresql.service
Wants=postgresql.service

[Service]
Type=forking
User=postgres
Group=postgres
ExecStart=/usr/pgsql-15/bin/repmgrd -f /var/lib/pgsql/repmgr/repmgr.conf
ExecReload=/bin/kill -HUP $MAINPID
KillMode=process
Restart=on-failure
RestartSec=5s

[Install]
WantedBy=multi-user.target

• Reload your system manager configurations, enable the daemon at boot, and start the tracking process:

Bash

sudo systemctl daemon-reload
sudo systemctl enable repmgrd
sudo systemctl start repmgrd

Phase 7: Failover Validation & Event Tracing

To monitor historical failover actions, switchovers, or cluster health trends over time, query the logging utility via:

Bash

/usr/pgsql-15/bin/repmgr -f /var/lib/pgsql/repmgr/repmgr.conf cluster event

Simulating a Graceful Switchover

When performing scheduled system patching, kernel upgrades, or hardware lifecycle replacements, you can safely trigger a controlled switchover. Always run with the --dry-run flag first to identify resource blocks:

Bash

/usr/pgsql-15/bin/repmgr -f /var/lib/pgsql/repmgr/repmgr.conf standby switchover --siblings-follow --dry-run

If the dry run finishes cleanly without identifying anomalies, remove the flag to execute the live command. The witness node will assist in promoting Node02 to primary, and will safely reconfigure Node01 to follow the new master once it recovers.

Failover scenario

To check the events for the cluster, run the below command in Master or Slave.

/usr/pgsql-15/bin/repmgr -f /var/lib/pgsql/repmgr/repmgr.conf cluster event

Now, in case the Master server goes down, the Witness server will try to Promote the standby server and it will act as a new Primary. Once the old Master server is online, it will follow the new Primary server.

Run the below command on the Standby server to make sure after switchover, Standby will promote to Primary and vice versa.

/usr/pgsql-15/bin/repmgr -f /var/lib/pgsql/repmgr/repmgr.conf standby switchover  --siblings-follow --dry-run

References:

*https://www.2ndquadrant.com/en/blog/how-to-automate-postgresql-12-replication-and-failover-with-repmgr-part-2/ [https://www.hostinger.in/tutorials/how-to-setup-passwordless-ssh/](https://www.hostinger.in/tutorials/how-to-setup-passwordless-ssh/) [https://www.devart.com/dbforge/postgresql/how-to-install-postgresql-on-linux/](https://www.devart.com/dbforge/postgresql/how-to-install-postgresql-on-linux/) [https://www.hostinger.in/tutorials/how-to-setup-passwordless-ssh/](https://www.hostinger.in/tutorials/how-to-setup-passwordless-ssh/)*