Keepalived, HAProxy, and the Firewall That Ate My VRRP Packets

High-Availability Gone Split-Brain: A Production Debugging Story

1. Introduction

High availability is one of those things you configure once and expect to fade into the background — until it doesn’t. In our environment, HAProxy load-balances backend traffic, and Keepalived provides the failover layer via the Virtual Router Redundancy Protocol (VRRP). A virtual IP (VIP) floats between two or more nodes; if the active node fails, a backup takes over within seconds.

VIPs claims can effect the applicaiton calls

Recently, that calm surface cracked. A routine check showed that every node in a VRRP cluster had claimed the VIPs. This is the classic split-brain scenario, and it can lead to routing loops, duplicate ARP responses, and unpredictable application behavior.

This article walks through the investigation, the two distinct root causes we found, and the security lessons we took away about VRRP and host-based firewalls.

2. Architecture Overview

Our stack looks like this:

┌─────────────────┐
     Client ───────►│  Virtual IP     │
     requests       │  10.10.3.9/10 │
                    └────────┬────────┘
                             │
            ┌────────────────┼────────────────┐
            │                │                │
     ┌──────▼──────┐  ┌──────▼──────┐  ┌──────▼──────┐
     │  HAProxy    │  │  HAProxy    │  │  HAProxy    │
     │  Keepalived │  │  Keepalived │  │  Keepalived │
     │  MASTER     │  │  BACKUP     │  │  BACKUP     │
     │  prio 101   │  │  prio 99    │  │  prio 98    │
     └─────────────┘  └─────────────┘  └─────────────┘

Keepalived elects a MASTER based on configured priority. The MASTER owns the VIP; BACKUP nodes monitor VRRP advertisements and take over if the MASTER disappears.

3. The Symptom: Three Masters, No Slaves

A simple Ansible check across the datacenter-us-west group returned the same VIPs on all three nodes:

sudo -u ansible ansible datacenter-us-west -i all.ini -b \
  -m shell -a "ip addr show br0"

Every host reported:

inet 10.10.3.9/32 scope global br0
inet 10.10.3.10/32 scope global br0

That is never correct. In VRRP, only the current MASTER should answer for the VIP.

4. Investigation

4.1 Verify the Configuration

We inspected /etc/keepalived/keepalived.conf on each node. The configuration was correct:

vrrp_instance VI_1 {
    state MASTER           # only on node 153
    interface br0
    virtual_router_id 52
    priority 101           # 99 and 98 on backups
    advert_int 1

    authentication {
        auth_type PASS
        auth_pass lb_sec
    }

    virtual_ipaddress {
        10.10.3.9
        10.10.3.10
    }

    track_script {
        check_lb_haproxy
    }
}

One MASTER, two BACKUPs, identical VRID, identical auth, identical VIPs.

4.2 Listen to the Wire

We captured VRRP traffic on br0:

sudo tcpdump -i br0 -nn host 224.0.0.18

The output showed advertisements from all three backend nodes and from a neighboring cluster:

10.10.3.12 > 224.0.0.18: VRRPv2, Advertisement, vrid 52, prio 51
10.10.3.13 > 224.0.0.18: VRRPv2, Advertisement, vrid 52, prio 101
10.10.3.14 > 224.0.0.18: VRRPv2, Advertisement, vrid 52, prio 99
10.10.3.15 > 224.0.0.18: VRRPv2, Advertisement, vrid 52, prio 98

So the network was carrying VRRP packets. But were they reaching keepalived?

4.3 Read the Logs

On each node, the story was the same:

(VI_1) Entering BACKUP STATE (init)
VRRP_Script(check_lb_haproxy) succeeded
(VI_1) Receive advertisement timeout
(VI_1) Entering MASTER STATE
(VI_1) setting VIPs.

Each node briefly started as BACKUP, then promoted itself to MASTER because it never received a valid advert from a higher-priority peer.

Key insight: tcpdump sees packets at the interface layer. The firewall lives between the interface and userspace. Packets visible to tcpdump can still be dropped before keepalived processes them.

4.4 Check the Firewall

On AlmaLinux 9, firewalld was active and br0 was in the public zone:

public (active)
  interfaces: bond0 br0 ens1f0np0 ens1f1np1
  services: ceph cockpit dhcpv6-client kafka ssh
  ports: 9100/tcp 5666/tcp 8500/tcp 80/tcp

There was no rule for VRRP (IP protocol 112). We confirmed with:

sudo iptables -L -n -v | grep -i 112
sudo nft list ruleset | grep -i 112
sudo firewall-cmd --zone=public --list-protocols

All returned empty.

5. Root Cause #1: VRID Collision

The tcpdump revealed another problem. Two independent clusters — datacenter-us-east (nodes 150–152) and datacenter-us-west (nodes 153–155)—shared the same L2 broadcast domain on br0 and both used virtual_router_id 52.

They had different VIPs:

ClusterVIPsdatacenter-us-east10.10.3.7, 10.10.3.8datacenter-us-west10.10.3.9, 10.10.3.10

When a node receives a VRRP advert for its own VRID but with a different set of VIPs, keepalived logs it as bogus and drops the packet. This creates log noise and can destabilize election logic.

Fix: make virtual_router_id a per-cluster variable in Ansible and assign unique values:

keepalived:
  virtual_router_id: 52   # datacenter-us-west

keepalived:
  virtual_router_id: 55   # datacenter-us-east

6. Root Cause #2: Firewall Dropping VRRP Protocol 112

VRRP is not TCP or UDP. It is IP protocol 112 and uses multicast destination 224.0.0.18. Because firewalld’s default public zone only allows explicitly listed services and ports, VRRP multicast was being dropped inbound.

Even though all nodes could send adverts (outbound traffic was not blocked), they could not receive each other’s adverts. The result: every node independently concluded it was the only surviving member and promoted itself to MASTER.

Fix: add a permanent rich rule to accept VRRP:

sudo firewall-cmd --permanent \
  --add-rich-rule='rule protocol value="vrrp" accept'
sudo firewall-cmd --reload
sudo systemctl restart keepalived

After applying this on all nodes, only the configured MASTER held the VIPs.

7. VRRP Protocol in Brief

For readers less familiar with VRRP, here is a quick primer:

PropertyValueProtocolIP protocol 112Multicast group224.0.0.18Virtual MAC00:00:5e:00:01:<VRID-hex>Election basisPriority (higher wins)Advert intervalConfigurable, commonly 1 secondAuthenticationSimple PASS (VRRPv2) or AH

VRRP is a Layer 2 protocol. It does not route across subnets. That means VRID uniqueness only matters within a broadcast domain, but within that domain it is absolutely critical.

8. Security Discussion

8.1 Should You Allow VRRP Through the Firewall?

Yes — but deliberately. VRRP requires multicast reception to function. Without it, HA fails. However, blindly opening all traffic is not the answer. The correct approach is:

• Allow only IP protocol 112 (VRRP).
• Restrict it to the interface(s) that participate in VRRP.
• Where possible, place VRRP traffic on a dedicated management or HA VLAN.

8.2 Authentication Matters

VRRPv2 supports a simple plaintext password. In our case, all cluster members share an 8-character password. While better than nothing, it is not encryption and can be sniffed on the local segment. For higher security:

• Use VRRPv3 with IPsec Authentication Header (AH) if your environment supports it.
• Segment HA traffic away from general workload traffic.
• Monitor for unexpected VRRP speakers on the same VRID.

8.3 Monitoring and Alerting

After the fix, we added these checks to our operational playbook:

• Count of nodes holding the VIP should be exactly 1.
• keepalived logs should not show repeated Receive advertisement timeout.
• VRRP peers should be visible in tcpdump with the expected VRID.

9. Best Practices

1. Unique VRIDs per broadcast domain — document them centrally.
2. Firewall-as-code — include VRRP allow rules in your Ansible/Puppet/Chef roles.
3. Don’t trust tcpdump alone — packets at the NIC do not guarantee delivery to the daemon.
4. Test failover after every keepalived change — stop the MASTER and verify BACKUP promotion.
5. Keep health checks lightweight and bounded — our new script uses curl --max-time 3 to avoid hung checks.

10. Conclusion

Split-brain in a VRRP cluster is usually blamed on network partitions or misconfiguration. In our case, the network was fine and the configuration was correct. The culprit was a silent firewall rule dropping VRRP protocol 112, compounded by a VRID collision that muddied the waters.

The experience reinforced an old SRE truth: visibility is not the same as deliverability. Just because you can see a packet on the wire does not mean your application can process it. Always verify the full path — from interface to kernel to userspace — especially for infrastructure protocols like VRRP.

References

• Keepalived User Guide
• RFC 3768 — Virtual Router Redundancy Protocol (VRRP)
• firewalld Rich Language

Thanks for reading. If you found this useful, share it with your SRE and infrastructure teams.