Photo by Albert Stoynov on Unsplash

When Your Firewall Isn’t the Problem: Chasing a Ghost Through a Virtualized Network

A home SOC lab investigation that started with a broken internet connection and ended with a lesson I won’t forget.

I’ll be honest — when my Kali box behind OPNsense started losing internet access every few minutes, my first instinct was to blame the firewall. It’s always the firewall, right? Turns out, I was wrong. And the actual culprit was hiding in plain sight at Layer 2, quietly causing chaos while I was busy poking at packet filters.

Here’s the full breakdown of how I tracked it down.

The Lab Setup

Before diving into the investigation, a quick picture of what I’m working with:

Hypervisor Host: Arch Linux running KVM/QEMU with libvirt and Virt-Manager.

Network Architecture:

• WAN — Libvirt NAT network (virbr0), subnet 192.168.122.0/24
• SOC-LAN — Libvirt internal network (virbr-lan), subnet 10.10.10.0/24
• DMZ — Libvirt internal network (virbr-dmz), subnet 10.10.20.0/24

Virtual Machines:

• OPNsense firewall (vtnet0 = WAN, vtnet1 = LAN, vtnet2 = DMZ)
• Kali Linux on SOC-LAN at 10.10.10.30
• Windows Domain Controller, security monitoring stack, and a few other SOC testing systems

Nothing exotic. A fairly standard home lab topology — which made the problem all the more confusing.

The Problem

Clients on the SOC-LAN network were intermittently losing internet access. Not all at once, not in any obvious pattern — just… gone, after a few minutes of working fine.

What made it weird:

• Clients could still reach the OPNsense LAN interface without issues
• OPNsense itself had no problem reaching the internet
• Running a service reload on OPNsense would temporarily fix things
• Then the failures would come back

The usual suspects showed up fast: firewall state corruption, NAT failure, routing problem, VirtIO driver bugs, gateway monitoring acting up. I started working through them one by one.

Phase 1: Firewall and State Table

My first stop was PF, OPNsense’s packet filter. I checked state table utilization, validated NAT rules, and even disabled PF entirely to remove it from the equation.

pfctl -si
pfctl -ss
pfctl -sn
pfctl -d

State table? Normal. NAT rules? Present and functional. Firewall disabled? Problem persisted.

PF was cleared. Moving on.

Phase 2: Routing and IP Forwarding

Next, I validated the routing table and confirmed IP forwarding was enabled.

netstat -rn
sysctl net.inet.ip.forwarding
traceroute

Default gateway was reachable, routing table looked healthy, forwarding was on. Nothing broken here.

Routing was eliminated. Still no culprit.

Phase 3: ARP and Layer 2 — Things Get Interesting

This is where things started to get strange.

I ran ARP table inspections and threw up some tcpdump captures on vtnet1 and virbr-lan:

arp -a
tcpdump -ni vtnet1
tcpdump -ni virbr-lan

I was seeing a lot of ARP traffic. Specifically, repeated requests for 10.10.10.10 — and more tellingly, inconsistent MAC address associations for the gateway IP 10.10.10.1.

That was the signal. ARP entries for the gateway were changing. Something was fighting over that IP.

Phase 4: VirtIO vs E1000

Before going further down the ARP rabbit hole, I swapped the virtual NIC drivers — replaced VirtIO with E1000 across the affected interfaces — and verified bridge connectivity and tap interface attachment.

The problem didn’t care what NIC driver I was using. Issue persisted.

Virtual NIC drivers ruled out.

Phase 5: Packet Captures Across All Layers

At this point I set up simultaneous captures on three interfaces — LAN (vtnet1), WAN (vtnet0), and the Linux bridge (virbr-lan) — to watch traffic flow in real time.

The WAN side was perfectly healthy. OPNsense was sending and receiving internet traffic without any issues. NAT was working. The internet path was intact.

The failure was entirely on the LAN side.

The Root Cause

With that confirmation, I went back to the ARP data and did something I should have done earlier: I compared the MAC address of the gateway IP against every interface claiming it.

ARP table showed:

10.10.10.1 -> 52:54:00:9e:72:3e

On the Arch Linux host:

ip addr show virbr-lan

Output: virbr-lan had IP 10.10.10.1/24 with MAC 52:54:00:9e:72:3e.

OPNsense LAN interface: also configured as 10.10.10.1/24.

There it was.

Two devices were claiming 10.10.10.1 simultaneously — OPNsense and the Arch Linux bridge interface. Client systems were getting confused, alternating between two different MAC addresses for the same gateway IP. Some packets were making it to OPNsense and routing properly. Others were landing on the Linux host bridge, which had no idea what to do with them, generating ICMP unreachable responses and silently dropping traffic.

The service reloads were temporarily fixing it because they forced a fresh ARP advertisement from OPNsense — but within a few minutes, the Linux bridge would reassert itself and the conflict would resume.

The Fix

Simple once you know what’s wrong:

Before:

OPNsense LAN: 10.10.10.1/24

After:

OPNsense LAN: 10.10.10.200/24

Moved OPNsense off 10.10.10.1 so it no longer competed with the libvirt bridge. ARP stabilized immediately, gateway resolution became consistent, and internet connectivity has been solid ever since.

What I Took Away From This

A few things stuck with me after wrapping this up:

Always validate Layer 2 before assuming higher-layer failures. It’s easy to jump to firewall rules or routing tables, but ARP conflicts can produce symptoms that look exactly like NAT or state table issues.

Duplicate IPs in virtualized environments are easy to miss. When you spin up a libvirt internal network, the bridge interface on the host gets an IP. If you configure your VM’s gateway to match that IP without thinking about it, you’ve created a conflict that will surface in frustrating and intermittent ways.

Packet captures on multiple interfaces simultaneously are invaluable. The moment I saw WAN traffic flowing cleanly while LAN traffic was failing, the investigation narrowed dramatically. There’s no substitute for actual evidence.

MAC address correlation closes the loop. Checking which physical interface owns a given IP — not just what the routing table says — is often the fastest way to find ARP-related problems.

Final Thoughts

This was a good reminder that in layered systems, the most confusing failures are often the simplest ones wearing a disguise. I spent time looking at firewalls, NAT tables, routing daemons, and virtual NIC drivers — all legitimate suspects — before the answer turned out to be a misconfigured IP address that I had set myself.

The fix took about thirty seconds. The investigation took considerably longer.

If you’re running a virtualized firewall setup at home or in a lab, take five minutes to check that your hypervisor bridge interfaces aren’t sharing IP space with your VM’s gateway interfaces. It’s an easy thing to overlook, and an annoying thing to debug from scratch.

Built on an Arch Linux / KVM / OPNsense home SOC lab. All traffic captures and diagnostic output were collected from the live environment during the incident.