When Nested Proxmox VE VMs Could ARP but Not Ping Across Nodes: Troubleshooting a VMware vSwitch Uplink Issue

Introduction

I recently investigated a strange networking issue in a nested virtualization lab:

• Proxmox VE nodes were running as VMs on VMware ESXi
• Guest VMs on the same Proxmox node could communicate
• Guest VMs on different Proxmox nodes in the same VLAN could not communicate properly

At first glance, it looked like a Proxmox bridge problem. However, after a detailed investigation with packet captures, bridge FDB checks, and coordination with Proxmox technical support, the issue turned out to be related to the VMware vSwitch / external network path, not Proxmox VE itself.

This article summarizes the symptoms, investigation steps, root cause, and workaround.

Lab Overview

The environment was roughly as follows:

• VMware ESXi 7.0 Update 2
• Standard vSwitch (VSS), not vSphere Distributed Switch (VDS)
• Proxmox VE nodes running as nested VMs on ESXi
• Guest VMs attached to the same VLAN (VLAN 200)
• Proxmox bridges configured on each Proxmox node
• Multiple uplinks configured on the ESXi vSwitch

Example communication under test:

• test-vm01 on proxmox01
• test-vm03 on proxmox02

Initial Symptoms

The main issue was:

• VMs on the same Proxmox node could communicate normally
• VMs on different Proxmox nodes could not ping each other
• ARP seemed to work, but ICMP replies did not arrive correctly

For example:

• test-vm03 → test-vm01: failed across nodes
• test-vm01 → default gateway: unstable or failed in some cases
• In some tests, ping replies showed **DUP!** or duplicate responses

That duplicate behavior was an important clue.

First Finding: The Linux Bridge Learned the Wrong Port

During troubleshooting, Proxmox support asked for:

• tcpdump captures on bridge / tap / guest interfaces
• bridge fdb show
• additional captures without filters

One key finding was that the Linux bridge FDB on the destination Proxmox node had learned the guest MAC address on the physical VLAN uplink interface, instead of the VM tap interface.

A typical wrong entry looked like this:

bridge fdb show vmbr0

And the problematic result was effectively:

bc:24:11:xx:xx:xx dev ens192.200 vlan 1 master vmbr0

Instead of something like:

bc:24:11:xx:xx:xx dev tap101i0 master vmbr0

This meant:

• packets destined for the guest VM reached the Proxmox node
• but the bridge believed the MAC lived behind the physical uplink
• so the bridge forwarded traffic out incorrectly
• as a result, the guest never received the packets

Why Was the Bridge Mis-Learning the MAC?

According to Proxmox support, the likely explanation was:

• ARP replies from the guest VM were somehow being seen again from outside
• the bridge then learned that MAC on the external-facing interface
• this caused wrong forwarding behavior

Their analysis suggested that the same MAC address appeared both:

• from the VM tap interface
• and again from the physical network side

That strongly implied that ARP replies were being reflected, duplicated, or mirrored somewhere outside the Proxmox host.

Things We Tried During Troubleshooting

1. Capturing Traffic on Proxmox VE

We captured packets on:

• vmbr0
• the VM tap interface
• the guest interface inside the VM

Examples:

tcpdump -envi vmbr0 -w capture-proxmox01-vmbr0.pcap
tcpdump -envi tap101i0 -w capture-proxmox01-tap101i0.pcap
tcpdump -envi ens18 -w capture-test-vm01.pcap

These captures showed:

• ARP requests and some ARP replies were visible on the bridge
• but expected packets were not always visible inside the guest
• the traffic flow was inconsistent with normal local switching behavior

2. Flushing the FDB

We also tried clearing the bridge forwarding database:

bridge fdb flush dev vmbr0

However, after rebooting the Proxmox node, the same wrong FDB learning came back quickly. So the issue was not just stale state; something in the environment was actively causing re-learning.

3. Disabling MAC Learning on the Bridge

Based on Proxmox support guidance, we also tested:

bridge-disable-mac-learning 1

on the Proxmox bridge configuration.

Even then, the issue persisted. This suggested that the problem was not a simple Proxmox bridge bug, but something more fundamental in how traffic was arriving from outside.

4. Checking Physical Switch Configuration

We reviewed the test environment’s physical switch configuration and looked for:

• forwarding loops
• port mirroring
• suspicious trunk configuration issues

At least from the configuration files we checked, we could not clearly identify:

• explicit port mirroring
• obvious forwarding-loop settings

But Proxmox support continued to suspect duplication or reflection somewhere outside the Proxmox nodes.

5. Capturing Traffic on ESXi vSwitch with pktcap-uw

Since the environment was nested on ESXi, we also investigated the VMware side.

We used pktcap-uw to capture packets from:

• the switch ports connected to Proxmox VMs
• the physical uplinks

Examples:

pktcap-uw --switchport 100664366 --capture VnicRx -o /tmp/vSwitch_from_proxmox01.pcap
pktcap-uw --switchport 100664368 --capture VnicRx -o /tmp/vSwitch_from_proxmox02.pcap
pktcap-uw --uplink vmnic4 -o /tmp/vSwitch_vmnic4_inbound.pcap
pktcap-uw --uplink vmnic6 -o /tmp/vSwitch_vmnic6_inbound.pcap

This investigation showed another important clue:

• ARP replies were visible on one uplink capture
• but not the other
• and some successful pings showed duplicate replies

That strongly suggested packet duplication somewhere along one uplink path.

Another Important Symptom: Duplicate Ping Replies

A very revealing observation was this:

• when a guest VM pinged the other Proxmox node, replies could show DUP!
• when a guest VM pinged its own Proxmox node, no duplication appeared
• when pinging the default gateway, duplication did not always appear

This pattern suggested that the problem depended on the traffic path between nodes, not just guest configuration.

Proxmox support pointed out that duplicate ICMP replies should not happen in a normal network. That was one of the strongest signs that packets were being duplicated by something outside Proxmox VE.

What Finally Solved It

The breakthrough came when we changed the ESXi standard vSwitch uplink configuration.

Originally, the vSwitch had two uplinks. We modified it so that it used only one uplink: vmnic6.

After that change:

• test-vm01 ↔ test-vm03 communication succeeded
• duplicate ping replies disappeared
• communication to the default gateway worked normally
• the cross-node guest connectivity issue was resolved

In other words, reducing the vSwitch uplinks from two to one fixed the issue.

Root Cause

Based on the troubleshooting results, the most likely root cause was:

Packet duplication / reflection somewhere in the VMware vSwitch or external network path when two uplinks were active

That duplication likely caused:

1. ARP replies from the guest VM to be seen again from the external-facing side
2. the Proxmox Linux bridge to learn the guest MAC on the wrong interface
3. traffic for that guest to be forwarded incorrectly
4. cross-node VM communication to fail

So while the symptom appeared inside Proxmox, the actual cause was most likely outside the Proxmox VE cluster.

Was Proxmox VE Misconfigured?

After the investigation with technical support, the conclusion was:

• there was no clear evidence of misconfiguration in Proxmox VE itself
• the issue was more consistent with:
• VMware vSwitch behavior
• uplink/network path duplication
• or external switching behavior

This was important because it changed the direction of the investigation completely.

Additional Notes About VDS and VMware Fling

During the investigation, Proxmox support referenced a Proxmox forum post about a similar issue in a VMware environment and mentioned a VMware Fling related to MAC learning.

However, in our case:

• the environment used VSS, not VDS
• the referenced workaround appeared to be VDS-oriented
• therefore it did not directly apply to our lab

Still, it may be worth testing in future environments that use vSphere Distributed Switch.

Lessons Learned

Here are the main takeaways from this incident:

1. ARP working does not mean L2 forwarding is healthy

A VM can respond to ARP, and yet traffic can still fail because the bridge learns the MAC on the wrong port.

2. Duplicate ICMP replies are a major warning sign

If you see DUP! in ping output, do not ignore it. It may indicate packet duplication, reflection, or loop-like behavior.

3. In nested environments, the problem may be outside the guest hypervisor

Even if the symptom appears on Proxmox, the real cause may be in:

• ESXi vSwitch
• uplink design
• physical switch behavior
• mirrored or duplicated traffic in the upstream network

4. Uplink design matters

When multiple uplinks are active, unexpected behavior can occur even without obvious configuration errors.

Practical Workaround

If you encounter similar behavior in a nested Proxmox-on-VMware lab, one practical test is:

• reduce the ESXi vSwitch uplinks from two to one
• retest guest-to-guest communication across Proxmox nodes
• check whether duplicate ping replies disappear

If they do, the problem is likely not inside Proxmox VE.

Conclusion

This issue started as what looked like a Proxmox Linux bridge problem:

• ARP was partially working
• ICMP was failing
• MAC learning looked wrong
• guest traffic never reached the destination VM correctly

But after detailed packet analysis and repeated testing, the true pattern became clear:

The cross-node VM communication issue was caused by packet duplication / reflection related to the VMware vSwitch or upstream network path when two uplinks were active.

By switching the ESXi vSwitch to a single uplink, communication recovered and duplicate replies disappeared.

For anyone building nested Proxmox labs on VMware, this is a useful troubleshooting pattern to remember.

Appendix: Commands Used

Proxmox bridge / guest capture

tcpdump -envi vmbr0 -w capture-proxmox01-vmbr0.pcap
tcpdump -envi tap101i0 -w capture-proxmox01-tap101i0.pcap
tcpdump -envi ens18 -w capture-test-vm01.pcap
bridge fdb show vmbr0
bridge fdb flush dev vmbr0

ESXi packet capture

pktcap-uw --switchport 100664366 --capture VnicRx -o /tmp/vSwitch_from_proxmox01.pcap
pktcap-uw --switchport 100664368 --capture VnicRx -o /tmp/vSwitch_from_proxmox02.pcap
pktcap-uw --uplink vmnic4 -o /tmp/vSwitch_vmnic4_inbound.pcap
pktcap-uw --uplink vmnic6 -o /tmp/vSwitch_vmnic6_inbound.pcap