Troubleshooting K3s Flannel VXLAN Cross-Node Communication Failure

TL;DR

• Problem: K3s cluster with Flannel VXLAN backend failed cross-node Pod communication (100% packet loss)
• Evidence: Source VM sent VXLAN packets successfully, but destination VM received zero packets (both at physical interface and tunnel interface)
• Root Cause: External network/hypervisor layer dropping VXLAN encapsulated traffic between VMs
• Solution: Migrated to Flannel host-gw mode, which bypasses VXLAN encapsulation and performs better on L2-connected nodes
• Recommendation: Use host-gw as the default for single-L2-cluster deployments; simpler, faster, and more compatible

1. Problem Description

In a production K3s cluster deployment, we encountered a critical networking issue where cross-node Pod communication failed completely when using Flannel’s default VXLAN backend. The symptoms were:

• Pods on the same node could communicate normally
• Pods on different nodes could not reach each other (100% packet loss)
• DNS resolution failed (CoreDNS timeout)
• Istio Ingress Gateway pods remained in 0/1 Running state due to inability to reach istiod
• Switching Flannel to host-gw mode immediately resolved the issue

This article documents the systematic troubleshooting process that isolated the root cause to external network restrictions beyond our control.

2. Environment Details

• Kubernetes: K3s v1.34.3+k3s1
• CNI: Flannel (VXLAN backend, port 8472)
• OS: Ubuntu 24.04.3 LTS
• Kernel: 6.8.0–100-generic
• Network: 172.16.36.0/24 (Layer 2)
• Pod CIDR: 10.42.0.0/16
• Hypervisor: aCloud (Sangfor HCI Platform)
• Virtual NIC: virtio_net

3. Troubleshooting Process

Phase 1: Basic Connectivity Verification

We first verified that the underlying infrastructure was sound:

# Node-to-node IP connectivity was normal
ping 172.16.36.9  # Success
# VXLAN port was reachable at transport layer
nc -vuz 172.16.36.9 8472  # Success
# Traceroute to Pod IP failed
traceroute 10.42.2.6  # All hops timeout

Finding: Layer 3 connectivity between nodes was fine, but Pod network (Layer 2 overlay) was broken.

Phase 2: Packet Capture Analysis

We performed simultaneous packet captures on both source and destination nodes during a ping test:

On Source Node (k3ssvr01–172.16.36.7):

sudo tcpdump -i ens18 -n -e -v udp port 8472 -w /tmp/vxlan_out.pcap
ping 10.42.2.6 -c 20

On Destination Node (k3ssvr03–172.16.36.9):

sudo tcpdump -i ens18 -n -e -v udp port 8472

Key Observations:

• Source: VXLAN packets were being sent (TX counters increasing, no errors)
• Destination ens18: No incoming VXLAN packets captured during the ping test
• Destination flannel.1: RX: 0 packets - confirming packets never arrived

# On k3ssvr03 during ping test
tcpdump: listening on ens18, link-type EN10MB (Ethernet)
# (No packets captured from k3ssvr01)
ip -s link show flannel.1
# RX:  bytes packets errors dropped  missed   mcast
#      0       0      0       0       0       0

Note: The vxlan_in.txt capture file shows VXLAN traffic from k3ssvr03 to other nodes (TCP SYNs to k3ssvr01 and k3ssvr02), not incoming traffic from k3ssvr01.

Conclusion: VXLAN packets left the source VM but never reached the destination VM’s network interface. This strongly indicated an external filtering issue, but we needed to rule out configuration-specific problems before escalating.

Phase 3: Configuration Verification

We verified the VXLAN tunnel configuration was correct:

# VNI matched (ID: 1)
ip -d link show flannel.1
# vxlan id 1 local 172.16.36.9 dev ens18 dstport 8472
# FDB entries were correct
bridge fdb show | grep flannel.1
# 7a:ce:2c:16:2e:90 dev flannel.1 dst 172.16.36.7 self permanent
# Routing table was correct
ip route | grep 10.42
# 10.42.0.0/24 via 10.42.0.0 dev flannel.1 onlink

Phase 4: Alternative Configuration Testing

While Phase 2 pointed to external filtering, we conducted targeted configuration tests to rule out specific VXLAN implementation issues that might trigger external security mechanisms:

Test 1: Modified VXLAN Source Port

Reasoning: Flannel’s default srcport 0 0 uses a fixed source port, which can cause ECMP hashing issues or trigger connection tracking limits in some network devices. Randomizing the source port range might bypass such restrictions.

Expectation: If the external network was dropping packets due to fixed-port patterns (e.g., DoS protection), random ports should resolve the issue.

ip link add flannel.1 type vxlan \
    id 1 \
    local 172.16.36.9 \
    dev ens18 \
    srcport 32768 60999 \
    dstport 8472

Result: Still no incoming packets on destination. Conclusion: Not a source-port-specific filtering issue.

Test 2: Disabled NIC Offloading

Reasoning: Virtualized NIC offloading (GRO/GSO/TSO) can sometimes corrupt or mishandle VXLAN packets, particularly with older virtio_net drivers or specific hypervisor versions. Disabling offloading forces the kernel to process packets in software.

Expectation: If packet corruption was occurring due to offloading, disabling it should allow packets to arrive intact.

ethtool -K ens18 gro off gso off tso off

Result: No improvement. Conclusion: Not a NIC offloading issue.

Test 3: Direct VXLAN Test

Reasoning: To isolate whether the issue was specific to Flannel’s implementation or VXLAN in general, we tested with a manually created VXLAN interface using the standard IANA port (4789).

Expectation: If port 8472 was specifically blocked but 4789 was allowed, this would identify a port-based firewall rule.

ip link add vxlan_test type vxlan id 100 local 172.16.36.7 dev ens18 dstport 4789 remote 172.16.36.9

Result: Also failed. Conclusion: The issue was not specific to port 8472 or Flannel’s implementation, but rather VXLAN encapsulation in general.

Phase 5: External Factor Investigation

Since all internal configurations were correct and packets were leaving the source but never arriving at the destination, we investigated external factors:

# Checking hypervisor details
cat /sys/class/dmi/id/product_name  # acloud
cat /sys/class/dmi/id/bios_version  # 1.13.0-20201211_142035
lscpu | grep "Hypervisor vendor"     # KVM

Critical Finding: The same standardized K3s configuration runs successfully on 50+ other customer environments with various hypervisors (VMware, Proxmox, public cloud), isolating this to environment-specific behavior.

4. Root Cause Analysis

The evidence conclusively points to external network/hypervisor restrictions on VXLAN traffic:

1. Packets leave the source VM successfully (TX counters increase, no errors at NIC level)
2. Packets never reach the destination VM (RX: 0 on flannel.1, no packets seen on ens18 tcpdump)
3. Host-gw mode works perfectly (proving Layer 2/3 underlay connectivity is intact)
4. Standardized configuration works on 50+ other customer environments (proving software stack is correct)
5. Limited visibility — We cannot access the hypervisor or physical network infrastructure for further investigation

Why Host-GW Works?

Host-GW mode uses direct IP routing without encapsulation:

# Host-GW: Direct IP packet (no special handling required)
[Eth: src=VM1_MAC, dst=VM2_MAC] [IP: src=10.42.0.11, dst=10.42.2.6]
# VXLAN: Encapsulated packet (may be filtered by security policies)
[Eth: src=VM1_MAC, dst=VM2_MAC] [IP: src=172.16.36.7, dst=172.16.36.9] [UDP: 8472] [VXLAN] [Inner Eth] [Inner IP]

The external network infrastructure handles these differently — standard IP packets pass through, while VXLAN-encapsulated packets may be filtered by security policies, ACLs, or virtual switch configurations beyond our visibility.

5. Solution

Immediate Fix: Switch to Host-GW Mode

Since all nodes were on the same Layer 2 network (172.16.36.0/24), we migrated to host-gw mode:

# On all nodes
sudo systemctl stop k3s
sudo rm -rf /run/flannel/ /var/lib/cni/
sudo ip link delete flannel.1 2>/dev/null || true
# Modify K3s service
sudo systemctl edit k3s --full
# Add: --flannel-backend=host-gw
sudo systemctl daemon-reload
sudo systemctl start k3s

Advantages:

• ✅ Better performance: No UDP encapsulation/decapsulation overhead
• ✅ Lower latency: Direct routing eliminates tunnel processing
• ✅ Higher compatibility: Works across diverse infrastructure types without requiring VXLAN support
• ✅ Simpler troubleshooting: Standard IP routing is easier to debug

Alternative Approaches (If Host-GW Not Suitable)

1. Contact Infrastructure Team: Request whitelist of UDP port 8472 or VXLAN protocol support
2. CNI Change: Migrate to Cilium with BGP mode (native routing, no tunneling)
3. Network Architecture Change: Deploy nodes across different subnets requiring NAT gateway (forces different CNI approach)

6. Host-GW Best Practices

Based on K3s documentation and community experience :

When to Use Host-GW

| Scenario                                     | Recommendation                         |
| -------------------------------------------- | -------------------------------------- |
| **Same L2 network** (on-premise, single VPC) | ✅ **Ideal for host-gw**                |
| **Cloud environments** (AWS, GCP, Azure)     | ❌ Use VXLAN or cloud-specific backend  |
| **Cross-subnet deployment**                  | ❌ Use VXLAN with `Directrouting: true` |
| **Maximum performance required**             | ✅ Use host-gw                          |

Performance Comparison

| Metric     | VXLAN           | Host-GW    | Improvement |
| ---------- | --------------- | ---------- | ----------- |
| Throughput | \~8 Gbps        | \~9.5 Gbps | +20%        |
| CPU Usage  | \~40-50%        | \~25-30%   | -25%        |
| Latency    | \~0.5ms         | \~0.3ms    | -40%        |
| Overhead   | 50 bytes/packet | 0 bytes    | None        |

7. Key Takeaways and Recommendations

For Production Deployments

Make Host-GW the Default for L2-Connected Clusters

Based on this experience and K3s best practices, we recommend using --flannel-backend=host-gw as the default configuration when:

• All nodes are on the same Layer 2 network
• No cross-subnet routing is required
• Maximum compatibility is desired

Benefits of this approach:

• Eliminates an entire class of VXLAN-related issues
• Better network performance (no encapsulation overhead)
• Easier packet capture and debugging
• Works reliably across diverse infrastructure types

For Troubleshooting Similar Issues

When facing cross-node communication failures:

1. Check RX counters on flannel.1: If RX is 0, problem is external
2. tcpdump on physical interface: Confirm whether packets arrive at the VM boundary
3. Test with host-gw mode: Quick way to isolate overlay vs. underlay issues
4. Verify environment consistency: Same software working elsewhere points to infrastructure
5. Know your limits: Accept when issue is beyond your control and design around it

8. Conclusion

This case demonstrates the importance of designing systems that gracefully handle infrastructure constraints. While VXLAN is a robust overlay technology, it introduces dependencies on the underlying network’s willingness to carry encapsulated traffic.

By recognizing that:

1. We proved the issue was external (packets left VM but never reached destination)
2. We could not investigate further due to lack of infrastructure access
3. Host-gw mode provided equivalent functionality with better performance and compatibility

We achieved a faster, more reliable resolution than continuing to debug an inaccessible system. Sometimes the best engineering decision is to eliminate the problematic component rather than fix it.

When all nodes share a network, route directly. Don’t tunnel unless you need to cross boundaries.

References

1. How to Implement Flannel with host-gw Backend for Performance — OneUptime Blog, 2026. Performance comparison showing 20% throughput improvement and 40% latency reduction with host-gw.
2. How to Handle K3s Networking — OneUptime Blog, 2026. “host-gw: Routes packets directly between nodes without encapsulation… Delivers the best performance when network topology allows.”
3. Basic Network Options — K3s Documentation — Official K3s docs. “--flannel-backend=host-gw: Use IP routes to pod subnets via node IPs. Requires direct layer 2 connectivity between all nodes in the cluster."
4. Flannel Backends Documentation — Flannel GitHub. “VXLAN is the recommended choice. host-gw is recommended for more experienced users who want the performance improvement and whose infrastructure support it.”
5. What flannel-backend are you using when deploying k3s? — K3s GitHub Discussions. Community discussion on backend selection, with recommendations for host-gw in L2-connected environments.