VXLAN over IPsec for Multi-VLAN Transport in Hybrid Networks

Introduction

When extending enterprise or ISP-grade networks across data centers or branch offices, engineers often need to preserve Layer 2 semantics (VLANs, ARP, broadcast) while securing traffic over untrusted WAN links. While GRE or Geneve encapsulations are common, VXLAN has become the de facto standard for cloud and SDN environments thanks to its wide support in NICs, switches, and hypervisors. In this article, we’ll explore how to deploy VXLAN over IPsec with strongSwan to carry multiple VLANs across sites, compare it with other tunneling methods, and provide practical configuration examples.

Why VXLAN?

VXLAN (Virtual eXtensible LAN) encapsulates Layer 2 frames inside UDP packets, using VXLAN Network Identifiers (VNIs) instead of VLAN IDs. VXLAN is attractive because:

• It scales to 16 million VNIs (vs. 4096 VLANs).
• It’s hardware-accelerated on many NICs and switches.
• It’s supported across Linux, VMware, OpenStack, Kubernetes, and most SDN controllers.
• Its encapsulation is well-understood, making it interoperable in multi-vendor environments.

Drawback: VXLAN adds more overhead (~50 B) than native VLAN bridging, and when combined with IPsec (ESP), the per-packet overhead grows further. MTU tuning is therefore crucial.

Design Goals

• Secure transport between Host A (site edge router/firewall) and Host B (remote site).
• Carry five VLANs (10, 11, 12, 20, 30) across the encrypted tunnel.
• Allow inter-VLAN routing at Host A.
• Provide NAT masquerading for outbound Internet access from Host B.
• Support DHCP and DNS for LAN clients.
• Ensure scalability to support up to 500–600 clients with an aggregate throughput of ~600 Mbps.

Encapsulation and Overhead

• VXLAN header: ~50 B (UDP + VXLAN).
• ESP (AES-GCM): ~50 B.
• Total overhead: ~100 B per packet.

For 1500 B Ethernet MTU:

• Effective payload ~1400 B.
• MTU should be clamped to ~1380 B in the tunnel.

Compared to GRE (~80 B overhead), VXLAN is slightly less efficient but widely supported in SDN/NFV stacks, making it a strong choice in cloud-native deployments.

System and Kernel Tuning

On both hosts (/etc/sysctl.conf):

# Forwarding
net.ipv4.ip_forward=1
net.ipv6.conf.all.forwarding=1

# Disable ICMP redirects
net.ipv4.conf.all.accept_redirects=0
net.ipv4.conf.all.send_redirects=0

# Increase buffer sizes
net.core.rmem_max=26214400
net.core.wmem_max=26214400

# Tunnel performance
net.ipv4.tcp_mtu_probing=1

Interface Configuration (Host A and B)

Host A (Gateway with NAT):

# VXLAN interface
ip link add vxlan1 type vxlan id 42 dev eth0 dstport 4789 remote B.B.B.B
ip link set vxlan1 up

# Bridge
brctl addbr br-vpn
brctl addif br-vpn vxlan1
brctl addif br-vpn eth1.10 eth1.11 eth1.12 eth1.20 eth1.30
ip link set br-vpn up

Host B (Remote site):

ip link add vxlan1 type vxlan id 42 dev eth0 dstport 4789 remote A.A.A.A
ip link set vxlan1 up

brctl addbr br-vpn
brctl addif br-vpn vxlan1
brctl addif br-vpn eth1.10 eth1.11 eth1.12 eth1.20 eth1.30
ip link set br-vpn up

Firewall with nftables (Host A)
table inet filter {
  chain input {
    type filter hook input priority 0;
    policy drop;

    ct state established,related accept
    iif "lo" accept

    # Allow SSH, DHCP, DNS
    tcp dport {22, 53} accept
    udp dport {67, 68, 53} accept

    # IPsec
    udp dport {500,4500} accept
    esp accept
  }

  chain forward {
    type filter hook forward priority 0;
    policy drop;

    ct state established,related accept

    # Allow LAN/VLANs via VXLAN
    iifname "vxlan1" accept
    oifname "vxlan1" accept

    # Allow VLAN interfaces
    iifname { "eth1.10", "eth1.11", "eth1.12", "eth1.20", "eth1.30" } accept
  }

  chain output {
    type filter hook output priority 0;
    policy accept;
  }
}

table ip nat {
  chain postrouting {
    type nat hook postrouting priority 100;
    oif "eth0" masquerade
  }
}

strongSwan Configuration

/etc/ipsec.conf

config setup
  charondebug="ike 1, knl 1, cfg 0"

conn vxlan-ipsec
  auto=start
  keyexchange=ikev2
  ike=aes256gcm16-prfsha256-ecp256!
  esp=aes256gcm16-ecp256!
  left=A.A.A.A
  right=B.B.B.B
  leftid=A.A.A.A
  rightid=B.B.B.B
  leftsubnet=0.0.0.0/0
  rightsubnet=0.0.0.0/0
  dpdaction=restart
  dpddelay=30s

/etc/ipsec.secrets

A.A.A.A B.B.B.B : PSK "strongsharedsecret"

Performance Analysis

• With a 1000 B average packet size, VXLAN+IPsec overhead ≈ 10%.
• With 1400 B packets, the overhead drops ≈ 7%.
• On a 600 Mbps link, expect ~540 Mbps usable payload for large packets.
• CPU requirements: An AES-NI capable CPU (with at least 2 cores at 2 GHz) is recommended for throughput greater than 500 Mbps.

Comparison with GRE and GENEVE

VXLAN offers an excellent trade-off: nearly universal support and integration with SDN/NFV systems, at the cost of slightly higher per-packet overhead.

Conclusion

VXLAN over IPsec offers a secure and SDN-friendly method for extending Layer 2 and Layer 3 domains across multiple sites. While it introduces slightly more overhead than GRE, it benefits from broad hardware offload and cloud-native support, making it an ideal choice for scalable deployments with 500–600 clients and multiple VLANs. With proper MTU tuning, sysctl optimizations, and strongSwan-based encryption, VXLAN over IPsec can deliver secure multi-VLAN transport at near line-rate performance.