PMTUD, MTU Mismatch, and the Hidden Fragmentation that Slows Your Cluster

A deep dive into how packets actually move through IPVS, IPIP tunnels, containers, and mixed MTUs

Engineers learn early that Path MTU Discovery (PMTUD) prevents fragmentation on the Internet. We are told a simple story. If a router sees a packet that is too large, it sends an ICMP message back and the sender reduces its size. Clean. Predictable. Safe.

Production networks tell a different story.

Modern clusters use containers, virtual interfaces, tunnels, offload engines, GRO, and IPVS. These layers change the rules. Packets often get fragmented even when DF is set. PMTUD fires at the wrong hop or does not fire at all. Latency spikes, unexplained tail delays, or drops appear out of nowhere.

This article breaks down why. We walk through real packet paths and trace exactly where PMTUD succeeds and where it fails. The examples use diagrams and walk the packet hop by hop.

The Basics You Must Understand

PMTUD relies on one simple rule:

A router must see a too large packet with DF=1 and it must choose to drop it and send ICMP Type 3 Code 4.

If any of the following happens first, PMTUD is bypassed:

• A NIC drops the frame before the kernel sees it.
• A virtual device fragments the packet locally.
• A tunnel reduces the effective MTU but never reports it.
• GRO merges packets into a single skb that hides the original packet boundaries.
• ICMP is filtered.
• A forwarding path avoids the logic that sends ICMP.

In container environments, these conditions appear constantly.

Case 1. Client MTU 9000 to Server MTU 1500

Packet size: 5000 bytes.

Client (9000) ---> Router (9000) ---> Server (1500)

What will happen

What sometimes happens

• The NIC on the server drops the packet before it reaches IP.
• No ICMP is generated.
• PMTUD never triggers.
• The client keeps sending packets the path cannot handle.

This is one of the most common silent packet drop because of MTU.

Case 2. Container MTU 9000 to Pod MTU 1500

A simple but important pattern.

Client Pod (9000)
   → Host A (9000)
   → Host B (9000)
   → Pod veth (1500)

The bottleneck is the pod veth.

What actually happens

1. Packet arrives on Host B with size 5000.
2. Host B tries to forward it to the pod veth with MTU 1500.
3. DF=1, packet is too big.
4. Host B emits ICMP “Frag Needed” to the client.
5. PMTUD works.

This is PMTUD working at the last hop.

Case 3. IPVS Load Balancer with IPIP Tunnels

IPIP adds an outer IP header:

Before IPIP:
MAC + IP + TCP + Payload   (~1500 bytes)

After IPIP:
MAC + Outer IP (20 bytes) + Inner IP + TCP + Payload

This small 20 byte increase often causes the packet to exceed the MTU.

Clean PMTUD case

Client sends 1500 byte packet (DF=1)
IPVS encapsulates → 1520
LB MTU is 1500
Packet is dropped
LB sends ICMP "MTU=1480" to client
Client adjusts PMTU

Diagram:

Client (1500)
   → IPVS encapsulation (1520)
   → LB (1500)
       X too large
       ↳ ICMP "Frag Needed, 1480"

This works as expected, but only when packets arrive individually.

Case 4. GRO Breaks PMTUD

This is one of the most misunderstood failure modes.

What GRO does

On the LB, the NIC aggregates multiple packets into one skb. Example:

Client sends 3 TCP segments:

[1500][1500][1500]

GRO merges them into:

[4500 byte GRO skb]

Now apply IPIP

• Inner 4500
• Outer 20
• Resulting skb is 4520 bytes

Now send through a 1500 byte link

Linux fragments it locally into multiple packets:

[Frag1][Frag2][Frag3][Frag4][Frag5][Frag6]

And here is the critical part:

No ICMP. No PMTUD. No adjustment.

Diagram:

Client → LB → GRO (4500)
     → IPIP (4520)
     → Egress (1500)
         → Local fragmentation into 6 packets
         → No ICMP

This is where unexpected latency, reorder, and packet loss start appearing. Your cluster is fragmenting packets at the worst possible place.

Step by Step Walkthrough of Real Topologies

Topology A

(Client 1500 → BM1 9000) → LB 9000 → (BM2 1500 → Pod 1500)

1. Client sends 1500 byte frame.
2. LB encapsulates to ~1520.
3. BM2's MTU is 1500, which can't accept packet with size 1520
4. Packets will be dropped silently

Topology B

(Client 9000 → BM1 9000) → LB 9000 → (BM2 9000 → Pod 1500)

1. Client has handshake with server, the packet will go through LB. 
2. Client will not sends packets with size > 1500.
3. LB accept them.
3. LB encapsulates and forwards, the packet size can be 1500 + 20.
4. BM2 accepts jumbo.
5. Final hop sees packet size 1520 > 1500.
6. ICMP is sent back. 
7. LB will update the Server's MTU in cache. 
8. Client send packets size = 1500
9. LB try to encapsulates and forward, then it found that 1500 + 20 > Server's MTU from cache
10. ICMP is sent back from LB. 
11. Client update server's MTU in cache with 1480

In this case, we can see two ICMP packets will be involved to make the MTU full matched from client to server. PMTUD still works, but only after the large frame travels across the network. This wastes bandwidth and increases retries.

Topology C

(Client 9000 → BM 9000) → LB 1500 → (BM 9000 → Pod 9000)

The LB is the bottleneck.

1. Client sends 9000 byte packet.
3. LB sees 9000 > 1500.(Let's suppose LB can accept jumbo frame) 
4. LB immediately sends ICMP.
5. Client lowers PMTU early.
6. All downstream traffic stays efficient.

Same Host MTU Conflict

A final case many clusters run into:

Client (9000)
  → IPVS VIP (9000)
  → IPVS Real Server (veth 1500)
  → Pod (1500)

Client and VIP are on the same node.

As we can see the request will be DNATed, but the MTU is the not the same between the VIP and Real Server. For this scenario, if MTU not match, in most of time, Kernel can take care via PMTUD, but in some scenarios we can see it cause headaches since something unexpected happened like node use the in-cluster VIP IP to access in-cluster VIP (client IP == in-cluster VIP IP).

What You Should Do

To avoid ghost fragmentation and random latency spikes in container networks:

1. Use a consistent MTU across all nodes whenever possible.
2. Clamp MSS to account for encapsulation overhead.
3. Clamp MSS at edges when dealing with mixed MTUs.
4. Disable GRO in specific LB interfaces if you need accurate PMTUD behavior.
5. Monitor for fragmentation using counters on tunnel interfaces and NICs.

PMTUD is a useful feature, but in modern networks it should not be your only safety mechanism.