필사 모드: ping works but only large requests hang — dissecting MTU, MSS and the PMTUD blackhole

Introduction — small requests work and only large responses hang

There is one class of incident with unusually distinctive symptoms.

• ping succeeds 100 percent of the time and the round trip time is normal.
• The health check endpoint responds fine.
• But as soon as a response grows a little, as with a list API, the request just stalls. Not an error, simply a stall.
• ssh connects, but typing ls -la in a large directory freezes the screen.
• git clone stops at some point in the progress bar and never finishes.
• Some sites hang during the TLS handshake.

Once you see this combination there is nothing else to look at. It is a path MTU problem. And it almost always appears right after a VPN, a tunnel or an overlay network was newly attached, or after the path changed.

This post covers why the outcome splits by size, and how to confirm it within a few minutes.

MTU and MSS — the arithmetic of 40 bytes

MTU is the maximum size of an IP packet that one interface can emit at a time. The Ethernet default is 1500 bytes.

TCP subtracts headers from that value to calculate how much data it can carry. Subtract the 20-byte IPv4 header and the 20-byte TCP header and you get 1460 bytes. That value is MSS, and it is carried as an option in the SYN packet of the handshake and delivered to the peer.

ip link show ens5

2: ens5: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP mode DEFAULT group default qlen 1000
    link/ether 06:1f:2a:3b:4c:5d brd ff:ff:ff:ff:ff:ff

sudo tcpdump -nn -i any 'tcp[tcpflags] & tcp-syn != 0' -c 2

14:22:03.104512 IP 10.0.1.20.51244 > 10.20.0.5.443: Flags [S], seq 2841029371, win 62727, options [mss 1460,sackOK,TS val 33012 ecr 0,nop,wscale 7], length 0
14:22:03.106880 IP 10.20.0.5.443 > 10.0.1.20.51244: Flags [S.], seq 991044820, ack 2841029372, win 65160, options [mss 1460,sackOK,TS val 88103 ecr 33012,nop,wscale 7], length 0

One decisive fact comes out of this. Both sides decide their MSS by looking only at the MTU of their own interface. Nobody knows whether there is a narrower segment somewhere along the path. In the capture above the two hosts agreed to exchange 1460-byte segments, but if a 1420-byte tunnel sits in between, that agreement cannot be honoured.

This is why small requests succeed. A health check response finishes in a single 200-byte segment, so it passes the narrow segment without trouble. ping has a default payload of 56 bytes and always gets through. Only packets whose size crosses the threshold disappear, which is why the symptom splits by size.

Hanging during the TLS handshake has the same cause. ClientHello is small, but the certificate chain the server sends is 3000 to 5000 bytes, which immediately becomes maximum-size segments. So you get the shape where the connection establishes but the handshake never completes.

The exact process by which a PMTUD blackhole is created

TCP does have a mechanism to solve this problem: Path MTU Discovery. The sequence is as follows.

1. Linux sets the DF (Don't Fragment) bit on TCP packets by default.
2. When an intermediate router receives a DF packet larger than the MTU of its next hop, it cannot fragment, so it discards the packet.
3. Instead it sends the sender an ICMP type 3 code 4, that is, Fragmentation Needed. That message carries the MTU value of the next hop.
4. The sender kernel stores that value in a per-route cache and resends with smaller segments.

When it works correctly it looks like this.

ip route get 10.20.0.5

10.20.0.5 via 10.0.1.1 dev ens5 src 10.0.1.20 uid 1000
    cache expires 588sec mtu 1420

mtu 1420 appearing in the cache means Path MTU Discovery worked.

The problem is step 3. If there is a firewall blocking all ICMP, a router that does not answer with ICMP, or an asymmetric path that cannot return the ICMP to the original sender, this message disappears. The sender then keeps sending large packets without knowing anything, and those packets keep getting discarded. Retransmissions are the same size and get discarded too. This state is called a PMTUD blackhole.

The characteristic of a blackhole is that no error is raised. The connection stays ESTABLISHED, and the application waits for a response until its own timeout finally ends it. This leads to the wrong conclusion that "the network is fine, the application is just slow".

A widespread misunderstanding needs correcting here. The policy of "blocking all ICMP for security" is itself a source of incidents. What you may block is roughly echo request and echo reply, and type 3 code 4 must always be allowed through. Under IPv6, Packet Too Big, that is ICMPv6 type 2, plays the same role, and there you do not even have the option.

Confirm whether the ICMP actually arrives.

sudo tcpdump -nn -i any 'icmp[icmptype] == 3 and icmp[icmpcode] == 4'

14:31:02.884411 IP 10.0.1.1 > 10.0.1.20: ICMP 10.20.0.5 unreachable - need to frag (mtu 1420), length 556

If you see this line, Path MTU Discovery is alive. If you keep sending large packets and this line never once appears, it is a blackhole.

Linux has a feature that detects blackholes on its own. It is off by default.

sysctl net.ipv4.tcp_mtu_probing

net.ipv4.tcp_mtu_probing = 0

# 1: probe only when a blackhole is suspected, 2: always probe
sudo sysctl -w net.ipv4.tcp_mtu_probing=1

With the value set to 1, when retransmissions repeat, the kernel shrinks the segment size on its own until it finds a size that gets through. It is not a root fix, but it is effective at keeping a service alive while the cause is still unknown.

The bytes a tunnel takes away

Most narrowing of the path comes from encapsulation. The original packet gets wrapped in another header, so the space usable inside shrinks. Memorising the numbers speeds up diagnosis.

A few points deserve care.

The reason wg-quick sets 1420 by default on a WireGuard interface is that it is a safe value accounting for an IPv6 outer header as well. If you only ever use IPv4, 1440 is possible, but it breaks the moment the endpoint switches to IPv6, so leaving 1420 as it is is the better choice.

IPsec overhead is not fixed, because of the cipher algorithm and padding. So you plan around the worst case. AES-CBC is variable due to block padding, while AES-GCM is comparatively small.

The cloud has a trap in the opposite direction. Most AWS instance types use an MTU of 9001 inside the VPC. It works well within the same VPC, but the moment traffic passes an internet gateway or a VPN it drops to 1500 or below. So it shows up in the form "internal communication is fine, but only external API calls hang on large responses".

Finding the path MTU by binary search

The surest method is to measure it directly. Set the DF bit, vary the size, and watch what gets through.

# -M do: set the DF bit, -s: ICMP payload size
ping -M do -s 1472 -c 1 10.20.0.5

PING 10.20.0.5 (10.20.0.5) 1472(1500) bytes of data.
From 10.0.1.1 icmp_seq=1 Frag needed and DF set (mtu = 1420)

--- 10.20.0.5 ping statistics ---
1 packets transmitted, 0 received, +1 errors, 100% packet loss

If you are lucky the router tells you the exact MTU like this. On a path where ICMP is blocked you simply get no answer, so you have to narrow it down yourself.

for s in 1472 1420 1400 1392 1380 1372; do
  printf '%5d bytes payload (MTU %5d): ' "$s" "$((s + 28))"
  if ping -M do -s "$s" -c 1 -W 1 10.20.0.5 >/dev/null 2>&1; then
    echo OK
  else
    echo FAIL
  fi
done

 1472 bytes payload (MTU  1500): FAIL
 1420 bytes payload (MTU  1448): FAIL
 1400 bytes payload (MTU  1428): FAIL
 1392 bytes payload (MTU  1420): OK
 1380 bytes payload (MTU  1408): OK
 1372 bytes payload (MTU  1400): OK

The 28 here is the 8-byte ICMP header plus the 20-byte IPv4 header. The largest payload that gets through is 1392, so the path MTU is 1420. That means the TCP MSS should be 1420 minus 40, which is 1380.

Using tracepath sometimes gives it to you in one shot. It even shows you at which hop the value drops.

tracepath -n 10.20.0.5

 1?: [LOCALHOST]                      pmtu 1500
 1:  10.0.1.1                                              0.412ms
 1:  10.0.1.1                                              0.318ms
 2:  10.0.1.1                                              0.286ms pmtu 1420
 2:  10.60.4.1                                             8.114ms
 3:  10.20.0.5                                             8.902ms reached
     Resume: pmtu 1420 hops 3 back 3

pmtu 1420 appeared at hop 2. That point is the tunnel entrance.

Check whether the kernel has already learned a value, too.

ip route get 10.20.0.5
ip route show cache

To discard a wrongly learned value, flush the cache.

sudo ip route flush cache

Why MSS clamping is the practical answer

You now know the path is narrow. Time to fix it. There are three candidates.

First, lower the interface MTU.

sudo ip link set dev ens5 mtu 1420

This method only affects packets that host sends. It does constrain the packets the peer sends as well, because the MSS I advertise shrinks. The problem is scope. If the narrow segment is a tunnel between two routers with hundreds of servers and thousands of clients on either end, you cannot touch the MTU of every device.

Second, let ICMP through and revive Path MTU Discovery. In principle this is the most correct fix and it must always be done in parallel. But the firewall in the middle of the path is often outside our control, and there is nothing we can do about ICMP being blocked somewhere on the other side of the internet.

Third, MSS clamping. The router the tunnel passes through directly rewrites the MSS option value in the SYN packets going by.

# Match it automatically to the path MTU
sudo iptables -t mangle -A FORWARD -p tcp --tcp-flags SYN,RST SYN \
  -j TCPMSS --clamp-mss-to-pmtu

# Constrain only traffic leaving via the tunnel interface, to a fixed value
sudo iptables -t mangle -A FORWARD -p tcp --tcp-flags SYN,RST SYN \
  -o wg0 -j TCPMSS --set-mss 1380

# nftables notation
sudo nft add rule inet mangle forward tcp flags syn tcp option maxseg size set rt mtu

The reasons this approach became the practical standard are clear.

You only touch the SYN and SYN-ACK of the handshake, so the cost is effectively zero. And since MSS is a value announcing "the maximum size I can receive", rewriting the SYN in both directions makes both sides send only small segments. Above all, it does not depend on ICMP at all, so it works reliably even in a blackhole environment.

After applying it, confirm with a capture.

sudo tcpdump -nn -i wg0 'tcp[tcpflags] & tcp-syn != 0' -c 2

15:02:11.104512 IP 10.60.1.20.51302 > 10.20.0.5.443: Flags [S], seq 118203993, win 62727, options [mss 1380,sackOK,TS val 41120 ecr 0,nop,wscale 7], length 0
15:02:11.112880 IP 10.20.0.5.443 > 10.60.1.20.51302: Flags [S.], seq 771102384, ack 118203994, win 65160, options [mss 1380,sackOK,TS val 90221 ecr 41120,nop,wscale 7], length 0

The mss on both sides changed to 1380.

There are caveats too. MSS clamping applies to TCP only. UDP-based protocols get no protection. HTTP/3 over QUIC, DNS receiving large responses over UDP, and another UDP tunnel inside WireGuard all fall into this category. QUIC performs DPLPMTUD on its own and uses 1200 bytes as a safe default, so it generally holds up, but applications that handle UDP directly must limit their datagram size themselves.

What changes with the Kubernetes CNI and IPv6

In Kubernetes this problem recurs regularly because of the overlay network. Flannel VXLAN, Calico VXLAN, and the VXLAN or Geneve modes of Cilium all use encapsulation.

The symptoms are characteristic.

• ping between pods works and small requests are fine.
• Only POST requests with a large body stall.
• kubectl exec works, but kubectl logs on a pod with lots of logs stops midway.
• Image pulls start but stall on a large layer.

Start the check by comparing interface MTUs.

# Physical interface on the node
ip link show ens5 | head -1
# Overlay interface
ip link show flannel.1 | head -1
# Inside the pod
kubectl exec -it deploy/api -- ip link show eth0 | head -1

2: ens5: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP mode DEFAULT group default qlen 1000
5: flannel.1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN mode DEFAULT group default
3: eth0@if42: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP mode DEFAULT group default

There is an error here. The MTU of flannel.1, the VXLAN interface, is 1500, the same as the node. VXLAN takes 50 bytes, so it should be 1450. The pod eth0 should be 1450 too.

A healthy state looks like this.

2: ens5: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 ...
5: flannel.1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 ...
3: eth0@if42: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 ...

Stating the MTU explicitly in the CNI is the surest approach. Leaving it to auto-detection goes wrong when the node interface changes or when the cloud uses jumbo frames.

# Flannel: net-conf.json
{
  "Network": "10.244.0.0/16",
  "Backend": { "Type": "vxlan", "MTU": 1450 }
}

# Calico: check the configured MTU
kubectl get configmap -n kube-system calico-config -o jsonpath='{.data.veth_mtu}'

# Cilium: check the detected value
kubectl exec -n kube-system ds/cilium -- cilium-dbg status --verbose | grep -i mtu

On AWS, if the node uses 9001 and the CNI comes out at 1500 there is no problem, but the reverse — the node at 1500 and the CNI at 9001 — becomes a blackhole immediately. You must keep the order of checking the node MTU first and then subtracting the encapsulation overhead from that value.

IPv6 has different rules. Three things to remember.

First, IPv6 routers do not fragment packets. In IPv4, a router could split and forward a packet when the DF bit was absent, but in IPv6 that option does not exist at all. Only the sending host can fragment.

Second, that is why the ICMPv6 Packet Too Big, type 2 message is mandatory. Block it and IPv6 communication breaks even more definitively than IPv4. Guidance for filtering ICMPv6 is laid out in RFC 4890, and type 2 must be on the allow list.

Third, the minimum MTU for IPv6 is 1280 bytes. Any path is guaranteed to carry at least that size, so if you have to keep a service alive temporarily while the cause is still unknown, 1280 is the safe lower bound.

# Measure the IPv6 path MTU
ping6 -M do -s 1232 -c 1 2001:db8::5

# Check whether IPv6 ICMP arrives
sudo tcpdump -nn -i any 'icmp6 and ip6[40] == 2'

15:18:44.201188 IP6 2001:db8:1::1 > 2001:db8:2::20: ICMP6, packet too big, mtu 1400, length 1240

Closing — when the outcome splits by size, suspect the MTU

The starting point of the diagnosis is the shape of the symptom. If the connection works but stalls only as size grows, and stalls with no error at all, measure the path MTU before looking at anything else. One command is enough.

ping -M do -s 1472 -c 1 destination-address

If that fails while small sizes succeed, it is settled.

There are three things to remember. TCP decides its MSS by looking only at its own interface, so it cannot know about a bottleneck in the middle of the path by itself. The only mechanism that tells it is ICMP type 3 code 4, and the moment you block that message a silent blackhole is created. And because we cannot control the entire path, MSS clamping at the tunnel entrance becomes the realistic answer.

When you introduce a new tunnel or overlay, it is best to build the overhead calculation and MSS clamping into the design from the start. Discover it later and the symptoms are so vague that you will burn days on it.