필사 모드: HTTP Keep-Alive and connection reuse — the timeout race condition behind intermittent 502s

Introduction — the backend is perfectly healthy and 502s still show up

Once traffic grew, reports like this start coming in.

• The load balancer access log has 502s mixed in at around 0.05 percent of all requests.
• At the same timestamp there is no trace of that request at all in the backend application log.
• CPU and memory are comfortable, and the health check has never once failed.
• It cannot be reproduced. Running a load test actually makes things look fine.

The common response here is to add backends or adjust the health check thresholds. Neither works. The cause of this symptom is not capacity but timing.

When the moment the server decides to close an idle connection overlaps with the moment the client decides to reuse it, the request and the FIN pass each other. The server answers data arriving on an already closed connection with an RST, and the proxy in the middle translates that into a 502. This race condition cannot be eliminated entirely given the structure of HTTP/1.1, and lining up the timeout order to drive the probability close to zero is the only response available.

This post covers why that order has to be what it is, and how to make the remaining decisions around connection reuse.

The cost of opening a single connection

First, look at why reuse matters in numbers. Here are the round trips needed to open one new HTTPS connection.

DNS lookup      0-1 RTT (0 if cached)
TCP handshake   1 RTT   (SYN, SYN-ACK, ACK)
TLS 1.3         1 RTT
TLS 1.2         2 RTT
                --------
Total           at least 2 RTT with TLS 1.3

Within the same region, where RTT is 1 millisecond, that is 2 milliseconds. Negligible. But from Seoul to the US East coast the RTT is around 180 milliseconds. That is 360 milliseconds spent purely on establishing the connection. Even with a server processing time of 20 milliseconds, the user experiences 380 milliseconds.

From this point on, connection reuse produces an overwhelmingly larger effect than application optimisation. On a reused connection those 2 RTT become zero outright.

Measuring it makes this clear.

cat > /tmp/curl-format.txt <<'EOF'
    time_namelookup:  %{time_namelookup}s
       time_connect:  %{time_connect}s
    time_appconnect:  %{time_appconnect}s
   time_pretransfer:  %{time_pretransfer}s
 time_starttransfer:  %{time_starttransfer}s
                      ----------
         time_total:  %{time_total}s
        num_connects:  %{num_connects}
EOF

curl -s -o /dev/null -w "@/tmp/curl-format.txt" \
  https://api.example.com/v1/orders \
  https://api.example.com/v1/orders

    time_namelookup:  0.004312s
       time_connect:  0.184901s
    time_appconnect:  0.371244s
   time_pretransfer:  0.371402s
 time_starttransfer:  0.393118s
                      ----------
         time_total:  0.394552s
        num_connects:  1

    time_namelookup:  0.000021s
       time_connect:  0.000029s
    time_appconnect:  0.000031s
   time_pretransfer:  0.000094s
 time_starttransfer:  0.022704s
                      ----------
         time_total:  0.023918s
        num_connects:  0

The first request took 394 milliseconds, the second 24. Server processing time was the same 22 milliseconds in both. The 370 millisecond difference was entirely the cost of establishing the connection.

What HTTP/1.1, HTTP/2 and HTTP/3 each eliminated

Keeping track of what each version solved makes the choice easy.

HTTP/1.1 made persistent connections the default. Instead of closing the connection after a single response, it keeps using it. That said, on one connection requests and responses travel one at a time in order. Pipelining was in the specification, but compatibility problems with intermediate devices effectively killed it. So concurrency is obtained only by opening more connections. The convention of browsers opening 6 connections per domain came from this. This is head-of-line blocking at the HTTP layer.

HTTP/2 multiplexes many streams over a single TCP connection. Send 100 requests at once and there is still one connection. Head-of-line blocking at the HTTP layer disappears. Header compression and server priorities arrived here too.

There is a misunderstanding to correct here. The explanation that "HTTP/2 has one connection so it is always faster" is only conditionally true. TCP guarantees ordering, so when a single segment is lost, everything that arrived after it is trapped in the kernel buffer. All 100 multiplexed streams stall together. With 6 connections under HTTP/1.1, the loss would have affected only one of them. In other words, on a mobile network with a high loss rate HTTP/2 can be slower. Head-of-line blocking at the TCP layer remains.

This is exactly why HTTP/3 moved on top of QUIC. QUIC does per-stream independent loss recovery on top of UDP. Even when packets of one stream go missing, the other streams keep going. As a bonus, TLS is integrated into the transport layer, so a first connection takes 1 RTT and a resumption can be 0 RTT. And because connections are identified by a connection ID, the connection survives a switch from Wi-Fi to LTE.

The practical conclusion is this. Between users and the edge, HTTP/2 or HTTP/3 is favourable. Between a proxy and a backend inside a data centre, on the other hand, the loss rate is close to zero, so HTTP/1.1 keepalive is plenty, and it is actually simpler to operate because connection counts are easy to control. Many real configurations do exactly this: HTTP/2 at the edge, HTTP/1.1 keepalive to the backend.

Check the protocol like this.

curl -sI --http2 https://api.example.com/v1/health -o /dev/null -w '%{http_version}\n'
curl -sI --http3 https://api.example.com/v1/health -o /dev/null -w '%{http_version}\n'

2
3

The race condition created when timeouts are out of order

Now the core. The path a single request travels has at least three idle timeouts.

[client pool]  ---- connection A ----  [load balancer/proxy]  ---- connection B ----  [backend]
   idle timeout                             idle timeout                          keepalive_timeout

Let us look at what happens on connection B in chronological order. Say the backend keepalive timeout is 5 seconds and the load balancer idle timeout is 60 seconds.

1. A request is processed and connection B becomes idle.
2. Five seconds pass and the backend sends a FIN.
3. That FIN takes half a round trip time to reach the load balancer.
4. In exactly that gap a new request arrives, and the load balancer writes it onto connection B, which is still in its idle list.
5. The request arrives at the backend. The socket is already closed there, so it answers with an RST.
6. The load balancer decides the backend dropped the connection without responding and emits a 502.

This is why nothing at all is left in the backend application log. The request never reached application code.

The probability is low but it is not zero. And at thousands of requests per second, a low probability becomes hundreds of occurrences per day. That is why it only becomes visible as traffic grows.

The principle is single. The decision to close a connection must always be made by the side that sends requests. If the sender closes, there is no race, because it is a connection the sender decided not to use. Conversely, if the receiving side closes, there is always a race.

Therefore timeouts must get longer as you move from the outside inward. The client pool idle timeout should be the shortest, the load balancer longer than that, and the backend the longest.

The default idle timeout on an AWS ALB is 60 seconds, so set the backend comfortably longer than that.

# When nginx is the backend — longer than the 60 seconds of the ALB
keepalive_timeout 75s;
keepalive_requests 10000;

# Node.js backend
# server.keepAliveTimeout = 75000;
# server.headersTimeout   = 80000;   # must be larger than keepAliveTimeout

# Go backend
# srv := &http.Server{ IdleTimeout: 75 * time.Second }

The fact that the Node.js default keepAliveTimeout is 5 seconds causes accidents especially often. Put a Node server behind an ALB or nginx as-is and the scenario above reproduces exactly. Keeping headersTimeout larger than keepAliveTimeout is worth remembering alongside it.

When nginx is on the proxy side, it is easy to leave out the settings that enable upstream keepalive. All three lines must be present for reuse to actually happen.

upstream backend {
    server 10.0.3.11:8080;
    keepalive 64;                 # idle connections to keep per worker
    keepalive_timeout 60s;
    keepalive_requests 10000;
}

server {
    location / {
        proxy_pass http://backend;
        proxy_http_version 1.1;   # without this it goes out as 1.0 and keepalive is void
        proxy_set_header Connection "";  # the default close header must be removed
    }
}

Miss either proxy_http_version 1.1 or the empty Connection header and the upstream connection is opened fresh on every request. Most reports of "the configuration is there but it has no effect" come down to these two lines. Confirm by counting on the backend.

# Count ESTABLISHED connections coming from the proxy, on the backend
ss -tan state established '( sport = :8080 )' | wc -l
watch -n1 "ss -tan state established '( sport = :8080 )' | wc -l"

If this number keeps rising and falling in proportion to request volume, reuse is not happening; if it holds at a stable level, reuse is working.

Separately from reducing the race probabilistically, client-side retries should be in place as well. Go net/http automatically retries an idempotent request once when it fails on a reused connection before receiving a single response byte. Java Apache HttpClient checks connection state before lending it out via validateAfterInactivity. For a client with no such safeguard, adding explicit retries limited to idempotent requests such as GET and PUT is a good idea.

Pinpointing the bottleneck by splitting phases with curl -w

When reporting latency, the word "slow" is useless. Which phase is slow is what determines the owning team and the action. The curl timing variables do exactly this job.

The way to read them is subtraction.

curl -s -o /dev/null -w "@/tmp/curl-format.txt" https://api.example.com/v1/orders

    time_namelookup:  0.004312s
       time_connect:  0.031104s
    time_appconnect:  0.098742s
   time_pretransfer:  0.098901s
 time_starttransfer:  0.341285s
         time_total:  0.343990s

• DNS: 4.3 milliseconds. Normal.
• TCP handshake: 31.1 minus 4.3 is 26.8 milliseconds, so the RTT is about 27 milliseconds.
• TLS: 98.7 minus 31.1 is 67.6 milliseconds. That is roughly 2.5 times the RTT, so it is likely TLS 1.2. With TLS 1.3 it should be near 27 milliseconds.
• Server processing: 341.3 minus 98.9 is 242.4 milliseconds. That is 70 percent of the total.
• Body transfer: 344.0 minus 341.3 is 2.7 milliseconds.

The conclusion is clear. The bottleneck of this request is server processing time, not the network. That said, moving to TLS 1.3 would cut another 40 milliseconds, and reusing the connection would remove 95 milliseconds outright.

To measure repeatedly and look at the distribution, use this.

for i in $(seq 1 20); do
  curl -s -o /dev/null -w '%{time_connect} %{time_appconnect} %{time_starttransfer} %{time_total}\n' \
    https://api.example.com/v1/orders
done | sort -k4 -n | tail -5

0.028112 0.094221 0.298104 0.301002
0.029004 0.096118 0.312880 0.315442
0.031104 0.098742 0.341285 0.343990
0.030221 0.097009 0.512033 0.515118
0.029880 0.095442 1.204118 1.207002

If the first two columns are stable and only the third one jumps, that is server-side tail latency, not the network. Conversely, if time_connect jumps, that is when it is time to look at the network.

Sizing the connection pool

Pool size is not a value you set by feel. Calculate it with Little's law.

The number of requests that must be in flight simultaneously is throughput times response time.

concurrent requests = requests per second × average response time

e.g.) one instance at 500 requests per second, average response 80 ms
      500 × 0.08 = 40

      leave 1.5x headroom for tail latency
      recommended pool size = 60

Two upper bounds have to be considered alongside this.

First, the total connection count. With 40 application instances and a pool of 60 each, the backend receives up to 2400 connections. The file descriptor limit and maximum connection setting on the backend have to absorb that. It is especially dangerous for a database. If Postgres max_connections is 200 while the application pools total 2400, a connection stampede right after a deploy takes the service down.

Second, the cost of holding idle connections. Make the pool large and most of it stays idle, and those connections become candidates for the race condition we saw earlier. There is no reason to size it larger than necessary.

It is worth knowing the defaults of the major runtimes too. In Go, http.DefaultTransport has MaxIdleConns at 100, but MaxIdleConnsPerHost, the idle connections per host, is 2. Unless you raise that value, no matter how many requests you send to one backend, only 2 idle connections are kept and the rest are opened fresh every time. This is the default most frequently missed in microservices.

# Values you must tune in Go
# t := http.DefaultTransport.(*http.Transport).Clone()
# t.MaxIdleConns        = 200
# t.MaxIdleConnsPerHost = 100
# t.IdleConnTimeout     = 90 * time.Second

Check the actual reuse rate like this.

# Distribution of connection states from the client host toward a specific backend
ss -tan dst 10.0.3.11 | awk 'NR>1 {print $1}' | sort | uniq -c | sort -rn

     58 ESTAB
      3 TIME-WAIT

If ESTAB holds steady and TIME-WAIT is small, reuse is working well. The opposite shape means a new connection is being opened for every request.

When TIME_WAIT is a problem and when it is not

People are often alarmed to see ss -tan | grep TIME-WAIT | wc -l print tens of thousands. Most of the time it is not a problem. You have to separate precisely when it is.

TIME_WAIT appears only on the side that closed the connection first, that is, the side that did the active close. On Linux it is fixed at 60 seconds and cannot be tuned with sysctl. It has two purposes: preventing an old, delayed segment from mixing into a new connection with the same 4-tuple, and allowing the final ACK to be retransmitted if it was lost.

Start with the cases where it is not a problem.

In a configuration where the server closes the connection after responding, TIME_WAIT accumulates on the server side. But those entries all have the same local port, 443, and differ only in remote address and port. The 4-tuples do not collide, so port exhaustion does not occur. The remaining cost is memory only, on the order of a few hundred bytes per entry. Even 30,000 entries is tens of megabytes. You can ignore it.

The cases where it does matter are different. It is the side that opens outbound connections in bulk. Proxies, API gateways and batch jobs fall into this category. Here the destination IP and port are fixed, so the only thing that distinguishes connections is the local port.

sysctl net.ipv4.ip_local_port_range

net.ipv4.ip_local_port_range = 32768	60999

There are 28232 usable ports and each port is tied up for 60 seconds. In other words, roughly 470 new connections per second toward one destination is the theoretical ceiling. Go past it and you get this error.

dial tcp 10.0.3.11:8080: connect: cannot assign requested address

This EADDRNOTAVAIL is the real signal of port exhaustion. What you should watch is whether this error occurs, not the TIME_WAIT count itself.

# Count TIME-WAIT per destination to see where exhaustion is happening
ss -tan state time-wait | awk 'NR>1 {print $4}' | cut -d: -f1 | sort | uniq -c | sort -rn | head

  26104 10.0.3.11
    412 10.0.3.12
     88 169.254.169.254

Some widely circulated bad prescriptions need sorting out here.

net.ipv4.tcp_tw_recycle must not be used. It had a serious problem where connections were randomly refused when multiple clients behind NAT sent different timestamps, and it was removed outright in Linux 4.12. Plenty of blogs still recommend this value.

net.ipv4.tcp_tw_reuse is conditionally valid. It applies only to outbound connections and requires the timestamp option to be enabled on both sides. It has no effect at all on TIME_WAIT accumulated on the server side.

# Meaningful only on a host with heavy outbound traffic
sudo sysctl -w net.ipv4.tcp_tw_reuse=1
sudo sysctl -w net.ipv4.ip_local_port_range="10240 65535"

Widening the port range raises the ceiling to about 920. But all of these are symptom relief.

The real solution is to reuse connections and reduce the number of new connections themselves. You should first ask why 470 new connections per second are needed at all. Chances are keepalive is off, there is no pool, or a client object is being created fresh on every request. Code that calls requests.get() directly in Python requests is the classic example. Create a Session object and reuse it, and this problem disappears wholesale.

Closing — line up the order and reuse

It comes down to three sentences.

If you have intermittent 502s and the backend log is empty, suspect a connection reuse race and check the timeout order. The idle timeout of the side that sends requests must be the shortest and the receiving side the longest. Setting the backend keepalive comfortably longer than the load balancer idle timeout is the concrete application of that principle.

When reporting latency, always split it into phases. With just the four points from curl — namelookup, connect, appconnect and starttransfer — you immediately separate a DNS problem from a distance problem from a TLS configuration problem from a server processing problem. Performance discussions held without these four numbers are mostly guesswork.

And before you get alarmed by the TIME_WAIT count, check whether cannot assign requested address actually occurs. Reusing connections is always better than touching sysctl. The cost of opening a connection can exceed half of the response time in long-distance communication, and removing that cost has a larger effect than most application optimisation.