필사 모드: Connection refused vs. timeout — narrowing the cause at the TCP level

Introduction — the same "connection failure", two kinds of message

Right after a deploy the health check failed. One pod says this.

dial tcp 10.0.3.11:8080: connect: connection refused

The pod next to it says this.

dial tcp 10.0.3.11:8080: i/o timeout

Many teams lump the two together as "the connection does not work either way". And in both cases they start by digging through security groups and firewall rules. That is where the most time gets wasted. At the TCP level the two messages state opposite facts, which means the places you should look are opposite as well.

This post covers which packet exchange each error is the result of, and how far that single fact lets you cut down the list of candidate causes. Commands and outputs are shown in their real form so you can copy and compare them directly.

Connection refused means you received an RST

There is exactly one condition under which the connect() system call returns ECONNREFUSED: the peer answered the SYN you sent with a TCP segment that has the RST flag set.

When a SYN arrives at a port with no listener, the kernel builds and sends an RST immediately, independently of any application. This is behaviour the TCP specification requires, and it keeps happening even when the process is dead, as long as the kernel is alive.

# A port with no listener — this finishes within one round trip
time nc -vz 10.0.3.11 8080

nc: connect to 10.0.3.11 port 8080 (tcp) failed: Connection refused

real    0m0.004s
user    0m0.001s
sys     0m0.002s

What matters here is not the failure itself but the elapsed time. Four milliseconds is a round trip. My SYN travelled to 10.0.3.11, and that host built an answer and sent it back to me. In other words, this is proof that the packet already passed every gate on the path: routing, security groups, network policy and NAT.

That leaves a set of candidate causes narrowed down to the inside of the destination host.

• The process is dead, or has not started yet.
• The process is alive but bound to a different port.
• The process is bound to 127.0.0.1 rather than 0.0.0.0.
• The destination is a load balancer or kube-proxy with no healthy backend behind it.

One common misunderstanding needs correcting here. The explanation "connection refused means a firewall blocked it" is wrong most of the time. Firewalls used in practice are almost always DROP policies. AWS security groups, Kubernetes NetworkPolicy and cloud firewalls silently discard traffic that no rule allows. A discarded packet produces no reply, so it shows up as a timeout, not as refused. Digging through firewalls because you saw refused means re-checking a gate the packet already passed.

The exception is a configuration that was explicitly told to answer with a rejection.

# This rule returns an RST, so it looks like refused
sudo iptables -A INPUT -p tcp --dport 8080 -j REJECT --reject-with tcp-reset

# This rule sends no reply, so it becomes a timeout
sudo iptables -A INPUT -p tcp --dport 8080 -j DROP

In an environment where you drive local iptables/nftables yourself, all you need is one check for the presence of a REJECT rule. Beyond that, treating refused as a process problem is almost always the right call.

Timeout means nobody answered the SYN

A timeout is a failure that carries far less information. I sent a SYN, retransmitted it the configured number of times, and during that whole window no SYN-ACK, no RST and no ICMP came back. The only fact available to the kernel is "there is no answer".

# A port where packets are silently discarded — this burns through the kernel default retries
time nc -vz 10.0.3.11 9090

nc: connect to 10.0.3.11 port 9090 (tcp) failed: Connection timed out

real    2m7.213s
user    0m0.002s
sys     0m0.003s

We will work out where the number two minutes and seven seconds comes from later. What matters right now is the candidate causes.

• A firewall or security group on the path is discarding the packets.
• The destination IP is wrong, and no host uses that address at all.
• Routing is wrong and the packet went somewhere else entirely.
• The return path is asymmetric, so the SYN-ACK cannot get back to me.
• The destination is alive and a listener exists, but the accept queue is overflowing and SYNs are being dropped.

That last item is the one people miss most often.

When a listener exists and you still get a timeout

Connections whose handshake has completed pile up in the accept queue, and the application pulls them out with accept(). If the application is slow and that queue fills up, Linux in its default configuration silently drops new SYNs. From the client side this is indistinguishable from a closed port, except that it appears as a timeout rather than as refused.

ss -ltn 'sport = :8080'

State   Recv-Q  Send-Q   Local Address:Port   Peer Address:Port
LISTEN  129     128            0.0.0.0:8080         0.0.0.0:*

On a listening socket, Recv-Q is the current accept queue length and Send-Q is the maximum size of that queue. The fact that 129 has gone past 128 means the queue is already saturated. Cumulative counters make it even clearer.

nstat -az | grep -E 'ListenOverflows|ListenDrops'

TcpExtListenOverflows           18437              0.0
TcpExtListenDrops               18437              0.0

If these values are climbing, the cause is application throughput, not the network. No amount of firewall archaeology will turn it up.

What the remaining error messages point at

There are in fact four more failures in the connect family. Each arises at a different layer, so reading the message accurately gets you half the way there.

Two points deserve emphasis.

First, Network is unreachable is the state where the kernel routing table has no route, so the packet never even left the interface. These days the overwhelming majority of this error is IPv6. If DNS returned an AAAA record but the host has no IPv6 default route, the application tries IPv6 first and fails instantly.

ip -6 route show default
# If the output is empty, no connection can be attempted with an AAAA result

getent ahosts api.example.com

2606:4700:4700::1111 STREAM api.example.com
1.1.1.1              STREAM

Second, Name or service not known is not a TCP problem. Not a single packet went out to the target server. In this case, finding nothing at all in the firewall logs is the correct outcome.

Seeing SYN and RST with your own eyes using tcpdump

When reasoning is not conclusive, looking at the packets directly is the fastest route. You only need the handshake, so keep the filter narrow.

sudo tcpdump -nn -i any "host 10.0.3.11 and tcp port 8080"

This is the output in the refused case.

14:02:11.104512 IP 10.0.2.7.54312 > 10.0.3.11.8080: Flags [S], seq 1829301744, win 62727, options [mss 8961,sackOK,TS val 913442 ecr 0,nop,wscale 7], length 0
14:02:11.104698 IP 10.0.3.11.8080 > 10.0.2.7.54312: Flags [R.], seq 0, ack 1829301745, win 0, length 0

Two lines and you are done. [S] went out and [R.] came back. An answer within 0.2 milliseconds means the same host or something very close to it.

The timeout case looks like this.

14:05:31.220114 IP 10.0.2.7.54390 > 10.0.3.11.9090: Flags [S], seq 402113877, win 62727, options [mss 8961,sackOK,TS val 1113209 ecr 0,nop,wscale 7], length 0
14:05:32.235880 IP 10.0.2.7.54390 > 10.0.3.11.9090: Flags [S], seq 402113877, win 62727, options [mss 8961,sackOK,TS val 1114225 ecr 0,nop,wscale 7], length 0
14:05:34.251874 IP 10.0.2.7.54390 > 10.0.3.11.9090: Flags [S], seq 402113877, win 62727, options [mss 8961,sackOK,TS val 1116241 ecr 0,nop,wscale 7], length 0
14:05:38.315876 IP 10.0.2.7.54390 > 10.0.3.11.9090: Flags [S], seq 402113877, win 62727, options [mss 8961,sackOK,TS val 1120305 ecr 0,nop,wscale 7], length 0

The same SYN, with the same sequence number, repeats at 1-second, 2-second and 4-second intervals. There is no reply line at all.

In an environment where you have no packet capture privileges, socket state alone is enough to tell them apart.

ss -tanp state syn-sent

Recv-Q  Send-Q   Local Address:Port    Peer Address:Port   Process
0       1             10.0.2.7:54390       10.0.3.11:9090   users:(("curl",pid=31544,fd=5))

If a socket sits in SYN-SENT for more than a few seconds with Send-Q at 1, there is one SYN outstanding that never got an answer. In the refused case this state passes too quickly for the eye to catch.

Predicting the timeout length from the SYN retransmission schedule

The two minutes and seven seconds above is not an accidental number. Linux retransmits the SYN as many times as net.ipv4.tcp_syn_retries says, doubling the interval each time.

sysctl net.ipv4.tcp_syn_retries

net.ipv4.tcp_syn_retries = 6

After the initial transmission it retransmits six times at 1, 2, 4, 8, 16 and 32 second intervals, then waits a further 64 seconds after the last retransmission before giving up. The sum is 1 + 2 + 4 + 8 + 16 + 32 + 64 = 127 seconds. That matches the measured two minutes and seven seconds exactly.

This arithmetic is useful in practice for two reasons.

One, a failure that cuts off at almost exactly 127 seconds means the kernel gave up on the SYN, which in turn means there was no answer at all. If the timeout value in your application log says 30 seconds but the failure actually took 127 seconds, that is also a signal that the timeout setting is not being applied.

Two, if you want a shorter connect timeout, specifying it per application or per socket is safer than changing the kernel value.

# Applies to this entire host — 3 retransmissions, about 15 seconds
sudo sysctl -w net.ipv4.tcp_syn_retries=3

# Limiting it per tool has fewer side effects
curl --connect-timeout 3 -sS http://10.0.3.11:8080/health
nc -w 3 -vz 10.0.3.11 8080

macOS behaves differently here. The default connect timeout is roughly 75 seconds and the sysctl name differs too, so you must not read a time measured on your local Mac as if it applied to a Linux server.

The 127.0.0.1 binding accident in containers and Kubernetes

Half of all connection refused cases in container environments have the same cause: the process is bound only to loopback.

In local development it comes up on 127.0.0.1:8080 and connects fine. Drop the same thing into a container, attach a Service, and the pod is Running and the process is alive, yet reaching it from another pod gives connection refused.

The reason lies in the destination address. When kube-proxy DNATs a Service ClusterIP to a pod IP, the destination address of the arriving packet is a pod IP such as 10.244.1.23. The socket is open only on 127.0.0.1, so the kernel finds no listener matching that destination and returns an RST.

kubectl exec -it deploy/api -- ss -ltnp

State   Recv-Q  Send-Q   Local Address:Port   Peer Address:Port  Process
LISTEN  0       4096        127.0.0.1:8080         0.0.0.0:*      users:(("api",pid=1,fd=7))

A healthy result should look like this.

State   Recv-Q  Send-Q   Local Address:Port   Peer Address:Port  Process
LISTEN  0       4096          0.0.0.0:8080         0.0.0.0:*      users:(("api",pid=1,fd=7))
LISTEN  0       4096             [::]:8080            [::]:*      users:(("api",pid=1,fd=8))

What drags this incident out is that the health check succeeds from inside the pod. Go in with kubectl exec and run curl localhost:8080 and it works. Naturally so, because the destination address is 127.0.0.1. You must check with the pod IP.

POD_IP=$(kubectl get pod -l app=api -o jsonpath='{.items[0].status.podIP}')
kubectl exec -it deploy/api -- curl -sS --connect-timeout 2 "http://${POD_IP}:8080/health"

curl: (7) Failed to connect to 10.244.1.23 port 8080 after 1 ms: Connection refused

The place to fix it is the application configuration. In Node, Express with app.listen(8080) attaches to every address, but app.listen(8080, '127.0.0.1') does not. In Go, http.ListenAndServe(":8080", ...) is right and "localhost:8080" is wrong. Spring Boot has the server.address property, Rails takes -b 0.0.0.0, Gunicorn takes --bind 0.0.0.0:8000, and Postgres has the listen_addresses setting. Cases where a Docker port mapping is in place and the connection still fails have the same root cause.

Turn the same logic around and it becomes a diagnostic. If the pod IP gives refused while localhost inside the pod succeeds, the cause is the bind address, not network policy. Conversely, if even the pod IP times out, that is when it is time to look at NetworkPolicy and the CNI.

There is one more case worth remembering, where refused only appears when going through the Service. If Endpoints is empty, kube-proxy has nowhere to send and refuses immediately.

kubectl get endpoints api

NAME   ENDPOINTS   AGE
api    <none>      12m

In that situation either the Service selector does not match the pod labels, or the readinessProbe is failing and not a single pod is ready.

Closing — refused is not bad news

To sum up, the skeleton of the judgement is short.

If you saw Connection refused, a packet travelled to the destination and came back. The path is already verified, so cross firewalls and routing off the list and look at the process on the target host, the bind address, the port number and whether any backend exists. That means the problem has shrunk to the inside of a single host, which is a welcome signal.

If you saw a timeout, no information came back at all. Keep firewalls, security groups, routing, asymmetric paths and accept queue overflow all open as candidates, and start by using tcpdump to confirm whether the SYN actually goes out and whether retransmissions repeat. Remembering the number 127 seconds here lets you tell instantly whether the kernel gave up or whether an application timeout fired first.

The moment you blur the two messages together into "connection failure", diagnosis time multiplies. Read that one line accurately instead, and more than half of the places you would have had to check disappear from the start.