필사 모드: dig works but the application fails — start by checking the DNS resolution order

Introduction — dig works, but only the application cannot find the name

There is one combination that confuses people more than anything else during an incident.

dig +short api.internal.example.com

10.20.4.31

DNS looks healthy. Yet on the same host the application dies like this.

java.net.UnknownHostException: api.internal.example.com

At this point most people jump to the conclusion that "the DNS server is fine, so something is wrong with the application", and start digging through application code and libraries. That direction is almost always wasted effort.

The cause is simple. dig and the application resolve names along different paths to begin with. The fact that dig succeeded is not evidence that the application will succeed. The reverse holds too: dig can fail while the application runs perfectly well.

This post walks the resolution path the application actually takes, step by step, and lays out what can go wrong at each stage.

The real path an application takes to resolve a name

C, Python, Ruby, PHP, the default resolver in Node, and even Java on Linux mostly end up calling glibc getaddrinfo(). glibc then walks the following order.

1. It reads the hosts line in /etc/nsswitch.conf to decide which sources to use and in what order.
2. For the files module it looks at /etc/hosts.
3. For the dns module it reads /etc/resolv.conf, applies nameserver, search and options, and sends the query.
4. For the resolve module it asks systemd-resolved over D-Bus.
5. Special modules such as myhostname or mdns4_minimal may sit somewhere in the middle.

Start with the first step.

grep '^hosts' /etc/nsswitch.conf

hosts: files mdns4_minimal [NOTFOUND=return] dns myhostname

This single line contains two traps.

mdns4_minimal [NOTFOUND=return] means that when a name ends in .local it is resolved via mDNS only, and if that fails, resolution stops right there. It never reaches the dns module that follows. Organisations that name internal domains something like cluster.local or svc.local fall into this trap often. dig does not consult nsswitch, so it returns a normal answer while only the application fails.

The other trap is the resolve module.

hosts: files resolve [!UNAVAIL=return] dns myhostname

With this configuration the application asks systemd-resolved over D-Bus. It does not use the nameserver values in resolv.conf directly. In other words, editing /etc/resolv.conf may not change application behaviour at all.

Check resolv.conf as well.

cat /etc/resolv.conf

nameserver 127.0.0.53
options edns0 trust-ad
search ap-northeast-2.compute.internal

dig and nslookup do not look at /etc/hosts

This is the most practical fact in the whole post. dig, nslookup and host are DNS-protocol-only tools. They do not consult nsswitch.conf, they do not consult /etc/hosts, and they do not consult the per-link configuration of systemd-resolved. They take only the nameserver address out of resolv.conf and fire a query at UDP 53.

That is why the following holds without any contradiction.

# A name that exists only in /etc/hosts — dig cannot find it
dig +short legacy-billing.internal

(empty output)

# The same path as the application — it resolves normally
getent hosts legacy-billing.internal

10.20.9.14      legacy-billing.internal

The reverse direction happens too. If a stale IP that someone put into /etc/hosts while debugging is still sitting there, dig returns the new IP while the application keeps connecting to the old one. In that case the symptom is not a DNS error but connection refused or a timeout, which keeps you from looking for the cause in DNS at all.

So the basic diagnostic tool has to change.

# Reproduce exactly the same path the application takes
getent hosts api.internal.example.com

# Show both IPv4 and IPv6, and the order they will be tried in
getent ahosts api.internal.example.com

10.20.4.31       STREAM api.internal.example.com
10.20.4.31       DGRAM
10.20.4.31       RAW

The output order of getent ahosts is the same order the application will actually try. If AAAA comes first but there is no IPv6 route, that means the first connection attempt will fail, and this becomes the cause of the "Network is unreachable" mentioned earlier.

To put it simply: dig is the tool for seeing the DNS server answer, and getent is the tool for seeing the answer the application will receive. When the two results differ, that difference itself tells you where the cause lives.

The confusion created by systemd-resolved and 127.0.0.53

On Ubuntu and other recent distributions, the nameserver in resolv.conf is usually 127.0.0.53. This is a stub resolver run by systemd-resolved, and the real upstream servers are somewhere else.

resolvectl status

Global
         Protocols: -LLMNR -mDNS +DNSOverTLS DNSSEC=no/unsupported
  resolv.conf mode: stub

Link 2 (ens5)
    Current Scopes: DNS
         Protocols: +DefaultRoute +DNSSEC=no/unsupported
Current DNS Server: 10.0.0.2
       DNS Servers: 10.0.0.2
        DNS Domain: ap-northeast-2.compute.internal

Link 5 (tun0)
    Current Scopes: DNS
         Protocols: +DNSOverTLS
Current DNS Server: 10.60.0.53
       DNS Servers: 10.60.0.53
        DNS Domain: ~corp.example.com ~internal.example.com

Two things matter here.

First, each link has its own DNS servers and domains. In the example above, names ending in corp.example.com go to 10.60.0.53 on the VPN interface, and everything else goes to 10.0.0.2. This is called split DNS. But if you specify an upstream server directly, as in dig @10.0.0.2 vault.corp.example.com, you bypass that routing and the query fails. The "dig fails but the browser opens the page" situation while the VPN is up comes from exactly this.

Second, the default target of dig is 127.0.0.53, so you are looking at the stub cache. Even after the upstream server has been fixed, the stub cache can keep handing you the old answer.

Checking the three layers separately is the accurate approach.

# 1) Exactly the application path
resolvectl query vault.corp.example.com

# 2) Straight to the stub resolver
dig @127.0.0.53 vault.corp.example.com

# 3) Straight to the upstream server
dig @10.60.0.53 vault.corp.example.com

vault.corp.example.com: 10.60.4.7                    -- link: tun0

-- Information acquired via protocol DNS in 2.1ms.
-- Data is authenticated: no; Data was acquired via local or encrypted transport: yes

What makes resolvectl query useful is that it tells you the link name too. Which interface the DNS query went out on is right there in the output.

If you suspect the cache, flush it layer by layer.

resolvectl flush-caches
resolvectl statistics

DNSSEC verdicts
Secure: 0
Insecure: 0
Bogus: 0
Indeterminate: 0

Transactions
Current Transactions: 0
Total Transactions: 184213

Cache
Current Cache Size: 0
Cache Hits: 156402
Cache Misses: 27811

The cache layers do not end there. Applications have their own internal caches. Java is the classic example. In a default environment with no security manager installed, the JVM caches successful resolutions for 30 seconds and caches failed resolutions for the networkaddress.cache.negative.ttl default of 10 seconds. In older deployments that do use a security manager, successful results are cached forever. If an IP changed during failover and only one particular service keeps clinging to the old IP, this is the value to look at first.

# Specify explicitly when launching Java
java -Dsun.net.inetaddr.ttl=30 -Dsun.net.inetaddr.negative.ttl=0 -jar app.jar

# Or in the java.security file
# networkaddress.cache.ttl=30
# networkaddress.cache.negative.ttl=0

To sum up, there are four cache layers: inside the application, the stub resolver, the recursive resolver, and the TTL handed down by the authoritative server. Unless you are explicit about which layer you flushed, you will keep getting reports of "I cleared the cache and nothing changed".

search domains and ndots — the real reason Kubernetes gets slow

The resolv.conf inside a Kubernetes pod looks like this.

kubectl exec -it deploy/api -- cat /etc/resolv.conf

search prod.svc.cluster.local svc.cluster.local cluster.local ap-northeast-2.compute.internal
nameserver 10.96.0.10
options ndots:5

ndots:5 means "if the name being queried has fewer than 5 dots, try the search list first, before trying it as an absolute name". That setting is what lets you call services from inside a pod with short names like api or api.prod.

The problem is external domains. api.stripe.com has 2 dots, which is fewer than 5. So the queries go out in the following order.

kubectl exec -it deploy/api -- sh -c 'tcpdump -nn -i any port 53' &
kubectl exec -it deploy/api -- curl -sS -o /dev/null https://api.stripe.com/v1/charges

10:11:02.512331 IP 10.244.1.9.41822 > 10.96.0.10.53: 41955+ A? api.stripe.com.prod.svc.cluster.local. (55)
10:11:02.512402 IP 10.244.1.9.41822 > 10.96.0.10.53: 8271+ AAAA? api.stripe.com.prod.svc.cluster.local. (55)
10:11:02.513880 IP 10.96.0.10.53 > 10.244.1.9.41822: 41955 NXDomain 0/1/0 (148)
10:11:02.513991 IP 10.96.0.10.53 > 10.244.1.9.41822: 8271 NXDomain 0/1/0 (148)
10:11:02.514552 IP 10.244.1.9.55110 > 10.96.0.10.53: 12043+ A? api.stripe.com.svc.cluster.local. (50)
10:11:02.514601 IP 10.244.1.9.55110 > 10.96.0.10.53: 39218+ AAAA? api.stripe.com.svc.cluster.local. (50)
10:11:02.516001 IP 10.96.0.10.53 > 10.244.1.9.55110: 12043 NXDomain 0/1/0 (143)
10:11:02.516120 IP 10.96.0.10.53 > 10.244.1.9.55110: 39218 NXDomain 0/1/0 (143)
10:11:02.516644 IP 10.244.1.9.39004 > 10.96.0.10.53: 55129+ A? api.stripe.com.cluster.local. (46)
10:11:02.516702 IP 10.244.1.9.39004 > 10.96.0.10.53: 21008+ AAAA? api.stripe.com.cluster.local. (46)
10:11:02.518233 IP 10.96.0.10.53 > 10.244.1.9.39004: 55129 NXDomain 0/1/0 (139)
10:11:02.518344 IP 10.96.0.10.53 > 10.244.1.9.39004: 21008 NXDomain 0/1/0 (139)
10:11:02.518881 IP 10.244.1.9.60122 > 10.96.0.10.53: 3311+ A? api.stripe.com.ap-northeast-2.compute.internal. (64)
10:11:02.518944 IP 10.244.1.9.60122 > 10.96.0.10.53: 61190+ AAAA? api.stripe.com.ap-northeast-2.compute.internal. (64)
10:11:02.523115 IP 10.96.0.10.53 > 10.244.1.9.60122: 3311 NXDomain 0/1/0 (157)
10:11:02.523240 IP 10.96.0.10.53 > 10.244.1.9.60122: 61190 NXDomain 0/1/0 (157)
10:11:02.523774 IP 10.244.1.9.44911 > 10.96.0.10.53: 9042+ A? api.stripe.com. (32)
10:11:02.523821 IP 10.244.1.9.44911 > 10.96.0.10.53: 55403+ AAAA? api.stripe.com. (32)
10:11:02.531002 IP 10.96.0.10.53 > 10.244.1.9.44911: 9042 3/0/0 A 34.111.87.9 ... (98)

Ten queries went out to resolve a single external domain, and eight of them went out only to collect an NXDOMAIN. When it is not one pod but hundreds doing this dozens of times per second, most of the load on CoreDNS consists of queries that are guaranteed to fail. And if there is even a little UDP packet loss, every lost query picks up a 5-second retry timeout and the latency spikes become visible.

There is a widely circulated prescription that needs correcting here. The advice "just lower ndots to 1" breaks the cluster. With ndots at 1, any name with one or more dots is queried as an absolute name first, without the search list. That breaks every cross-namespace call written with a namespace suffix, such as payments.prod, because with one dot the search list no longer applies.

The safe lower bound is 2. With ndots at 2, api and payments.prod still go through search, while external domains with two or more dots are queried immediately as absolute names.

spec:
  dnsConfig:
    options:
      - name: ndots
        value: '2'

If you can change the application, there is an even more definitive method. Adding a trailing dot to an external domain makes it an absolute name and skips the entire search list.

# That one trailing dot removes four search domains
curl -sS -o /dev/null -w '%{time_namelookup}\n' https://api.stripe.com./v1/charges

Running NodeLocal DNSCache is another good mitigation. Failed queries terminate at the node-local cache instead of reaching CoreDNS, and the 5-second retries caused by UDP loss drop sharply.

NXDOMAIN, SERVFAIL, and the empty NOERROR

Reporting response codes as a blurred "DNS is broken" narrows nothing down. You have to check the status field in the dig output.

dig api.internal.example.com A

;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 51204
;; flags: qr rd ra; QUERY: 1, ANSWER: 0, AUTHORITY: 1, ADDITIONAL: 1

ANSWER is 0, yet the status is NOERROR. This combination is the one most often misdiagnosed.

The classic situation that produces an empty NOERROR is IPv6. When only the AAAA record is missing and the application looks up AAAA first, the log can end up saying "name not found". Confirm it by naming the query type explicitly.

dig +noall +answer api.internal.example.com A AAAA CNAME

api.internal.example.com. 300 IN CNAME api-lb.internal.example.com.
api-lb.internal.example.com. 60 IN A 10.20.4.31

SERVFAIL is a problem in the resolver itself. There is a quick way to separate out whether DNSSEC validation failure is the cause.

# Ask again with validation off. If it succeeds now, it is a DNSSEC problem
dig +cd api.internal.example.com

# Follow the delegation path down to the authoritative server
dig +trace api.internal.example.com

When you see an NXDOMAIN, read the queried name exactly as written. If tcpdump shows an NXDOMAIN for a name with a search domain appended, such as api.stripe.com.prod.svc.cluster.local, that is normal behaviour and not the cause of your problem.

512 bytes, TCP fallback, and the case where only large responses fail

DNS originally could not send a response larger than 512 bytes over UDP. If it went over, the server set the TC (truncated) flag and the client resent the same query over TCP 53. EDNS0 is the extension that reduces this round trip by letting the client announce that it can accept a larger UDP buffer.

# Turn EDNS0 off and constrain it to 512 bytes
dig +notcp +bufsize=512 TXT _dmarc-report.example.com

;; Truncated, retrying in TCP mode.
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 40912
;; flags: qr tc rd ra; QUERY: 1, ANSWER: 8, AUTHORITY: 0, ADDITIONAL: 0

Seeing tc in the flags is the evidence of truncation.

This is where a real-world incident is born. Older firewall configurations frequently open UDP 53 only and leave TCP 53 blocked. Normally there is no problem at all, because the responses are small. But the moment records grow or DNSSEC signatures push a response past 512 bytes, that one name stops resolving. Reports of "only this one domain does not work" sometimes have exactly this cause.

# Check directly whether TCP 53 is open
dig +tcp @10.0.0.2 example.com

# Check whether the EDNS0 path is blocked by a middlebox
dig +bufsize=4096 @10.0.0.2 dnssec-failed.org

# Check the response size itself
dig +noall +stats example.com

;; Query time: 12 msec
;; SERVER: 10.0.0.2#53(10.0.0.2) (UDP)
;; MSG SIZE  rcvd: 512

If MSG SIZE rcvd is pinned right at 512 or near 1232 and only that one name fails intermittently, you have enough grounds to suspect a size limit on the path. For reference, modern resolvers lower the advertised EDNS buffer value to 1232 bytes to avoid IP fragmentation.

Closing — name resolution does not end with dig

There is one sentence to remember. dig shows you the DNS server answer and getent shows you the answer the application will receive. If the two results differ, that difference is the location of the cause.

In practice, narrowing it down in the following order usually finishes the job in a few minutes. First run getent hosts and dig +short side by side and see whether the results diverge. If they do, check the hosts line in nsswitch.conf, /etc/hosts, and the per-link configuration of systemd-resolved. If they agree, the problem is upstream of the resolver, so use the status field in dig to separate NXDOMAIN, SERVFAIL and the empty NOERROR, and follow the delegation path with +trace if needed. If the answer is right but takes too long, count the queries that actually go out with tcpdump. The unnecessary queries generated by search domains and ndots show up there directly.

A large share of DNS incidents originate not in the server but in the client-side resolution order. Check the order first and you will not need to dig through the server at all.