Skip to content

필사 모드: The Complete Guide to DNS: Following a Name All the Way to an Address

English
0%
정확도 0%
💡 왼쪽 원문을 읽으면서 오른쪽에 따라 써보세요. Tab 키로 힌트를 받을 수 있습니다.

Introduction

The moment somebody says "looks like a DNS problem" in an incident call, the investigation usually stops right there. DNS is not one system: it is five segments of very different character, chained together. A diagnosis that does not name the segment is not a diagnosis.

This blog already has DNS Deep Dive and Debugging DNS Resolution Order. The first covers protocol structure, the second covers the resolution order on a Linux host. This guide fills the gap between them. It is written for the operator who has to change a record, wait for propagation, flush a cache, and work out which segment is guilty — procedures and decision criteria.

Commands assume BIND 9 dig, the glibc stub resolver, and systemd-resolved. Every option here was checked against the manual pages and RFC text, and the sources are listed at the end with URLs and the date they were checked.

dig -v
resolvectl status

1. The resolution path — the five segments a query crosses

Turning one name into one address crosses the following five segments. Each fails for different reasons and with different symptoms.

  1. The application and the stub resolver. When a program calls a name resolution function, the system stub resolver handles it. RFC 8499 defines a stub resolver as "a resolver that cannot perform all resolution itself. Stub resolvers generally depend on a recursive resolver to undertake the actual resolution function." /etc/hosts and the name service switch configuration also sit in this segment.
  2. Resolver selection. /etc/resolv.conf decides which server the query goes to and which search domains get appended.
  3. The recursive resolver cache. If the answer is cached, it ends here. Most queries end in this segment — which is exactly why most change-related incidents happen here too.
  4. Following the delegation. On a cache miss the resolver starts at the root, passes through the TLD, and walks down to the authoritative server. RFC 1034 describes this as looking "for locally-available name server RRs, starting at SNAME, then the parent domain name of SNAME, the grandparent, and so on toward the root," falling back to a "safety belt" configuration containing the root servers.
  5. The authoritative answer. RFC 8499 defines an authoritative server as "a server that knows the content of a DNS zone from local knowledge, and thus can answer queries about that zone without needing to query other servers."

The reason to split the path this way is that each segment can be questioned separately.

getent hosts api.internal.example.com
cat /etc/resolv.conf
resolvectl status
dig @127.0.0.53 api.internal.example.com
dig @10.0.0.53 api.internal.example.com
dig @ns1.example.com api.internal.example.com

getent goes through the name service switch; dig does not. If those two disagree, the problem is not DNS — it is the stub resolver segment.

The directives in /etc/resolv.conf are as follows. The values are from the manual page.

  • nameserver — the address of a server to query. Only three are used.
  • search — the list of domains appended to short names. Up to 6 domains and 256 characters total in glibc 2.25 and earlier; unlimited since 2.26.
  • options ndots:n — the threshold for how many dots a name must contain before an initial absolute query is made instead of appending the search domains. Default 1, capped at 15.
  • options timeout:n — how long to wait for a response. Default 5 seconds, capped at 30.
  • options attempts:n — how many times to query before giving up. Default 2, capped at 5.
  • options rotate — round-robin selection among the listed name servers.
  • options single-request — performs the IPv6 and IPv4 requests sequentially instead of in parallel.

Two calculations fall out of this. First, if the first name server is dead, users feel the default five seconds directly. Unless you shorten the timeout and the attempt count, your redundancy exists on paper only. Second, a high ndots means even names containing a few dots get the search domains tried first, so resolving one external name is preceded by several failing queries. That is the classic reason name resolution is slow inside containers.

You can tell whether the search list is involved by splitting it out.

dig +search api
dig +nosearch api
dig api.internal.example.com.

As in that last line, a trailing dot makes the name absolute and skips the search list. Writing external domains with a trailing dot in application configuration removes the wasted queries.


2. Record types — what goes where

TypeWhat it holdsThe constraint that bites in operations
A / AAAAIPv4 / IPv6 addressThe basics. With both present the client chooses
CNAMEAn alias to another nameCannot coexist with other data at the same name
MXMail receiving serverThe target must not be an alias
NSAuthoritative servers for a subzoneThe target must not be an alias
SOAZone authority and the negative cache TTLMust exist at the zone apex
TXTFree textOwnership proofs and mail policy ride on it
PTRAddress to nameLives under in-addr.arpa / ip6.arpa
SRVHost and port of a serviceThe target must not be an alias
CAACAs allowed to issue certificatesLocated by climbing the tree upward

Several of these are worth quoting the source for.

A CNAME has to be alone. RFC 1034 states flatly that "if a CNAME RR is present at a node, no other data should be present," so that the data for a canonical name and its aliases cannot diverge. And RFC 8499 defines the zone apex as "the point in the tree at an owner of an SOA and corresponding authoritative NS RRset." The apex necessarily has an SOA and NS records, so the apex cannot hold a CNAME. That is the exact basis for the wall you hit trying to point a root domain at a CDN.

NS and MX targets must not be aliases. RFC 2181 rules that "the domain name used as the value of a NS resource record, or part of the value of a MX resource record must not be an alias." SRV is the same: RFC 2782 requires of the target that "there MUST be one or more address records for this name, the name MUST NOT be an alias." The SRV format is _Service._Proto.Name TTL Class SRV Priority Weight Port Target; clients try the lowest-numbered priority they can reach, and among equal priorities selection is proportional to weight.

CAA is found by climbing upward. RFC 8659 defines the search as one that "climbs the DNS name tree from the specified label up to, but not including, the DNS root until a CAA RRset is found." The tags are issue, issuewild, and iodef, and if no tag restricts issuance, CAA does not restrict issuance. This is where "we never put a CAA on that subdomain, yet issuance was blocked" comes from.

There are size limits worth remembering too. RFC 1035 set the ceilings at 63 octets for labels, 255 octets for names, and 512 octets for UDP messages. The 512-octet limit is extended by EDNS(0) in RFC 6891, where the requestor writes the largest UDP payload it can reassemble into the CLASS field of the OPT record. That document offers 4096 octets as a starting point and a fallback around 1280 to 1410 bytes. A response larger than the advertised size sets the TC bit and has to be asked again over TCP.


3. TTL and caching — what "propagation" actually is

DNS has no push. Changes do not spread; caches expire. Understanding that one sentence removes most propagation incidents.

RFC 1034 defines the TTL as "how long a RR can be cached before it should be discarded." RFC 2181 adds the rules that bite in practice. Every record in one RRSet must carry the same TTL, and a server must never send an RRSet whose TTLs are not all equal. TTL is also an unsigned 32-bit value with a maximum of 2147483647, and values received with the most significant bit set are to be treated as zero.

Absence is cached too. RFC 2308 distinguishes two kinds.

  • NXDOMAIN — the response code is Name Error, and the queried domain does not exist.
  • NODATA — the response code is NOERROR but there are no relevant answers in the answer section. The name exists; only that type does not.

The cache lifetime for a negative answer is taken "from the minimum of the SOA.MINIMUM field and SOA's TTL." The same document leaves an operational recommendation: "Values of one to three hours have been found to work well and would make sensible a default. Values exceeding one day have been found to be problematic." When a typo in a record is corrected and lookups keep failing for half a day, this value is usually the reason.

Plan the change around the TTL.

  1. Lower the TTL (say to 300 seconds) at least one old TTL before the planned change.
  2. Wait until the old TTL has fully elapsed. Skip this wait and you leave resolvers whose caches never saw the lowered TTL.
  3. Make the change.
  4. Verify against the authoritative servers and against several resolvers separately.
  5. Once things are stable, restore the original TTL.

The time remaining in a cache is observable.

dig +noall +answer +ttlunits example.com
dig +noall +answer example.com
dig @1.1.1.1 +noall +answer example.com
dig @ns1.example.com +noall +answer example.com

Ask a resolver repeatedly and you can watch the TTL count down. The seconds remaining are exactly the time left before that resolver expires the entry. Ask the authoritative server directly and you always get the configured value.


4. Delegation and name servers — where authority splits

RFC 8499 defines delegation as the process of creating a separate zone by adding an NS RRset in the parent zone for the child origin, and calls the boundary a zone cut. There is one asymmetry here every operator must know. RFC 2181 rules that "the NS records that indicate a zone cut are the property of the child zone created." In other words the NS list held by the parent (the TLD) and the NS list the child zone answers with can differ, and the real authority is the child side.

Two common incidents come out of this.

  • The name servers were changed at the registrar but nothing seems to take effect. What changed is the parent delegation. If the child zone still answers with the old server list, the result depends on what each resolver cached.
  • The name server list contains a server that does not actually answer. The resolver retries when it hits a silent server, so names still resolve but intermittently slowly.

This is also where glue comes from. RFC 1034 says the NS records describing the bottom edge of a zone "are NOT part of the authoritative data of the zone," and that the parent supplies address records for the delegated name servers. When a name server name lives under the very zone it serves, you need its address to query the zone and you need to query the zone to learn its address. The address records the parent hands out break that loop.

The core of diagnosis is asking the parent and the child separately and comparing.

dig NS example.com
dig +trace example.com
dig +nssearch example.com
dig SOA example.com @ns1.example.com
dig SOA example.com @ns2.example.com

In the words of the manual, +trace "toggles tracing of the delegation path from the root name servers," and with +nssearch "dig attempts to find the authoritative name servers for the zone." Use the last two lines to check that every authoritative server reports the same SOA serial. Mismatched serials mean a zone transfer is lagging, and users are getting different answers depending on which server they reached.


5. How to read dig

The basic form is dig @server name type. The manual describes the three slots as "the name or IP address of the name server to query," "the name of the resource record that is to be looked up," and what "indicates what type of query is required."

Output is divided into the header, question, answer, authority and additional sections, plus statistics. Read it in the order status, then flags, then the sections.

The values in status are the response codes from RFC 1035.

  • NOERROR — no error condition. But an empty answer section means NODATA, and the name exists.
  • NXDOMAIN — Name Error. That name does not exist.
  • SERVFAIL — server failure. It does not mean the name is absent, it means the server could not produce an answer. Failure to reach upstream and DNSSEC validation failure are the classic causes.
  • REFUSED — refused. A policy decision not to answer, which is what you get asking a resolver that does not offer recursion to you.

The letters in flags each mean something too.

  • qr — this message is a response.
  • aa — authoritative answer. Without this letter, the answer came from cache.
  • tc — truncated. Ask again over TCP.
  • rd / ra — recursion desired / recursion available.
  • ad — the resolver is signalling that the data passed DNSSEC validation.
  • cd — the query was made with checking disabled.

The combinations worth keeping, organised by purpose:

dig +short A example.com
dig +noall +answer example.com
dig +noall +authority +additional example.com
dig @1.1.1.1 example.com
dig +norecurse @10.0.0.53 example.com
dig -x 93.184.216.34
dig -t MX example.com
dig +dnssec example.com
dig +cd example.com
dig +multiline SOA example.com
dig +tcp example.com
dig +time=2 +tries=1 example.com
dig +stats example.com
dig +yaml example.com

Each option comes straight from the manual. +short "toggles whether a terse answer is provided," +norecurse "disables the RD (recursion desired) bit in the query," +dnssec "requests that DNSSEC records be sent by setting the DNSSEC OK (DO) bit," and +cd "sets the CD (checking disabled) bit in the query." -x "sets simplified reverse lookups, for mapping addresses to names," +time=T and +tries=T set the wait and the retry count, and +subnet "sends an EDNS CLIENT-SUBNET option with the specified IP address."

One note on where +norecurse earns its keep. Query a resolver with recursion off and it answers only from its own cache. Use it to check whether an entry is cached, or to inspect state without warming the cache.

+trace has the opposite hazard. Because +trace queries each step from the root itself, it bypasses your resolver. So "+trace looks right but the application still fails" is common — and in that case the culprit is the resolver segment, not the authoritative server.


6. Common failures — propagation delay, stale caches, CNAME constraints

Propagation delay. As above, propagation is expiry. The "it takes 48 hours" advice is mostly a conservative statement built on stale TTL settings. If you did not lower the TTL before the change, there is nothing to do but wait out the old TTL, and if the authoritative server already returns the new value while users still see the old one, that is correct behaviour.

Stale caches. Caches do not live in one place. At minimum, check each of these.

  • The cache in the internal recursive resolver
  • The local cache on the host (systemd-resolved and the like)
  • The runtime own cache (some runtimes keep their own name cache)

The command differs per layer.

sudo rndc flush
sudo rndc flushname api.example.com
sudo rndc flushtree example.com
sudo rndc dumpdb -cache
resolvectl flush-caches
resolvectl query api.example.com
resolvectl statistics

rndc flush "flushes the server's cache," while flushname "flushes the given name from the view's DNS cache and, if applicable, from the view's nameserver address database, bad server cache, and SERVFAIL cache." flushtree removes "the given name, and all of its subdomains" from the same set. resolvectl flush-caches "flushes all DNS resource record caches the service maintains locally." Try the per-name flush before the whole-cache flush. Throwing the whole cache away spikes upstream queries for the next several minutes.

CNAME constraints. Three of them recur. No CNAME at the apex; no alias as the target of MX, NS or SRV; and no other records alongside a name that has a CNAME. The first two are more dangerous because a violation can appear to work for a while. A specification violation that appears to work surfaces at the latest and worst possible moment.

Mistaking NODATA for NXDOMAIN. If the status is NOERROR and the answer is empty, the name exists. The textbook case is a client trying IPv6 against a name that simply has no AAAA, and deleting and recreating the name achieves precisely nothing.

Blocked TCP. When DNSSEC signatures or record counts make a response large, the TC bit is set and the query is repeated over TCP. If the firewall opened only UDP 53, everything looks fine until one particular name fails. Telling that apart is simple.

dig +tcp example.com
dig +dnssec +tcp example.com

The name resolves but the connection fails. If dig answers correctly, the DNS segment is innocent. What follows is a connectivity problem and needs different tools.


7. DNSSEC — what it guarantees and what it does not

In the words of RFC 4033, DNSSEC provides "data origin authentication and data integrity to the Domain Name System." The same document is explicit about what it does not provide: "DNSSEC is not designed to provide confidentiality, access control lists, or other means of differentiating between inquirers." It goes further, stating that "DNSSEC provides no protection against denial of service attacks," and warns that it creates a new class of denial of service attacks based on cryptographic operations. If you want protection against eavesdropping you need a measure at another layer.

There are four record types.

  • DNSKEY — carries the public keys of the zone.
  • RRSIG — the digital signature over an RRset.
  • DS — sits at the delegation point in the parent zone and points at the child key.
  • NSEC — authenticates the non-existence of a name or a type.

The chain of trust is an alternating sequence of DNSKEY and DS RRsets in which each link vouches for the next, and validation starts from a trust anchor. Validation lands in one of four states: Secure with a trust anchor, a chain of trust, and all signatures verified; Insecure with signed proof that the data is not signed; Bogus when data that should be signed fails to validate; and Indeterminate when no trust anchor says that part of the tree is secure.

What matters to the operator is that failure shows up as SERVFAIL. The zone contents are unchanged, but once the signature validity period passes, lookups fail only on validating resolvers. On non-validating resolvers everything looks fine. You tell them apart by asking with validation off.

dig +dnssec example.com
dig +cd example.com
delv example.com

If adding +cd produces an answer, this is a DNSSEC validation failure. delv is the purpose-built tool the manual describes as "a tool for sending DNS queries and validating the results."

Internal environments hit one more thing. RFC 4033 calls a signed zone with no authentication chain from its delegating parent an "island of security," and explains that such a zone can only be authenticated if its keys can be authenticated out of band. How a validating resolver treats an unsigned internal zone falls under this, so the resolver configuration has to handle internal zones explicitly.


8. Running DNS inside a company

Split horizon. One name answered differently inside and outside. Convenient, but the answer now depends on where the question was asked from, so the first thing to establish on a report is the location of the reporter. Whether they were on the VPN changes the conclusion.

Forwarder or recursion. RFC 8499 defines a forwarder as "a nameserver used to resolve queries instead of directly using the authoritative nameserver chain." If your internal resolver forwards upstream, then when the upstream dies the internal resolver ends up alive and unable to answer anything. That is why the health check has to be a successful query, not a live process.

The reality of resolver redundancy. As above, only three nameserver entries are used, the default timeout is 5 seconds, and the default attempt count is 2. If the first server does not answer, users feel that time. Put the timeout and the attempt count in the standard company resolv.conf, and use rotate to spread the load when it helps.

Change and verification. After changing a zone, check that every authoritative server gives the same answer.

sudo rndc reload
sudo rndc status
sudo rndc querylog on
dig SOA example.com @ns1.example.com +short
dig SOA example.com @ns2.example.com +short
sudo rndc querylog off

rndc reload "reloads the configuration file and zones," whereas rndc reconfig "reloads the configuration file and loads new zones, but does not reload existing zone files even if they have changed." Not knowing that difference produces a recurring "I reloaded, why did nothing change." querylog turns query logging on and off, and on a busy server you turn it on for the investigation window only, not permanently.

Container environments. The resolv.conf the orchestrator injects often carries several search domains and a high ndots. The result is that resolving one external name is preceded by several failing queries. Adding a trailing dot to external domains in application configuration removes a large part of it.

What to monitor. These four catch most incidents before users do.

  • Matching SOA serials across every authoritative server
  • Days remaining on signature validity, if you sign
  • Query latency and SERVFAIL ratio on the recursive resolver
  • Expiry dates of the domain and the delegation

9. The diagnostic order, summarised

Given a symptom, narrow it in this order. Each step earns its place by ruling the previous one out.

  1. Application or system. Compare getent hosts against dig. A difference means the stub resolver segment.
  2. Which resolver. Confirm the server actually in use with /etc/resolv.conf and resolvectl status, then query that server directly.
  3. Cache or authority. Compare the resolver answer with the authoritative answer. A difference means cache, and the TTL tells you how much time is left.
  4. Delegation. Check the delegation with dig +trace and by comparing parent and child NS.
  5. Validation. On SERVFAIL, ask again with +cd. An answer means DNSSEC validation failure.
  6. Not DNS. If the name resolves correctly, close the DNS investigation here and hand it to the connectivity layer.
getent hosts example.com
dig +short example.com
dig @1.1.1.1 +short example.com
dig @ns1.example.com +short example.com
dig +trace example.com
dig +cd example.com

Quiz: check your understanding

Quiz 1: dig answers correctly but only the application reports "name not found". What do you check first?

Answer: The stub resolver path. dig does not go through the name service switch and reads only /etc/resolv.conf, so the two paths differ

Why: Start by putting getent and dig side by side.

getent hosts api.example.com
dig +short api.example.com
cat /etc/resolv.conf

If only getent fails, look at the name service switch configuration, /etc/hosts, the resolv.conf mounted into the container, and the search domain and ndots settings. Some applications embed their own resolver, so check the runtime name resolution settings as well.

Quiz 2: You changed an A record and some users still reach the old address. What is happening, and what will you do differently next time?

Answer: Resolver caches have not expired yet. Next time, lower the TTL at least one old TTL before the change

Why: DNS has no push. Changes do not spread; caches expire. The time remaining differs per resolver, and it is observable.

dig @1.1.1.1 +noall +answer +ttlunits example.com
dig @ns1.example.com +noall +answer example.com

The resolver-side TTL counts down as you repeat the query, while the authoritative side always shows the configured value. Those two differing is normal.

Quiz 3: The internal resolver returns SERVFAIL while a public resolver answers fine. What do you suspect, and how do you settle it in one step?

Answer: Suspect DNSSEC validation failure, and settle it by asking again with validation disabled

Why: SERVFAIL does not mean the name is absent; it means the server could not produce an answer. Validation failure and failure to reach upstream are the classic causes.

dig @10.0.0.53 example.com
dig @10.0.0.53 +cd example.com
delv example.com

If adding +cd produces an answer, you were blocked at the validation step — check signature expiry and DS mismatch first. If it still fails with +cd, the problem is reachability or the upstream response, not validation.

Quiz 4: You tried to point the root domain example.com at a CDN host name with a CNAME and the zone was rejected. What is the specification reason?

Answer: The zone apex must carry an SOA and NS records, and a CNAME cannot coexist with other data at the same name

Why: RFC 1034 rules that if a CNAME is present at a node, no other data should be present. And the zone apex is defined as the owner of an SOA and the corresponding authoritative NS RRset. The two rules cannot hold at once.

The same reasoning makes this a violation:

example.com.        IN  MX   10 mail-alias.example.com.
mail-alias          IN  CNAME real-mail.example.net.

RFC 2181 rules that NS and MX targets must not be aliases, and RFC 2782 places the same requirement on SRV targets.

Quiz 5: The response is NOERROR but the ANSWER section is empty. Does that mean the name does not exist?

Answer: It exists. This is NODATA — the name is there and only the queried type is missing

Why: RFC 2308 separates NXDOMAIN from NODATA. NXDOMAIN means the name itself does not exist; NODATA means the response code is NOERROR with no relevant answers in the answer section.

dig AAAA example.com
dig A example.com
dig ANY example.com

Both are cached as negative responses, for the smaller of the SOA MINIMUM field and the TTL of the SOA record itself. That is why failures can persist for a while after you fix the wrong record.

Quiz 6: dig +trace shows the right address but the application on the server still connects to the old one. Where do you look?

Answer: The resolver segment. +trace queries from the root itself and bypasses the resolver that host actually uses

Why: A clean +trace only tells you the authoritative side is fine. To reproduce the real path you have to ask the resolver that host uses.

cat /etc/resolv.conf
dig @10.0.0.53 +noall +answer example.com
dig @10.0.0.53 +norecurse example.com
sudo rndc flushname example.com

+norecurse makes the resolver answer only from its own cache, so you learn immediately whether the cache is the cause. If there is a separate local cache, flush that too.


Closing

DNS incidents are hard not because DNS is hard, but because people discuss them without saying which segment they are looking at. State every time whether the answer came from a resolver or an authoritative server, from cache or a fresh lookup, validated or not, and the conversation gets dramatically shorter.

Three habits to build immediately. Ask getent and dig side by side to rule out the stub resolver segment first, query the resolver and the authoritative server separately to separate cache from origin, and lower the TTL at least one old TTL before you change a record. Those three alone make most DNS incidents disappear, or at least resolve the question of which segment is guilty within minutes.


References


Further reading

현재 단락 (1/233)

The moment somebody says "looks like a DNS problem" in an incident call, the investigation usually s...

작성 글자: 0원문 글자: 25,838작성 단락: 0/233