Skip to content
Published on

Following 657,607 Links, and the Lifespan of a URL — Why Links Die of Connection Failure, Not 404

🇰🇷KO🇺🇸EN🇯🇵JA🇨🇳ZH
Split View필사 모드
Share
Authors

This post is based on items I read directly from the Hacker News API and the GeekNews feed on 2026-08-15. Scores and rankings keep moving.

What was up there

An item read from the Hacker News API. The title is Where did the old web go? We followed 657,607 links to find out, the item number is 49289532, and as of 2026-08-15 it stood at 221 points with 207 comments. The link points to a post in which 0.mk, a Macedonian link shortener, examined its own data.

The methodology

The post is relatively honest about its method. The subject is 657,607 links created on that service between 2009 and 2014, restored from an old database backup. Excluding 2,429 malformed or inaccessible records left 655,178 actually crawled. The crawl was performed in August 2026.

The crawler followed up to five redirects and retried connection failures once from a second network. Success is defined as HTTP 2xx and 3xx.

The composition of the failures matters more than the total

The overall result: 23.32% loaded normally, 25.44% returned an HTTP error, and 51.24% could not be connected to at all. DNS, timeout, and TLS problems are listed as the causes of connection failure.

That third number is the most important finding in the study.

We usually picture a link dying as a 404 — the site is alive but that one document moved or was deleted. In reality that was the minority. The majority of dead links failed to reach the server at all, meaning the domain and hosting disappeared wholesale, not the document.

That distinction matters practically because the response is entirely different. A 404 is in principle fixable: add a redirect, or find the new location. But when DNS does not answer, there is nothing to fix. In other words, most link rot is not a content management problem but an organizational-disappearance problem.

There are deduplicated results too. Of 494,781 distinct destinations, only 21.3% loaded. At the hostname level, only 34,827 of 133,605 had even one successful URL. The most common HTTP error was 404, across 76,403 URLs.

The per-year numbers and their trap

Failure rates by year are given as 64.58% for 2009, 60.39% for 2010, 92.53% for 2011, 59.43% for 2012, 75.06% for 2013, and 78.16% for 2014.

A caveat the post supplies itself matters here: 2011 is unusually high because a single bulk batch of 83,398 links falls in that year and skews the result. Including that note is a trustworthy signal, but it is simultaneously a warning that the yearly trend must not be read at face value.

And there is one more thing to raise about the study design. Defining success as 2xx and 3xx means soft 404s are counted as alive. A domain sold off and turned into an ad page, or a site redesign that redirects every old address to the front page, all count as successes. So the real survival rate is likely lower than 23.32%. This study is skewed toward understating link rot, not overstating it.

What survived

The pattern the post describes is predictable and painful. Large platforms survived well; personal blogs, forums, local news, and photo hosts did not. The post sums it up as the centralized web having generally held up better than the small web.

And there is one detail unique to this study: 4,478 links pointed at other shortening services. In that case failure probabilities multiply. Even if the original site is alive, the link dies when the intermediate shortener closes. The longer the redirect chain, the more the lifespan follows the weakest link.

Among the recurring threads in the comments were an argument over when exactly the old web was, and the irony of the study itself — a shortener that was offline for about a decade writing about other sites disappearing.

What should change in our systems

The conclusion this study offers a practitioner is not nostalgia about the old web. It is that an external URL is an asset that loses value over time, and if such a column exists in your database, you are already carrying that decay.

It splits three ways.

First, store the evidence alongside the reference. When a user submits a link or your system cites an external document, do not keep only the URL; keep the title at that moment, the publication date, and where possible a portion of the body or a snapshot. Even after the original disappears, what it pointed at remains. This trades storage for not being unable to investigate later.

Second, shorten redirect chains. As those 4,478 records show, intermediate hops are multiplied risk. In particular, if an internal system issues short URLs that get scattered through your documents, that shortener's lifespan becomes the ceiling on the lifespan of all those documents.

Third, design the addresses you publish with lifespan in mind. If your addresses embed a framework name, a deployment method, or your org chart, then all of them change when any one of those three does. Leaving redirects behind is the right answer, but in practice it is the last item on a migration checklist and gets dropped. REST API design best practices and HTTP caching strategies are worth reading alongside this.

One addition: periodically checking whether links are alive is easy to automate. But because of the soft 404 problem above, you cannot judge by response code alone. At minimum, also check whether the final URL diverged substantially from the original path and whether the response length is abnormally short.

Who this does not apply to

Systems that do not handle external links as data are unaffected. If you reference only internal resources whose lifespan you control, this is not link rot but ordinary migration management.

The same for short-lived data. If a link is discarded in a few weeks, its ten-year survival rate is meaningless.

Conversely, what this study targets directly is systems where references serve as evidence: citation-keeping tools, compliance records, knowledge bases, and pipelines that store source URLs alongside training data. In those places, being unable to produce the evidence a few years later is a real loss.

Summary

The number to remember from this study is not 23.32% but 51.24%. Links mostly die not because a document moved but because whatever was serving it ceased to exist. That changes the response: the work is not fixing links but capturing the evidence at the moment you reference it. And even this study's numbers, generous enough to count 3xx as success, are likely the optimistic side.

Sources and related reading

The point about understatement from soft 404s, and the suggestions for your own systems, are my own, built on the methodology described in the source.

Discuss on Twitter • View on GitHub