Scrapers Now Come From Millions of Home IPs — Residential Proxies and Open-Source Infrastructure

• Introduction — The Day an IP Stopped Being an Identity
• Why Residential Proxies Cannot Be Blocked
• The AI-Crawler Surge Hit Open-Source Infrastructure First
• Defenses and Their Cost
• Closing
• References

Introduction — The Day an IP Stopped Being an Identity

On July 10, 2026, Jonathan Corbet of LWN published an update on the scraper situation. It follows an early-2025 piece, and its calm conclusion is what makes it heavy: the problem has not been solved, it has kept growing, and LWN recently absorbed its heaviest scraper attack yet.

If you run any public infrastructure at all, this is not someone else's story. A git server, a wiki, a docs site, a personal blog — all of them are targets now. And the defensive playbook we leaned on for a decade — rate-limit by IP, block bad user-agents, filter by geography — barely works anymore. Two things overlapped to cause this. Residential proxies broke the assumption that an IP equals an identity, and the AI-crawler gold rush supplied the demand. This post walks through why blocking fails structurally, and the honest fact that every defense on the table taxes the open web.

Why Residential Proxies Cannot Be Blocked

Start with the shape of the attack. Here is what Corbet reports about the recent one.

Observed in a single attack (LWN report, 2026-07-10)
- Millions of distinct IP addresses over a few hours
- Each IP hits the site at most two or three times
- Bots usually skip images and CSS (a not-a-human signal)
- But by the time you notice, that IP never appears again

The crux is that each of those IPs is a real home connection. A residential proxy is software installed on ordinary people's devices — phones, set-top boxes, routers — that takes commands from a central control node, fetches pages on its behalf, and relays the results back. The owner of the device usually has no idea.

There are two supply chains. One is criminal botnets: malware-compromised devices, with media-streaming boxes named as a major carrier. The other is the "quasi-legitimate" operator. Corbet points to Bright Data as the most prominent example — it openly advertises its ability to get around access controls and traffic limits, and it offers a "free" VPN in exchange for the right to route traffic through the user's device. Other, unnamed operators hand app developers an SDK and pay them to resell their users' network connections.

That is why blocking fails. Each request arrives from a genuine home address wearing a plausibly fabricated user-agent. By the time you fingerprint it as a bot, that IP has already retired. IP reputation, rate limits, and geo-blocks all rest on the premise that IP identity is stable — and that premise has collapsed. Takedowns help briefly: Google dismantled IPIDEA at the start of the year, and on July 2 it took down NetNut in coordination with the FBI and others, and each time LWN's traffic dipped. But within months the attacks returned. It is whack-a-mole.

The AI-Crawler Surge Hit Open-Source Infrastructure First

One thing in the interest of honesty: the body of the LWN piece is mostly LWN's own operational account. But the place hit first and hardest by this wave was open-source infrastructure, and the best-known defense was born there.

That defense is Anubis. Xe Iaso built it in early 2025 after Amazon's crawler ignored robots.txt and trampled their git server. It is an MIT-licensed reverse proxy: a gatekeeper that makes a visitor's browser solve a SHA-256 proof-of-work puzzle before any page is served. Git forges and FOSS projects adopted it in a row — the Linux kernel mailing list archive, SourceHut, FFmpeg, Wine, GNOME's GitLab, and more.

Why git forges specifically? Dynamic endpoints like blame, diff, and the blob at every commit are expensive per request. A crawler that walks every commit of every file is a pathological load all by itself. And a self-hosted open-source project has no budget for Cloudflare Enterprise. So a lightweight proof-of-work wall was a rational choice.

The trouble is that this is an arms race. Codeberg reported by mid-2025 that many bots had already learned to solve Anubis challenges. A reader comment on the LWN piece says the same thing — AI crawlers are hammering git endpoints and going right past the Anubis checks, and at least some appear to be driving full browsers. Proof-of-work raises the cost for cheap bots, but a well-funded crawler simply runs a real browser.

Defenses and Their Cost

Here is the menu of defenses Corbet surveys. None of them is free.

The CAPTCHA escalation is bleak. Corbet lists finding streetlights, placing puzzle pieces, and even a biometric challenge that asks you to hum a song while holding down the space bar. The smarter the bots get, the heavier the load pushed onto humans. LWN itself decided not to use Anubis. Corbet gives two reasons: it imposes annoying delays on real visitors, and it seems inevitable that scrapers will eventually find their way around it. Instead LWN aggressively optimized the site, cut expensive operations, and treats anonymous and logged-in users differently. One honest and ironic result: response times while under attack are often better than during the calm periods.

But the fundamental cost remains, in three layers. First, a tax on everyone: operators must build and maintain this machinery, and users must cope with the friction. Second, collateral damage: the harder you crank the defenses, the more you catch legitimate search engines and the Internet Archive. And if you allowlist only the dominant engines, you further entrench a monopoly that, in Corbet's words, already serves us poorly. Third, every defense is temporary. Even when it works today, you have to plan now for the day it no longer does.

Closing

There is no clean win here. Residential proxies shattered the old assumption that an IP is an identity, the AI gold rush stacked unlimited demand on top, and every countermeasure is a trade that converts part of the open web into a moat. That is why Corbet's worry is not hyperbole — there is a real risk that the entire internet retreats behind defensive walls.

The practical lesson for anyone running public infrastructure is this. Assume IP-based defenses are dead. Budget for the friction cost. And accept that this is not a problem to solve but an arms race to keep running. The honest response is not to lean on a magic wall but to design for graceful degradation — cheap endpoints, caching, a clean split between anonymous and logged-in traffic. Walls get breached eventually. Better to build so you are still standing when one does.

References

• An update on the scraper situation — Jonathan Corbet, LWN.net (2026-07-10)
• Anubis (software) — Wikipedia
• Anubis: Fighting off the hordes of LLM bot crawlers — The Register (2025-07-09)
• Using proof-of-work to stop excessive crawling — Forgejo discussion (Codeberg)