https://www.youngju.dev/blog 천천히 올바르게. AI Researcher & DevOps Engineer Youngju's tech blog. GPU/CUDA, LLM, MLOps, Kubernetes AI workloads, distributed training, and data engineering. ko fjvbn2003@gmail.com (Youngju Kim) fjvbn2003@gmail.com (Youngju Kim) Sat, 11 Jul 2026 00:00:00 GMT https://www.youngju.dev/blog/2026-07-11-residential-proxies-scrapers.en https://www.youngju.dev/blog/2026-07-11-residential-proxies-scrapers.en On July 10, 2026, Jonathan Corbet of LWN published an update on the scraper situation. Two facts sit at the center: residential proxy networks have effectively broken IP-based blocking, and the AI-crawler gold rush supplies the demand. In a single attack, millions of distinct IP addresses can hit a site over a few hours, each one only two or three times. This post explains why blocking fails structurally, why open-source git forges were hit first (and why Anubis was born there), and what tax the defenses — proof-of-work, CAPTCHAs, data poisoning — place on the open web. Sat, 11 Jul 2026 00:00:00 GMT fjvbn2003@gmail.com (Youngju Kim) web-scrapingresidential-proxyai-crawlersanubisinfrastructurelwn https://www.youngju.dev/blog/2026-07-11-residential-proxies-scrapers.ja https://www.youngju.dev/blog/2026-07-11-residential-proxies-scrapers.ja 2026年7月10日、LWNのJonathan Corbet氏がスクレイパー状況のアップデートを公開しました。中心には二つの事実があります — residential proxy(家庭用プロキシ)がIPベースのブロックを事実上無力化し、AIクローラーのゴールドラッシュがその需要を満たしています。一度の攻撃で、数時間のあいだに数百万の異なるIPアドレスがサイトを叩き、それぞれ二〜三回しかアクセスしません。この記事は、なぜブロックが構造的に失敗するのか、なぜオープンソースのgitフォージが最初に狙われたのか(そしてAnubisがなぜそこで生まれたのか)、そして作業証明・CAPTCHA・データ汚染といった防御が開かれたウェブに課す税を、正直に整理します。 Sat, 11 Jul 2026 00:00:00 GMT fjvbn2003@gmail.com (Youngju Kim) web-scrapingresidential-proxyai-crawlersanubisinfrastructurelwn https://www.youngju.dev/blog/2026-07-11-residential-proxies-scrapers https://www.youngju.dev/blog/2026-07-11-residential-proxies-scrapers 2026년 7월 10일 LWN의 Jonathan Corbet가 스크래퍼 상황 업데이트를 냈습니다. 핵심은 두 가지입니다 — residential proxy(가정용 프록시)가 IP 기반 차단을 사실상 무력화했고, AI 크롤러의 폭증이 그 수요를 채웠습니다. 한 번의 공격에서 몇 시간 동안 수백만 개의 서로 다른 IP가 각자 두세 번씩만 사이트를 두드립니다. 이 글은 왜 차단이 구조적으로 실패하는지, 오픈소스 git 포지가 왜 가장 먼저 맞았는지(그리고 Anubis가 왜 거기서 태어났는지), 그리고 작업증명·CAPTCHA·데이터 오염 같은 방어가 개방형 웹에 물리는 세금을 정직하게 정리합니다. Sat, 11 Jul 2026 00:00:00 GMT fjvbn2003@gmail.com (Youngju Kim) web-scrapingresidential-proxyai-crawlersanubisinfrastructurelwn