https://www.youngju.dev/blog 천천히 올바르게. AI Researcher & DevOps Engineer Youngju's blog. GPU/CUDA, LLM, MLOps, Kubernetes AI workloads, and data engineering — plus mindset essays on confidence, routines, health, and sport psychology. ko fjvbn2003@gmail.com (Youngju Kim) fjvbn2003@gmail.com (Youngju Kim) Sun, 09 Aug 2026 00:00:00 GMT https://www.youngju.dev/blog/network/2026-08-09-you-cannot-see-your-own-traffic.en https://www.youngju.dev/blog/network/2026-08-09-you-cannot-see-your-own-traffic.en Drawing on one year of defending a 1.5-million-page site against scrapers, this post sets out the principles of dealing with bot traffic. JavaScript-based analytics cannot count bots, so you have to read server logs, and rules must be built on origin information that is hard to forge — ASN, geolocation, cryptographically verified bot status — rather than on self-reported values such as the user agent. It also covers the crawl-to-refer metric, why the ratios Cloudflare published diverge from what an individual site measures, a case where the defensive machinery itself ate the performance budget, and the point at which network-layer blocking is defeated in principle by residential IP botnets. Sun, 09 Aug 2026 00:00:00 GMT fjvbn2003@gmail.com (Youngju Kim) networkbotcloudflarewafscrapingobservability https://www.youngju.dev/blog/network/2026-08-09-you-cannot-see-your-own-traffic.ja https://www.youngju.dev/blog/network/2026-08-09-you-cannot-see-your-own-traffic.ja 150万ページのサイトを1年間スクレイパーから守った記録をもとに、ボットトラフィック対応の原則を整理します。JavaScriptベースの分析ツールはボットを数えられないのでサーバーログを見る必要があり、ユーザーエージェントのような自己申告値ではなく、ASNや地理的位置、暗号学的に検証されたボットかどうかといった偽装しにくい出自の情報で規則を立てるべきです。クロールあたりの流入訪問者数という指標、Cloudflareが公開した比率と個別サイトの実測値が食い違う理由、防御機構そのものが性能予算を食い潰していた事例、そして住宅用IPのボットネットの前でネットワーク層の遮断が原理的に行き詰まる地点までを扱います。 Sun, 09 Aug 2026 00:00:00 GMT fjvbn2003@gmail.com (Youngju Kim) networkbotcloudflarewafscrapingobservability https://www.youngju.dev/blog/network/2026-08-09-you-cannot-see-your-own-traffic https://www.youngju.dev/blog/network/2026-08-09-you-cannot-see-your-own-traffic 150만 페이지짜리 사이트를 1년간 스크레이퍼로부터 방어한 기록을 근거로, 봇 트래픽 대응의 원칙을 정리합니다. 자바스크립트 기반 분석 도구는 봇을 세지 못하므로 서버 로그를 봐야 하고, 사용자 에이전트 같은 자기 신고 값 대신 ASN과 지리 위치, 암호학적으로 검증된 봇 여부처럼 위조하기 어려운 출처 정보로 규칙을 세워야 합니다. 크롤당 유입 방문자 수라는 지표, Cloudflare가 공개한 비율과 개별 사이트 측정치가 갈리는 이유, 방어 장치 자체가 성능 예산을 잡아먹은 사례, 그리고 주거용 IP 봇넷 앞에서 네트워크 계층 차단이 원리적으로 막히는 지점까지 다룹니다. Sun, 09 Aug 2026 00:00:00 GMT fjvbn2003@gmail.com (Youngju Kim) networkbotcloudflarewafscrapingobservability https://www.youngju.dev/blog/network/2026-08-09-you-cannot-see-your-own-traffic.zh https://www.youngju.dev/blog/network/2026-08-09-you-cannot-see-your-own-traffic.zh 以一个 150 万页面的站点抵御爬虫一整年的记录为依据，梳理应对机器人流量的原则。基于 JavaScript 的分析工具数不到机器人，所以必须去看服务器日志；规则也不能建立在 User-Agent 这类自我声明的值上，而要建立在 ASN、地理位置、是否为经密码学验证的机器人这些难以伪造的来源信息上。文中还会谈到每次抓取带来多少访客这个指标、Cloudflare 公布的比率与单站实测值为何会分歧、防御装置本身吃掉性能预算的案例，以及在住宅 IP 僵尸网络面前网络层拦截为何在原理上走不通。 Sun, 09 Aug 2026 00:00:00 GMT fjvbn2003@gmail.com (Youngju Kim) networkbotcloudflarewafscrapingobservability