Split View: 방문자 통계로는 트래픽의 0.5퍼센트만 보인다 — 봇 방어는 자기 신고가 아니라 출처로 한다

Visitor Analytics Shows Only 0.5 Percent of Your Traffic — Judge Bots by Origin, Not by Self-Report

• Introduction — the 99 percent your dashboard cannot see
• JavaScript analytics cannot count bots
• Crawl-to-refer ratio — compressing the judgment into a single number
• Why the published ratio and your own site's ratio differ
• Rules that trust self-reports and rules that look at origin
• Even origin rules are powerless against residential IP botnets
• Measure the cost of the defensive machinery itself
• What proof of work solves and what it does not
• References

Introduction — the 99 percent your dashboard cannot see

On 7 August 2026 the operator of a site called PatronView published a year of records from defending against scrapers. It is a database of American charitable donors, with 1.5 million individual profile pages built from IRS 990 filings and public donor lists.

In the week the post went up, the server received 2.5 million requests from outside and served 1.28 million complete pages. Yet the visitor analytics recorded only 5,977 pageviews.

That means for every visible page load, roughly 214 invisible ones.

What there is to learn here is not that there are a lot of bots. Everyone knows there are a lot of bots. What there is to learn is that the metric you are currently looking at is structurally incapable of showing you this situation.

JavaScript analytics cannot count bots

The cause is simple. Client-side analytics tools, Plausible and Fathom and Google Analytics included, count only visitors that execute JavaScript. Most bots do not execute JavaScript.

So the dashboard shows a cozy little site with around 500 visitors a day, while the server answers millions of requests a week. Both numbers are true. They are simply counting different things.

There is one practical conclusion. When you are judging bot traffic, read server logs or edge logs, not your analytics tool. And check the gap between the two on a regular schedule. The moment the gap starts widening is the moment something started.

The first bot detection story in the original went exactly that way. In November 2025, over several days, 4,000 "visitors" appeared. Each viewed exactly one page, with a 99 percent bounce rate and no referrer. And they were sweeping only one particular type of page, one that only 10 percent of real visitors ever look at.

None of these signals is decisive on its own, but together they are unambiguous. And these signals are visible in an analytics tool too, because these were bots that execute JavaScript.

Crawl-to-refer ratio — compressing the judgment into a single number

The most practical part of the original is the metric the author settled on. For every so many pages a crawler reads, how many visitors does it send you?

Here are the values the author measured.

The basis on the Claude side is concrete. In one particular week Claude-SearchBot requested 420,680 pages, and in the same week the visitors that arrived via Claude-User — a separate user agent used when an actual person requests a page — numbered 12. In bandwidth terms that was 4.63GB to the bot and 175KB to the people the bot sent.

On the Amazon side it was around 117,000 requests a day, the top crawler at that point, and it sent not a single visitor.

The good thing about this metric is that it turns a policy decision into arithmetic rather than morality. Not "AI crawlers are bad" but "this crawler uses my bandwidth and gives nothing back." The author writes that Bingbot, at 406 to 1, is nine times worse than Google, but that referrals are actually growing, so it stays allowed. Applying the same criterion and arriving at different conclusions is evidence of a good criterion.

The post-blocking observation is worth recording too. When Claude-SearchBot was blocked at the firewall, requests fell from 60,000 a day to something like 25 attempts a day. In the author's words, "well-behaved AI companies really do respect a 403."

Why the published ratio and your own site's ratio differ

The author quotes that "Cloudflare put the Anthropic crawler at roughly 1 visitor per 3,000 crawls" and writes that on their own site they measured 35,000 to 1.

It matters to understand why these two numbers differ.

The post where Cloudflare first published this metric is dated 1 July 2025 and states the calculation explicitly. It is the total of requests from user agents associated with that platform whose response content type was HTML, divided by the referral traffic coming from that platform. In the window from 19 to 26 June 2025, Anthropic was reported at 70,900 to 1.

The same post carries a caveat Cloudflare attached itself. Referral traffic sent by the Claude native app carries no referrer header, and other native apps are likely the same. So they write that their calculation "may overstate the ratio, and it is unclear by how much."

Three things follow from this.

First, this ratio moves a great deal depending on when you look. 70,900, 3,000, and 35,000 are all values from different moments and different populations. Quoting any one of them as "the" ratio for AI crawlers is wrong.

Second, your own site's value matters more than the aggregate. It varies greatly with the nature of your content, the geography of your readership, and your indexing status.

Third, referrals with no referrer header are undercounted. Which means the real ratio may be more favorable than the measured one. You need to know this limitation before you make a blocking decision.

Rules that trust self-reports and rules that look at origin

The appendix of the original publishes the full firewall rule expressions. More important than any individual rule is what kind of information it depends on.

Some rules depend on self-reported values. The user agent string is the classic case.

(lower(http.user_agent) contains "semrushbot") or
(lower(http.user_agent) contains "ahrefsbot") or
(lower(http.user_agent) contains "mj12bot")

Rules like this work only on honest crawlers. The author writes of the SEO crawlers that they identify themselves honestly and that this is appreciated. Against anything that lies, they have no effect at all.

Rules that depend on origin information are different. Which country an IP came from and which ASN it belongs to are not things the sender gets to choose.

((ip.src.asnum in {212238 139341 9009}) or
 (ip.src.continent in {"AF" "AN" "AS" "OC" "SA" "T1" "EU"}))
and not cf.client.bot
and not (ip.src.country in {"GU" "AS" "MP"})

The two rules the author kept longest are both of this kind. Challenge continents outside North America, and challenge 46 large cloud ASNs. Because the actual readership is 97 percent North American, and people do not browse the web from AWS us-east-1 IPs. The reason it is a challenge and not a block is that someone using a cloud desktop or a VPN may be inside.

The third kind is cryptographic verification. Cloudflare verifies the identity of Googlebot, Bingbot, and Applebot, and rules refer to this with cf.client.bot. The ordering the author points out is the trick. Put this bypass rule after the blocking rules, and the verified bots you decided to block stay blocked while a challenge never goes to Google.

When designing a rule, always ask this question first. Can the other side change the value this rule refers to at will? If they can, the lifetime of that rule runs until the day they start caring.

Even origin rules are powerless against residential IP botnets

Origin-based rules have their own limit, though, and the original runs straight into it.

In July 2026 a new wave came from the United States. From a family home in Ohio, on a Spectrum line. It passes the continent rule and it passes the datacenter rule. Both of them simply wave this traffic through. On 31 July unique IPs for the day climbed to 124,000, against a normal baseline of about 18,000.

That is what a residential proxy network is by definition. Because it rents thousands of real household internet lines by the request, every request comes out of a real person's line. The author's conclusion is exact: blocking these at the network layer is impossible by design.

The response the author found for this wave was on a slightly different axis. The same botnet was pretending to be Chrome 118 through 120. A 2023 browser version had been frozen into the scraping toolkit as-is.

So old browsers get a challenge. Chrome 100 through 130, and old Firefox. Real traffic was checked first: among actual visitors arriving from search, the share using browsers that old was 0.54 percent, and most of those were Firefox 115 ESR, so that one was excepted.

This rule too, in the end, depends on a self-reported value, the user agent. Fix one string and it is powerless. The author knows this and calls it a stupid rule that they happen to like. The point is this. If you are going to use a rule that depends on self-reporting, first measure what percentage of your real users it hits, then use it.

Measure the cost of the defensive machinery itself

The most unexpected and most broadly applicable lesson in this post is here.

Cloudflare's "JavaScript Detections" feature had been injecting a verification script into every page for a full year. The author found it while trying to make the site faster.

That script consumed 2,875 milliseconds on a mid-range phone. The site's own JavaScript, all of it, was 278 milliseconds. It was the single biggest reason the mobile Lighthouse score was 58, and on top of that, the plan in use did not even allow the resulting verdict to be read from firewall rules. Forty points of performance were being paid for telemetry nobody could read.

Turning the feature off on 5 August took the Lighthouse score to 99 within an hour.

And five hours later, a scraper on Azure took 23,000 pages in an hour. It used more than 80 IPs, each one sitting politely under the rate limit. Which is to say the feature had in fact been doing work.

The lesson of this story is neither "do not turn it off" nor "turn it on." It is have you ever measured what your defenses cost. Bot mitigation features generally come as an "on" toggle, and once switched on nobody looks at them again. In the meantime that feature keeps collecting a fee from your performance budget and your user experience.

The trick for operating in a measurable form is in the original too. Use a challenge instead of a block and you are left with a pass rate. In the author's 48-hour window, 252 of 106,437 challenges were passed, a pass rate of 0.24 percent. The author's criterion is clear. A pass rate of 0.2 percent means these are bots, so keep the rule; a pass rate of 30 percent means you are taxing humans, so fix the rule.

What proof of work solves and what it does not

The densest argument in the Hacker News discussion of this post was about proof-of-work schemes such as Anubis. Both sides are worth carrying over as they were.

The critical side goes like this. The Anubis challenge takes the form of appending a nonce to a challenge string and computing a SHA-256 hash, but because the nonce comes at the end, the compression rounds over the earlier part can be cached. The midstate reuse technique from Bitcoin mining applies directly. The criticism is that putting the nonce first would have avoided the problem. So a native-code solver is thousands of times faster than browser JavaScript, and solves in milliseconds a difficulty that would take ten minutes in a browser.

The defending side answers empirically. Being bypassable and actually being bypassed are different things. A purpose-built scraper could bypass it far more efficiently, but almost nobody does, because most bots are not targeting you specifically — they are just trying to collect in bulk at low cost.

The Anubis maintainer joined the discussion as well, saying that a WASM solver is in preparation for after the next release, and that verifying it down to old smart TV browsers is taking time.

To sum up, proof of work is not a device for distinguishing humans from bots but a device for raising the unit cost of bulk collection. Understood that way, the evaluation criteria become clear as well. It does little against someone who has singled you out, and it works well against indiscriminate collection. And because it forces JavaScript execution, it is a real cost imposed on text browser users and users on slow connections.

The author's own conclusion is economic rather than technical. Scraping keeps getting worse because it keeps getting cheaper, and the real solution is a market, something like pay-per-crawl. Until that market exists, the author's rule is one sentence. A crawler that sends not a single visitor gets blocked.

References

• 99% of My Website Traffic Is Bots — PatronView, Nick Gray, 2026-08-07 (all the figures and rule expressions are carried over from this post)
• The crawl before the fall of referrals — Cloudflare Blog, 2025-07-01 (the definition of the crawl-to-refer ratio and Cloudflare's own caveat)
• Hacker News discussion thread (the source of the proof-of-work argument)
• Anubis

The figures in this post are measurements from a single individual site, and they vary greatly with the character of the site and its readership. They are not values I reproduced myself.