CF GarageFor confidentiality reasons, and because attackers or scrapers may be reading these lines, part of the data has been altered to anonymise the case: the domain name, the targeted pages and the identifiers of the rules in place are not reproduced. Volumes, signatures and reasoning are faithful to the measurements; the wave was still running at the time of writing. The dashboard screenshots have been cropped with the same goal in mind.
The site and the symptom
A European marketplace connects professionals. To do so it publishes a public catalogue of several hundred thousand offer pages, whose prices are watched both by competitors and by the platform's customers. The catalogue is queried by the website and by a mobile application sharing the same domain, for roughly five million requests a day.
The administrator first suffered severe slowdowns on their server, then, while looking for the cause, found a spike in unique visitors on the Cloudflare dashboard. No advertising campaign could explain such a rise: unique visitors had been multiplied by ten to twenty. Hence the question: where was this traffic coming from?
Day Requests Page views Unique visitors
D-4 5.33 M 443 219 43 309
D-3 5.23 M 429 719 44 173
D-2 4.96 M 409 933 64 532
D-1 2.37 M 230 294 47 137
D 3.27 M 208 304 816 627
D+1 6.28 M 444 606 721 136
D+2 6.03 M 460 856 522 820
D+3 5.48 M 420 887 378 014The diagnosis is almost entirely in the shape of those three columns. Unique visitors were multiplied by eight to nineteen depending on which day you compare. Page views did not move, and on day D they actually went down. Traffic that arrives without generating page views is traffic that does not navigate: it fetches one page, it leaves, it comes back under another address. The hourly series then pinned the switch to the hour, from 30 223 requests at 02:00 UTC to 80 908 at 03:00, with no return to baseline afterwards.
Why the usual readings showed nothing
Three separate things explain why this attack wave could go unnoticed, and what it was actually costing. Each of them is common enough to be worth naming.
- The source networks mirrored the audience. Over one four-hour sample, the top source networks were a Vietnamese operator, three French consumer ISPs, a Romanian and Portuguese operator, an Emirati ISP, a Moroccan incumbent, a Brazilian carrier, a Pakistani operator, a Turkish one and a Mexican one, with cloud providers such as Google, Microsoft and Amazon making up a minority. In other words, the geographic spread of the traffic looked like the geographic spread of the site's real audience. Residential bot pools are sold precisely so that a country or ASN reading returns nothing.
- Attacks were already being caught, and already explained. The zone was already receiving a fair amount of attacks and abusive requests: a custom rule against web scraping had matched more than 1.5 million requests over two days. Seeing a large number rise on a counter that is supposed to rise is not an alarm. It reads as the protection doing its job, not as evidence that a second wave is passing beside it.
- The origin server, precisely, had collapsed. The slowdowns that alerted the administrator came from there: the origin took the first waves head-on before the edge cache absorbed most of the load, as shown below. But the underlying problem went beyond server load: the bots were scraping the catalogue's prices, genuine commercial espionage with a very significant economic impact for the administrator. That damage, far more than the load, was what made this urgent.
Our analysis method made the wave visible
Rather than slicing by country or network, we crossed four dimensions in a single query: the browser as parsed by Cloudflare, the HTTP protocol version, the TLS version, and the source of Cloudflare's bot classification. Over a four-hour window, restricted to requests that actually reached the origin:
191 225 Chrome HTTP/1.1 TLSv1.3 heuristics
10 966 Chrome HTTP/1.1 TLSv1.3 machine_learning
6 944 Chrome HTTP/2 TLSv1.3 machine_learningThe same traffic at the same hours, the previous day, came to 4 444 requests. A factor of forty-three, in one line. Three elements converge on the same reading.
- A protocol contradiction. A real Chrome negotiates HTTP/2 through ALPN against a Cloudflare edge and does not spontaneously fall back to HTTP/1.1. "Chrome over HTTP/1.1" is the signature of an HTTP library, whether Go
net/http, Pythonrequests,curlornode-fetch, carrying a Chrome user agent string. - Deterministic classification. Cloudflare labelled this traffic
heuristics, meaning it matched a known automation signature, rather thanmachine_learning, which is the probabilistic scoring applied to ambiguous traffic. The edge had already recognised the shape; nothing had been configured to act on it. - A hard-coded user agent pool. Fourteen distinct user agent strings, each carrying between 14 059 and 14 451 requests. Real traffic is never that evenly distributed; a three percent spread across fourteen values is a round-robin through a list. The announced versions were old and inconsistent with each other, Chrome 99 to 116 on Windows and 119 to 136 on Mac, while the site's legitimate traffic was dominated by Chrome 152.
A second discriminator supports the reading: the referer. Filtering that same window down to Chrome over HTTP/1.1 classified as heuristics, and grouping by referer host:
191 228 no referer (99.99 %)
14 referer presentReal navigation to a deep catalogue page almost always carries a referer, internal or external. This pool never sent one. That matters for what comes next: the referer signal survives a switch to HTTP/2, which the protocol clause does not.
It is not free of false positives, since it also catches direct entry through a typed URL or a bookmark, measured at roughly 3 500 requests over twenty hours on this zone. Which is exactly why it gets measured before it gets enforced.
Steady around the clock, then moved into the cache
Three days after the scraping wave began, it was still running, but it no longer looked the same depending on where you observed it from.
On total traffic, the suspect traffic stayed constant: around 25 000 requests an hour day and night, 471 370 requests over a full day, 8.5% of the zone's traffic, with no trace of a day and night cycle and an unexplained trough between 09:00 and 12:00 UTC. Over the same day, legitimate traffic varied by a factor of six between its 02:00 trough and its 14:00 peak. A curve that ignores the cycle of its own audience is automation, on its own, without any other evidence.
On the traffic that actually reached the origin server, by contrast, the same curve had collapsed after two days on the plateau: most of the scraping was now being served by the edge cache, from pages that had already been requested. Nothing had changed on the attacker's side; the edge was doing the absorbing.
This is the part that catches most administrators out: against cached static assets (images, CSS, fonts), scraping can keep running at full speed without moving a single server monitoring curve.
Two traps on the way to a fix
Per-URL rate limiting does nothing here
Of the 1 544 207 requests the existing rule had blocked over two days, the fifty most-hit paths accounted for fewer than 55 700, less than 4% of the volume.
The rest was a long tail of detail pages enumerated by sequential identifier across a range of roughly 39 000 values, each individual page sitting at 400 to 430 hits over two days.
Every single URL stayed below any threshold you could reasonably set, so a per-URL rate limit would have matched nothing at all. Counting has to aggregate by URI pattern or by source IP. This is the single most common reason a rate limiting rule ships and changes nothing.
Read the actions before you read the volumes
Skip rule carries more traffic than the blocking rule; its volume alone makes it neither a flaw nor a candidate for tightening. The volumes shown cover a different window than the ones quoted in the text.The zone had a second rule matching more than 1.4 million requests on the mobile application's endpoints. Judged on volume alone, that looks like a large amount of unfiltered traffic and a candidate for tightening.
It was in fact a deliberate Skip allowlist, put there so that the mobile app would never be challenged. Turning it into a block would have taken the application down for every customer. We raised it as a candidate mid-analysis and withdrew the suggestion as soon as we identified which rule carried which action.
Event counts tell you what a rule matched, never what it did with it; on a zone you did not configure yourself, establish the action of every rule before you propose touching any of them.
The recommendation: a stack of rules, not a single one
The protocol rule stops this wave, not this attacker. The day the attacker switches their library to HTTP/2, which curl --http2, Python httpx and Go net/http all do natively, the contradiction disappears and the rule matches nothing.
Hence a stack of rules organised in five layers. Each layer makes every new IP address more expensive, and the attacker is paying for that traffic.
Layer 1, whitelisting legitimate automated traffic
Before blocking anything, we explicitly whitelist the bots we want to let through: search engine crawlers (Googlebot, Bingbot), monitoring, the site's mobile application. Without this layer at the top of the list, every subsequent rule risks hitting a useful automation and creating a costly false positive.
Layer 2, whitelisting business flows
Webhooks and partner IP addresses are explicitly allowed, for the same reason: these are legitimate automations that must never meet a challenge.
Layer 3, blacklisting unwanted traffic
Blacklisting relies on everything that identifies unwanted traffic:
- on user agents: the pool of fourteen hard-coded strings, with announced versions so old (Chrome 99 to 116 on Windows, 119 to 136 on Mac) that the legitimate residual is near zero. Fragile by construction, since the pool can be regenerated in an afternoon, but free and immediate;
- on the protocol level: the "Chrome over HTTP/1.1" contradiction (
http.user_agent contains "Chrome" and http.request.version eq "HTTP/1.1"), with a Managed Challenge action rather than a block, because some corporate proxies do downgrade to HTTP/1.1 and a real browser behind one will solve the challenge where a Go client will fail. The mobile app is unaffected, its user agent contains no browser string, and Android WebView traffic speaks HTTP/2 and escapes the clause; - on URLs: a lock on the targeted section. Requests to the enumerated section without a
cf_clearancecookie get a Managed Challenge. This is the main lever against IP rotation: every new address has to solve a challenge before seeing a single page, which is what makes a metered residential bot pool expensive. It needs a clause exempting the application's own session cookie, so that signed-in customers do not meet a challenge on their first click.
On the same URL scope, rate limiting complements the blacklist: counting on the URI pattern rather than the exact URL, at a threshold of fifteen to twenty requests per IP over ten minutes, action Managed Challenge. A human consults a handful of offers; a sequential scraping across tens of thousands of identifiers crosses that line even at a deliberately slow pace.
Layer 4, Managed Challenge on unserved zones
For geographic zones and ASNs from which the site sells nothing, a Managed Challenge rather than an outright block: a real visitor who ended up there will solve the challenge where a bot will fail. This particular wave escapes this layer, its addresses being precisely diluted across consumer operators in the site's real markets; the layer still earns its place, raising the cost of any less careful wave while costing little to maintain.
Layer 5, rate limiting the automated traffic that was let through
Everything whitelisted at layers 1 and 2 remains subject to a quota: AI crawlers, API partners, mobile application users. The authorisation says "you may pass", the rate limiting says "not indefinitely". A legitimate AI crawler or partner sits comfortably below a reasonable threshold; that is exactly what separates a tolerated automation from a bot hiding behind the same signatures.
Underneath these five layers, Super Bot Fight Mode set to challenge definitely-automated traffic acts as the net if everything above is circumvented.
And every one of these rules starts in Log action, for a few hours, to measure the legitimate residual against the baseline. Expressions are then refined if needed, before moving to real blocking actions. One change at a time, never several, or no effect can be attributed to its cause.
The item that outranked every rule
The enumerated pages sat under an account path (/my-account/) and were addressed by a sequential identifier in the URL, of the form offer-details-5232469.html: the detail page of offer number 5 232 469.
This setup goes straight at the item ranked A01, "Broken Access Control", at the top of the OWASP Top 10: when a URL designates a resource by an identifier, the application must check, on every request, that the authenticated user is actually allowed to see that resource. Its absence has a name: IDOR, for Insecure Direct Object Reference. The textbook case is a user who changes the identifier in the URL and sees someone else's resource: offer-details-5232469.html becomes offer-details-5232470.html, and the page displays even though it belongs to another person's account.
Here, the suspicion was reinforced by the very profile of the scraping: requests came from the outside, without an authenticated session, and came back 200. Two possible readings, both bad. Either the application does not check record ownership and exposes account data to anyone who knows the URL scheme, in which case every ID in the range is a leak in itself. Or the application does require authentication, and the 200s return only an empty shell, but then a bot massively identified and classified heuristics by Cloudflare has no reason to persist: it would not scrape for days a section that turns it away.
This point does not depend on traffic volume: an attacker who slows down to one request per minute still walks the whole catalogue, and no amount of edge filtering will fix a missing authorisation check on the application side. Verifying it therefore ranked above deploying any security rule, and we said so before proposing the rules. If those pages require authentication, the scraping becomes pointless and the filtering layers become a comfort rather than a necessity.
This is a recurring pattern worth stating plainly. Edge filtering buys time and raises cost; it does not replace an authorisation check.
When a diagnostic surfaces something the application should be doing instead, saying so is part of the job, even when it shortens the engagement.
What this case shows
- A unique visitor count that rises suddenly while page views stay flat is a signature. Traffic that does not navigate is not an audience. Before investigating anything else, compare those two curves.
- Source geography is the first thing an attacker dilutes. Country and network readings are cheap to defeat with a residential bot pool. Protocol coherence, the agreement between the claimed browser, the negotiated HTTP version and the headers actually sent, is far harder to forge and far more discriminating.
- A quiet origin server proves nothing. Against cacheable URLs or assets, scraping is served by the edge and stops appearing in server monitoring. And when it does appear, load is not the most expensive part: here, the goal was price harvesting, commercial espionage causing direct harm to the administrator.
- Measure with the Log function, then block. Every rule starts in Log action, long enough to measure the legitimate residual, then gets refined before moving to real blocking actions. The cost of a badly scoped security rule is paid by real customers, worldwide, within seconds of deployment: a few hours of measurement is cheap insurance.
If you are seeing the same shape on your own zone, the two related guides below cover the fixed parts of this work in tutorial form: rate limiting and bot rules on sensitive endpoints, and separating crawlers you want from crawlers you do not.
Frequently asked questions
Why did the country and ASN breakdowns show nothing?
Because the scraping wave ran through rotating residential bots located in the markets the site actually serves. Consumer ISPs carried more of the traffic than datacenter networks, so every geographic and network reading looked plausible. Source location is the first dimension an attacker dilutes, because it is the cheapest one to fake. Protocol coherence is much harder to disguise, which is why crossing browser, HTTP version and TLS version made the wave readable in one query.
How can a browser user agent over HTTP/1.1 be a reliable signal?
A real Chrome negotiates HTTP/2 through ALPN against a Cloudflare edge and will not fall back to HTTP/1.1 on its own. A user agent claiming Chrome while speaking HTTP/1.1 is almost always an HTTP library wearing a browser string. The signal is not perfect: some corporate proxies downgrade connections, which is exactly why the rule should use a Managed Challenge rather than an outright block.
Why does per-URL rate limiting fail against a scraper?
Because a scraper enumerating a catalogue spreads its requests across tens of thousands of distinct URLs. In this case the fifty most-hit paths accounted for less than four percent of the blocked volume, and every individual page sat at a few hundred hits over two days, far below any sane threshold. Rate limiting has to aggregate by URI pattern or by source IP, never by exact URL.
Does the origin server load tell you whether you are being scraped?
Not necessarily. Here, the first waves brought the origin server down, and it was those slowdowns that alerted the administrator. But once the edge cache warms up on cacheable URLs or assets, most scraping requests are answered by Cloudflare and no longer reach the origin: monitoring can go quiet again while the content keeps being harvested. Above all, the indicator that matters is not server load but what the content is used for: in this case, the bots were collecting prices, genuine commercial espionage with a significant economic impact.