Cloudflare just put a tollbooth on the open web for AI crawlers—shifting from assumed consent to explicit permission and, in some cases, payment. Here’s what changed, why it matters, and how brands and publishers can adapt.
Cloudflare now blocks AI answer-engine crawlers by default and is piloting a Pay-Per-Crawl program that lets sites charge for access. The company also called out alleged ‘stealth crawling’ by Perplexity, delisting it from verified bots. Behind the headlines is a larger reset: AI answers are re-intermediating discovery, compressing referral traffic in classic search. This article explains the technical changes, competing claims about traffic impact, economic implications for publishers and creators, and a pragmatic playbook to stay visible—owning distribution while optimizing for both humans and machines.
If the last decade was the era of “opt-out” for data collection and crawling, the next may be the era of opt-in. On July 1, 2025, Cloudflare—the connectivity cloud that sits in front of a huge portion of the public web—announced it would begin blocking AI web crawlers by default, unless a site explicitly allows them or enrolls in a new Pay-Per-Crawl program. The company’s framing is straightforward: AI answer engines and model builders should respect publishers’ preferences and, when appropriate, compensate content owners for the data they extract.
Days later, Cloudflare published a forensic write-up alleging that Perplexity—the AI search startup—had been evading robots.txt and other no-crawl signals by rotating IPs and disguising itself, behavior Cloudflare labeled “stealth crawling.” It then delisted Perplexity from its verified bot list. Perplexity publicly disagreed with Cloudflare’s claims, arguing misidentification and blaming a third-party service in part. Regardless of who’s right on the specifics, the clash clarified a bigger reality: the norms that governed web crawling for decades are being stress-tested in the AI era.
This piece isn’t a courtroom brief. It’s a practical map for brands, publishers, and creators who need to stay visible and solvent while discovery fragments across social feeds, classic search, and AI answers. We’ll look at (1) what Cloudflare changed and how it works, (2) what the data says—on both sides—about AI’s impact on traffic, (3) where the economics are heading, and (4) how to adapt your content strategy, automation, AI editing, scheduling, and analytics so you’re not just defending the old game, but playing the new one.
Cloudflare’s July announcement did two big things:
For site owners, this means controls are now closer to the edge—tighter than a robots.txt hint and, in theory, harder to ignore. For AI companies, it means a world where billions of pages now require explicit consent (or a micropayment) to train models or fuel answer engines.
To demonstrate why controls are necessary, Cloudflare says it seeded hidden test domains with hard no-crawl directives and then observed Perplexity fetching content from those sites anyway, often via rotating IPs and user-agents that didn’t match Perplexity’s declared crawler. The company claims this behavior hit tens of thousands of domains at “millions of requests per day.” Perplexity replied that Cloudflare’s analysis was flawed and that third-party services contributed to the signatures in question. Cloudflare, unconvinced, removed Perplexity from verified bots and tightened blocks.
Even if you never use Perplexity, the episode marks a line in the sand: the gentlemen’s agreement around robots.txt—born in 1994—isn’t strong enough on its own when the incentives for AI data are this high. Enforceable controls (and economics) are replacing “please don’t” notes.
This is where takes get spicy—but let’s anchor in the best available, recent reporting.
Similarweb trendlines (as summarized by TechCrunch, Search Engine Roundtable, and Press Gazette) show zero-click searches increasing since Google rolled out AI Overviews: from around 56% (May 2024) to roughly 69% (May 2025). News referrals reportedly fell from about 2.3B at peak in mid-2024 to <1.7B by May 2025, while news prompts in ChatGPT grew 212% over the same period.
Pew Research Center (July 22, 2025) found users are less likely to click when an AI summary appears. That doesn’t mean “no clicks,” but it does support what many publishers feel: fewer click-outs when an AI box answers the intent up top.
Google’s response (through Head of Search Liz Reid) is that total outbound clicks are “relatively stable” year-over-year, while acknowledging distribution is uneven (some site types up, others down). In other words: overall pie size may not be collapsing, but the slices are shifting in ways that don’t always favor news or reference publishers.
It’s fair to say the evidence is mixed at the macro level and acute at the publisher level. If your content sits in categories that AI Overviews summarize cleanly (definition queries, simple how-tos, commodity news), your exposure to zero-click outcomes is higher. If your content resists compression (original reporting, deeper analysis, strong multimedia), you may be more resilient. None of this is radically new in search; featured snippets and rich results have long abstracted answers. AI Overviews simply accelerate and generalize that trend across more queries.
This is not a “kill AI” motion. Think of it as pricing and permissions entering the web-to-AI supply chain. For years, the grand bargain of search was: you let engines crawl your page; in return, they send clicks. AI answer engines complicate that bargain; if a synthetic paragraph answers the query, the click incentive can weaken. Cloudflare is nudging the industry toward a new bargain: either clicks, or compensation, or both.
Economically, two paths emerge:
Cloudflare’s Pay-Per-Crawl doesn’t solve everything (scope, enforcement, fair rates), but it gets the billing primitive into the stack—important if we’re going to have a functioning market instead of a tragedy-of-the-commons sprint.
Legacy signals: robots.txt, noindex, nofollow, and crawler user-agents have always been voluntary. Good actors comply; bad actors don’t. That’s the core weakness.
What Cloudflare adds: Rules at the edge (before your origin) to block or bill specific AI crawlers; verified bot lists; and heuristics to detect masked agents (rotating IPs, non-declared user-agents). In Pay-Per-Crawl, the edge negotiates payment using headers; if absent, it returns HTTP 402 with pricing.
Detection reality: Even strong heuristics can produce false positives/negatives. That’s part of Perplexity’s rebuttal. But moving enforcement from best-effort metadata to active gating is still a step toward operational control.
What this isn’t: It’s not DRM; it doesn’t encrypt your pages. It’s a gate for known (or suspected) crawlers and a meter for those who choose to pay.
No one party’s claims fully settle the matter. But we don’t need perfect consensus to make practical decisions. We need a working strategy.
You can’t control macro distribution, but you can control what you publish, how it’s structured, where it travels, and how you measure.
Your paid search and SEO won’t vanish, but their marginal returns are changing as answers compress intent. That makes the brand layer—distinct POV, distinctive visuals, durable experiences—more critical. Answer engines flatten commodity claims; identity differentiates.
Your moat is voice and community. As answers abstract topics, process, backstage, and point-of-view become scarcer and more valuable.
You have leverage. You also have pressure.
Cloudflare’s CEO Matthew Prince frames three futures: content collapse, oligarchic control, or a cooperative model where AI companies compensate creators—akin to music streamers paying rights-holders. His broader point isn’t moralism; it’s market design. If we want abundant, high-quality content, we need to align incentives so its creation is sustainable.
Should we block every AI crawler?
Not necessarily. Decide by business model and category. If you’re highly dependent on ad impressions and your pages are easily summarized, you’ll skew toward blocking or pricing. If you’re brand-led and want broad reach, you may allow verified crawlers that cite sources.
Does AEO replace SEO?
No. Think AEO + SEO. SEO brings you intent across classic SERPs; AEO helps you surface in the answer layer with attribution. Structured blocks and schema are the connective tissue.
What if Google says clicks are stable?
That can be true in aggregate and still mask category-level pain. Use your own analytics. Pew’s study shows click-through probability falls in the presence of AI summaries; Similarweb shows more zero-click outcomes. Your mileage depends on query mix and content type.
Could this hurt smaller AI startups more than giants?
Yes—content costs and compliance tooling may favor incumbents. That’s why transparent licensing frameworks (and reasonable rates) matter if we want a healthy multi-vendor AI ecosystem.
At the edge (Cloudflare or equivalent):
On your site (CMS + data):
In content operations:
In partnerships:
We’re in another cycle of re-intermediation. First, the homepage lost primacy to search; then search shared it with feeds; now AI answers run a traffic layer above both. None of these are permanent monopolies. The actors change, but the job stays the same: build systems that let your work be seen, credited, and chosen wherever attention flows.
Cloudflare’s move doesn’t end the story. It just inserts the price and permission levers the web never had for this specific phase. From here, the market will test outcomes: some AI firms will pay; some will comply without paying; some will resist and be blocked; publishers will tune their policies; the numbers will tell the tale.
If you’re a brand or publisher, the answer isn’t to wager everything on one surface. It’s to diversify discovery, own distribution, and engineer your content for both humans and machines—with structure that machines can cite and voice that humans want to follow.
No panic. No partisanship. Just clear-eyed adaptation.
Read our articles Meta Follows YouTube: Facebook Cracks Down on Unoriginal Content and Did YouTube Just Ban AI Content? to learn more about this cultural shift.
Alberto Luengo is the founder and CEO of Rkive AI. He writes about the collision of content strategy, automation, and platform shifts—translating technical change into practical playbooks for creators, brands, and enterprises.