Cloudflare split Search/Agent/Training bot controls are live — and your web-fetching agent just became a target

Cloudflare now lets site owners block AI crawlers by category — Search, Agent, and Training — separately, and it blocks Agent and Training bots on ad-supported pages by default. If you ship an agent that fetches the open web with a declared user-agent, expect more 403s and challenge pages starting now, with new domains auto-enforced from September 15. If your agent hides behind a browser UA, nothing changes yet — which tells you how this actually plays out.

What actually changed in the bot config?

The old world was binary: allow bots or block bots. The new split treats three behaviors differently. Search crawlers (index-and-link-back) get treated more leniently than Agent crawlers (fetch-on-behalf-of-a-user, right now) and Training crawlers (scrape-for-model-weights). A publisher can now say "Google Search yes, your agent no, training absolutely not" without a single robots.txt rule.

The default is the part that bites. On ad-supported pages, Agent and Training bots are blocked unless the operator opts them back in. So the burden flipped: you no longer get access until told no — you get denied until someone whitelists your category.

What breaks if you're shipping an agent?

Anything that does real-time retrieval against a Cloudflare-fronted site with an honest AI user-agent. That's a big chunk of the web — Cloudflare sits in front of roughly a fifth of it. Your RAG pipeline that fetches a URL a user pasted? That's an "Agent" fetch, and it can now get a 403 or a managed challenge even though a human asked for it in real time.

Here's the gotcha nobody's saying out loud: the honest move — declaring `MyAgent/1.0 (+https://...)` — is the one that gets you blocked. Spoofing a Chrome UA sails through until Cloudflare's fingerprinting catches up. So the policy currently punishes well-behaved crawlers and rewards the ones lying about who they are. That asymmetry is the whole story.

If you're building on this, assume your agent needs a fallback: cache aggressively, respect the block instead of retrying into a challenge loop, and surface "this site blocks AI agents" to the user rather than silently returning garbage. Don't build a product that depends on unrestricted real-time fetch of arbitrary sites — that assumption is now false.

Is this actually about training or about money?

It's about money. The distinction between Agent and Training exists so publishers can keep the search traffic that sends them visitors while blocking the fetches that don't. Cloudflare's framing in the original report leans on "content independence," but the mechanism is a paywall for machines. Fine — that's a legitimate thing to want. Just don't confuse it with a technical safeguard against training data.

Should you change anything today?

If you run a site: the split controls are worth turning on, because now you can allow the crawlers that send you humans and block the ones that don't. If you build agents: treat any Cloudflare-fronted domain as opt-in only, and design for graceful denial.

Verdict: I'd use the publisher-side controls — they're the first bot policy that maps to how sites actually make money. But as an agent builder, this makes the open web meaningfully less open to fetch, and the enforcement currently rewards dishonest user-agents. Build your retrieval layer assuming the door is closed by default.

FAQ

Does this block agents that use a normal browser user-agent?

Not reliably, not yet. Cloudflare's default blocks target declared AI crawler categories. An agent spoofing a Chrome UA can still get through until behavioral fingerprinting flags it — which is exactly why the current design rewards dishonest agents.

When does auto-enforcement kick in for new domains?

New domains onboarding to Cloudflare get the Agent/Training block applied by default from September 15. Existing sites can opt into the split Search/Agent/Training controls now.

Will this break real-time retrieval where a user pasted a URL?

Yes, potentially. A user-initiated fetch still counts as an "Agent" request under this scheme, so it can be blocked or challenged. Design your pipeline to detect the block and tell the user, rather than retrying into a challenge loop.