Cloudflare Now Blocks AI Crawlers by Default. Decide Who Gets In.
- Decide separately whether search, agent and training crawlers should read your pages.
- New domains, new sites on existing accounts, and free-plan accounts with unchanged settings are covered.
- Blocking agents can prevent an AI from reading your pages during a provider comparison.
- My AI visibility method checks real buying questions and official domains.
- One engagement showed a directional change in recommendation rate; another stayed flat.
Cloudflare's defaults changed on September 15, 2026. Many businesses may be invisible to AI agents without having decided whether those agents should be able to read their pages.
I help companies decide how they should be understood on the web and by AI, then test whether that understanding is showing up. I have built for the web since 1994.
The takeaway
Check your crawler settings and choose deliberately which visitors can read your pages. Then measure AI recommendations separately from page access.
What changed in Cloudflare's crawler defaults?
Cloudflare began sorting automated crawlers into search, agent and training categories on July 1, 2026. Starting September 15, 2026, its default settings block agent and training crawlers on pages that carry ads, while search crawlers stay allowed. Northeast Times reported on the change.
The defaults apply to new domains, new sites on existing accounts, and free-plan accounts that have not changed their settings. Crawlers that do more than one job are treated under the strictest rule that applies to them. That means a crawler's other job does not exempt it from the stricter setting.
What does each crawler do?
Search crawlers index pages and bring referral traffic. Agent crawlers retrieve pages in real time when an AI acts for a user. Training crawlers gather content for AI model training. Since each role has different benefits, set access according to what you want that type of visitor to do.
A company may want search crawlers to send people to its pages and agent crawlers to read those pages during a provider comparison. Training access is another decision. Treating all automated traffic as one group can hide those different choices.
When agents are blocked, an AI acting for a user may be unable to read your pages when asked to compare providers.
How can I tell whether AI recommends my company?
I measure AI visibility by asking a fixed set of a category's real buying questions of ChatGPT, Claude, Perplexity, Gemini and Grok on a fixed schedule. I check official domains so a company with the same name does not count as a win. I also make the model explain its choice and keep records in one format, so week 11 compares cleanly with week 1.
Across 11 weeks, the AI recommendation win rate for one engagement rose from 6.2% to 36.2%. Trinzik ran the engagement on the platform I built. This is directional evidence, not a controlled study. In another program, I published content to fill the gaps, but the win rate did not move. I keep retesting instead of calling it a win.
How should I check whether my site is readable by agents?
I published a rubric for scoring a site's readability by agents, so any firm can score itself and get the same answer. Readability can help an agent access your pages, but it does not tell you whether the agent will recommend your company.
First, check whether the relevant Cloudflare settings have changed. Then choose which crawler types should have access. To test readability, score the site with my rubric and review the pages an agent would need during a provider comparison.
Common questions
Does blocking agent crawlers block search crawlers too?
Cloudflare's default settings keep search crawlers allowed while blocking agent and training crawlers on pages that carry ads. A crawler that does more than one job is treated under the strictest rule that applies to it.
Does allowing agents mean AI will recommend my company?
No. Page access and recommendation are different outcomes. To assess recommendation, track answers to buying questions and confirm that the company named is your official-domain business.
What should I do after scoring my site's readability?
After scoring, review whether an agent can read the pages it would need during a provider comparison. Then test AI recommendations separately with real buying questions and official-domain checks.
If you want help deciding which crawlers should read your pages and measuring what AI does with them, contact me.