Should You Block AI Crawlers? Training vs Search Bots (2026)

Should I block AI crawlers? Start with the business goal

Separate three decisions: whether to participate in future model training, whether to be discoverable through AI search, and whether an assistant may fetch a page when someone asks. A single blanket block can sacrifice the second and third goals along with the first. The framework below is a practical recommendation based on the operator purposes documented below, not a prediction of traffic or revenue.

For a public business site seeking discovery, start by reviewing dedicated training blocks and keeping search available. For material you must keep confidential, require authentication and authorization. Do not publish it behind a robots.txt rule. If server load is the problem, inspect request volume and paths before choosing a use-based ban.

Training vs search bots: use the exact operator tokens

These are robots.txt product tokens, not complete browser User-Agent strings. The table covers four operators used in the examples, not every AI service. Consult the longer AI crawler user-agent list before extending the policy.

Google is a reason to avoid calling every rule a pure training switch: disallowing Google-Extended also affects the grounding uses listed above. Blocking Googlebot at the network layer is a different action with search consequences. OpenAI's training and search settings are independent; Anthropic also documents separate bots for these purposes.

Block AI crawlers with robots.txt: three copy-paste choices

Choose one policy for https://example.com/robots.txt. These full-site examples cover only the named operators. They are alternatives, not three blocks to concatenate. Keep your Sitemap lines. Review existing groups before replacing them, and repeat the review for each origin and subdomain.

A named bot group uses its own rules rather than inheriting the wildcard group's path restrictions. If you need restrictions such as /private/ or /account/, repeat them in any named group you allow. Combine duplicate matching groups deliberately; the most specific matching path wins, and Allow wins a tie. See RFC 9309.

Choice 1: allow crawling of the public site

Use this when you accept collection by compliant crawlers for their documented purposes. This removes robots restrictions on the public site; check that this matches your intended scope.

User-agent: *
Disallow:

Choice 2: block selected training uses, keep search and fetchers available

This opts out of OpenAI and Anthropic training collection and Google's documented Extended uses. Search and fetchers remain open under the shown rules. It does not express a universal opt-out from every training dataset.

User-agent: GPTBot
Disallow: /

User-agent: ClaudeBot
Disallow: /

User-agent: Google-Extended
Disallow: /

User-agent: OAI-SearchBot
User-agent: Claude-SearchBot
User-agent: PerplexityBot
User-agent: ChatGPT-User
User-agent: Claude-User
User-agent: Perplexity-User
Allow: /

User-agent: *
Disallow:

Choice 3: disallow the named AI training, search and fetch tokens

Use this when you accept reduced AI discovery and assistant retrieval for these operators. Ordinary Google Search remains outside the Google-Extended block. Add network controls if denying fetches is required: the two user-fetcher exceptions in the table still apply.

User-agent: GPTBot
User-agent: ClaudeBot
User-agent: Google-Extended
User-agent: OAI-SearchBot
User-agent: Claude-SearchBot
User-agent: PerplexityBot
User-agent: ChatGPT-User
User-agent: Claude-User
User-agent: Perplexity-User
Disallow: /

User-agent: *
Disallow:

None of these files can retract material already collected or bind every downstream user. An Allow directive describes crawling, not a content license. Copyright and licensing questions require separate review. This is not legal advice.

Does Cloudflare block AI crawlers by default?

Documentation checked October 23, 2026. The AI bot controls docs describe separate Search, Training and Agent policies, with Allow, whole-zone Block and Block on pages with ads. Classification covers verified bots and additional unverified bots. Allow adds no block; it does not override other security rules.

The docs page, updated July 1, describes September defaults as ads-only Training and Agent blocks. The September 15 announcement recommends for ad-supported onboarding: Search allowed, Training set to Disallow AI Training, and Agent blocked on ad pages. Non-ad domains get all three allowed. Both enable Preference Sync; existing domains follow migration rules.

That announcement distinguishes Disallow AI Training, which preserves search access for Accountable mixed-use crawlers while blocking other training crawlers, from Block, which also stops mixed-use crawlers such as Googlebot. Ads-only Block also includes mixed-use crawlers on affected pages. It retires the legacy Block AI Bots and Managed Robots.txt settings. It also notes that Bing's robots-based training preference support is planned for 2027, so this setting alone does not yet convey that preference to Bing.

The October 22 robots.txt docs say Bot Preference Sync prepends managed rules when enabled with a restrictive policy; with all policies allowed, the file is unchanged. For Training set to Disallow, it adds directives and edge blocking that excludes mixed-use crawlers. Review the delivered file after saving, including interactions with your original groups.

These sources differ in their description of defaults. Use the later announcement and current robots documentation for the distinctions above, then verify the actual settings and responses for your domain. No account-specific rollout or configuration was tested for this guide.

robots.txt vs network-level blocking

robots.txt publishes instructions that compliant crawlers read. It is not access authorization or a firewall. A CDN, WAF or application can instead deny the HTTP request before serving the content. A permissive robots.txt can coexist with a denied page request; a restrictive file can coexist with a successful response to an ordinary client.

Do not identify a crawler from its User-Agent alone. Use the operator's published IP or verification method and your security logs. Perplexity's official docs recommend combining user-agent and IP checks. Anthropic cautions that IP blocking can obstruct reading robots.txt, so keep the published opt-out accessible when configuring enforcement. A denied request shows an access result, not necessarily which rule caused it.

For a search-friendly policy, align both layers: allow the intended search crawler in robots.txt and ensure your security policy permits its verified requests. For unwanted user fetches, use enforced restrictions where robots.txt is not honored. Neither approach establishes what an operator did with earlier copies.

How to verify your AI crawler policy afterwards

1. Save the intended policy, date and chosen tokens. Test the root and representative article, product and restricted paths. Record each expected allow or disallow before publishing.
2. Fetch the public robots.txt after the change, through the same hostname visitors use. Keep the response headers and body; inspect status, redirects and whether you received text or an HTML challenge.
3. Evaluate the delivered rules for each token and path. Check named-group precedence, duplicate groups and managed Cloudflare additions. The manual crawler policy audit guide includes a local Python check and parser limitations.
4. Inspect real bot requests in server or CDN logs. Verify identity, path, time, response status and the matched security action. A forged User-Agent test cannot prove how the real operator is treated.
5. Repeat after caches and operator updates. OpenAI and Perplexity document roughly 24-hour search-policy update windows. Keep failures separate from confirmed policy changes, and recheck after CDN configuration changes.

curl --silent --show-error --location --max-time 20 \
  --dump-header robots.headers \
  --output robots.txt 'https://example.com/robots.txt'
cat robots.headers
cat robots.txt

For a reproducible review, retain the origin, UTC time, final URL, HTTP status, policy body, selected token, tested path and expected result. A root-only check is insufficient for a policy with path exceptions. Search eligibility is not a promise of citations, visits or conversions.

The easy option: audit domains and monitor changes

The AI Crawler Policy Auditor accepts domains, selected bot tokens and an optional testPath. Its documented output includes observed robots rules, effective bot verdicts, root/test-path decisions, llms.txt presence and selected TDM signals. It evaluates the test path locally without fetching that page.

{
  "domains": [
    "example.com",
    "example.org"
  ],
  "bots": [
    "GPTBot",
    "ClaudeBot",
    "Google-Extended",
    "OAI-SearchBot",
    "Claude-SearchBot",
    "PerplexityBot",
    "ChatGPT-User",
    "Claude-User",
    "Perplexity-User"
  ],
  "testPath": "/news/",
  "maxResults": 10,
  "monitorMode": false
}

Enable monitorMode with a stable monitorKey for comparisons. The first successful observation saves a silent baseline unless emitBaselineOnFirstRun is enabled. Inspect resource errors and RUN_SUMMARY when there are no changes.

The Actor reports published signals received by its own client. It does not configure your firewall, impersonate verified crawlers, verify actual bot compliance or crawl the whole site. Its score is an informational policy measure, not legal advice or a rights determination.

Frequently asked questions

Should I block AI crawlers if I still want AI search traffic?

Consider blocking dedicated training crawlers while allowing search crawlers. Check mixed-purpose controls and firewall settings separately. Allowing access does not guarantee citations or visits.

Does blocking GPTBot also block ChatGPT search?

No. GPTBot and OAI-SearchBot have independent controls. ChatGPT-User is a separate user-request fetcher whose requests may not follow robots.txt.

Does Cloudflare block AI crawlers by default?

The September 15, 2026 announcement describes different recommendations for new domains depending on advertising. Existing settings can differ. Check your zone rather than assuming every AI bot is blocked.

Can robots.txt stop every user-triggered AI fetch?

No. Anthropic says Claude-User honors robots.txt, but OpenAI says robots.txt may not apply to ChatGPT-User and Perplexity says Perplexity-User generally ignores it. Enforce required access restrictions at the network or application layer.