AI Crawler User Agents List and robots.txt Examples (2026)

AI crawler tokens: training, search and user fetches

This is a reviewed selection, not an exhaustive inventory of AI traffic. The robots column reports the operator's position, not a measurement of behavior. Follow each primary-source link for scope and updates.

Google-Extended does not control inclusion or ranking in Google Search. Applebot-Extended is also a training-use control rather than a separate crawler. Blocking these differs from blocking Googlebot or Applebot network access.

MistralAI-Index and MistralAI-Training are documented here for manual use, but are absent from the Actor's reviewed default registry. Pass them explicitly in bots to inspect custom tokens. Custom tokens retain unknown purpose in the Actor and do not become verified training classifications.

robots.txt example: block training, allow search

Merge this example with your policy after reviewing existing restrictions. It covers selected operators, not every training crawler. Explicit search groups below allow every path for those tokens; carry over any private-path restrictions you still need. Put confidential pages behind authentication.

# Selected training crawlers and training-use controls
User-agent: GPTBot
User-agent: ClaudeBot
User-agent: Google-Extended
User-agent: Applebot-Extended
User-agent: CCBot
User-agent: MistralAI-Training
Disallow: /

# Search crawlers
User-agent: OAI-SearchBot
User-agent: Claude-SearchBot
User-agent: PerplexityBot
User-agent: Googlebot
User-agent: Applebot
User-agent: MistralAI-Index
Allow: /

User-agent: *
Disallow: /private/

CCBot builds an open archive, so blocking it also stops that crawl contribution. User-triggered fetchers need separate review. OpenAI warns that robots rules may not apply to ChatGPT-User, and Perplexity says its user fetcher generally ignores them. A rule alone cannot promise total exclusion.

Block an archive but allow an excerpt

User-agent: GPTBot
Disallow: /archive/
Allow: /archive/public-excerpt/

Test both paths with a parser whose precedence you understand. Read RFC 9309 and the crawler checking workflow. Do not assume named groups inherit wildcard fallback restrictions. Duplicate groups and conflicting rules deserve explicit tests.

Publish the file at /robots.txt on each relevant origin. Keep it readable, archive the prior version and confirm the CDN serves the new body. A published file and a crawler's cached copy need not change at the same instant.

llms.txt, TDMRep and ai.txt signals

llms.txt is a proposal for making site information easier for language models to use. It is not an IETF or W3C access-control standard. Its presence does not grant a license or prove training permission. Fetch it separately:

curl --fail --show-error --location --max-time 15 'https://example.com/llms.txt'
curl --fail --show-error --location --max-time 15 'https://example.com/.well-known/tdmrep.json'

TDMRep is a W3C Community Group report, not a W3C Recommendation. It describes reservation and policy signals in HTTP headers, HTML metadata and a policy array at /.well-known/tdmrep.json. Values 1 and 0 mean reservation and no reservation under that protocol; legal effect needs separate assessment.

[{"location":"/","tdm-reservation":1}]

That example expresses a reservation for the root scope. Do not collapse a path-specific TDM entry, homepage metadata and a crawler Disallow into one permission flag. The auditor also detects a limited ai.txt-style wildcard disallow. This file convention is not a universal technical or legal standard.

Actor coverage and unverified crawler commitments

The default Actor registry contains 30 tokens. Besides the default-registry tokens in the table, it checks anthropic-ai, Meta-ExternalAgent, Meta-ExternalFetcher, FacebookBot, Bytespider, cohere-ai, cohere-training-data-crawler, YouBot, Timpibot, ImagesiftBot and omgili. Some are historical labels. Current purpose and robots commitments were not established for all of them from reachable operator docs, so they are not labeled verified current crawlers here.

The auditor checks robots.txt, llms.txt, llms-full.txt, TDMRep, ai.txt and selected homepage headers/meta tags. It does not observe real bot requests or audit every page. Its metadata parser does not execute JavaScript. A policy score is not a compliance assessment.

{
  "domains": ["example.com"],
  "bots": ["GPTBot", "OAI-SearchBot", "ClaudeBot", "Claude-SearchBot"],
  "monitorMode": true,
  "monitorKey": "publisher-policy",
  "emitBaselineOnFirstRun": false,
  "maxResults": 100
}

Keep bot selection and test path stable when comparing runs. A scope change resets the comparison baseline. Inspect RUN_SUMMARY when a scheduled run emits no rows.

Frequently asked questions

Which agents should I block to opt out of training?

Start with training tokens and use-control tokens documented by your chosen operators. No fixed list covers every crawler or downstream dataset.

Is Google-Extended the same as Googlebot?

No. Google-Extended is a use-control token with no separate HTTP user agent. Googlebot controls Google Search crawling.

Does llms.txt override robots.txt?

The proposal does not establish an override of robots.txt or a universal permission system. Keep the observations separate.

Are all Actor tokens verified current bots?

No. Its registry includes historical and less well-documented tokens. This guide distinguishes verified documentation from unknown current behavior.