AI discovery, measured
AI Search & Crawler Readiness Auditor
See how your public crawler policy and discovery files support AI search visibility. Keep search and citation access open while making a deliberate choice about model training.
Last reviewed: October 26, 2026
Free live website check
Live AI Search & Crawler Readiness Auditor
Can AI search engines discover and cite your site? Check 12 AI bots, robots.txt, llms.txt, sitemaps and homepage signals. Get a visibility score and ready-to-copy configs in seconds.
Open the standalone Live Auditor for scoring details and the full AI bot reference.
How the AI Search Visibility Score works
The 0-100 score weighs access for search and citation bots, homepage indexability and public discovery signals. Blocking training bots does not reduce the search access score. These are published signals, not a promise of inclusion or citation in AI answers.
| Signal | Maximum points | Calculation |
|---|---|---|
| Search & Citation Bot Access | 45 | 9 points each for OAI-SearchBot, ChatGPT-User, Claude-SearchBot, PerplexityBot and Amazonbot with open access including /. Mixed path access earns 5; blocked or unverified access earns 0. |
| Homepage Indexability & Meta Signals | 15 | 15 for an HTTP 200 HTML homepage without noindex or none in meta robots or X-Robots-Tag; otherwise 0. noai and noimageai are reported separately. |
| Explicit AI Crawler Governance | 15 | 10 for readable HTTP 200 robots.txt plus 5 for a named AI bot group. A clean open general policy earns 12 in total. Missing or unverified files earn 0. |
| llms.txt AI Discovery File | 15 | 12 for a nonempty text or Markdown llms.txt at HTTP 200; 3 extra for llms-full.txt. HTML soft 404s do not count. Save the starter template at /llms.txt to add discovery links. |
| Structured Discovery | 10 | 4 for sitemap.xml containing urlset/sitemapindex or an HTTP(S) Sitemap directive in robots.txt, 3 for a canonical link, 3 for valid JSON-LD. |
Scores of 80-100 indicate strong readiness, 60-79 developing readiness, 40-59 limited readiness and 0-39 low readiness under this rubric. llms.txt is an optional discovery convention, and noai/noimageai are policy signals whose support varies.
Each audit requests five resources over HTTPS, shares an 8-second deadline, follows at most three validated HTTP(S) redirects per resource and reads at most 256 KB per file. Rules use case-insensitive named bot groups before wildcard fallback, merge repeated groups, choose the longest matching path and favor Allow when specificity ties. No matching directive means default access; a missing robots.txt (404/410) is open by default. A fetch failure, HTML substitute or truncated robots file leaves access unverified.
The table's matched directive describes /. Partial access identifies a mix of allowed and blocked paths from rule-path probes; it is not an exhaustive crawl of every URL. The tool does not test firewall rules or impersonate each bot. Training control tokens such as Google-Extended and Applebot-Extended describe use of content, rather than separate search crawlers.
Protocol reference: RFC 9309: Robots Exclusion Protocol. Background on discovery files: The llms.txt proposal.
The 12 audited AI bots
The audit separates model training, search and citations, and user-requested fetches so you can choose an intentional policy.
| User-agent | Operator | Category | Role |
|---|---|---|---|
| GPTBot | OpenAI | Model Training | Model training crawler |
| OAI-SearchBot | OpenAI | Search / Citation | ChatGPT Search citation crawler |
| ChatGPT-User | OpenAI | User Fetch | ChatGPT user-initiated page fetch |
| ClaudeBot | Anthropic | Model Training | Model training crawler |
| Claude-SearchBot | Anthropic | Search / Citation | Claude search and citation crawler |
| PerplexityBot | Perplexity | Search / Citation | Perplexity search and citation crawler |
| Google-Extended | Model Training | Gemini training control token | |
| Applebot-Extended | Apple | Model Training | Apple Intelligence training token |
| Bytespider | ByteDance | Model Training | ByteDance LLM training crawler |
| CCBot | Common Crawl | Model Training | Open web training dataset crawler |
| Meta-ExternalAgent | Meta | Model Training | Meta AI training and indexing crawler |
| Amazonbot | Amazon | Search / Citation | Amazon and Alexa search/citation crawler |
Turn a single check into ongoing monitoring
Use the AI Crawler Policy Auditor on Apify to bulk-audit domain lists or schedule policy drift checks. Connect Official n8n Template #21572 for weekly email or Slack alerts. Review current Actor pricing and any platform fees before starting a bulk run.