How to Check Which AI Crawlers a Website Blocks (GPTBot, ClaudeBot, Google-Extended)

Fetch robots.txt with curl

Start with the exact origin, including scheme, host and any non-default port. A policy on example.com does not automatically describe www.example.com. Save the headers and body together with your review date.

curl --silent --show-error --location --max-time 20 \
  --dump-header robots.headers \
  --output robots.txt 'https://example.com/robots.txt'
cat robots.headers
cat robots.txt

Inspect the final HTTP status and content type. An HTML error page with status 200 is not a usable policy. Record timeouts and server failures as unknown in your audit. RFC 9309 distinguishes unavailable 4xx responses from unreachable 5xx or network errors. See RFC 9309.

How robots.txt groups and longest matches work

A group contains user-agent lines followed by rules. Product-token matching is case-insensitive. RFC 9309 requires combining groups matching the same product token; the wildcard group is the fallback when none matches. Operators can document more specific token-selection behavior.

Within applicable rules, the most specific matching path, measured in octets, wins. At equal specificity, Allow should win over Disallow. Paths are case-sensitive. An asterisk matches a sequence and a dollar sign anchors the end. A longer Allow can create an exception to a block.

User-agent: GPTBot
Disallow: /
Allow: /public/

User-agent: *
Disallow: /private/

Here, GPTBot may fetch /public/report, but its root block disallows /private/report. Do not add the wildcard group to the named GPTBot group. Test more than the root path: one allowed path does not establish that an entire site is open.

A Python robots.txt checker with urllib.robotparser

Save the following as check_robots.py. Run python3 check_robots.py https://example.com/news/ https://example.org/. It makes one robots request per supplied URL and tests three tokens locally. Use only public sites you intend to inspect. Unknown is an audit result, not a crawler access decision.

import sys
from urllib.request import Request, urlopen
from urllib.error import HTTPError, URLError
from urllib.parse import urlsplit
from urllib.robotparser import RobotFileParser

BOTS = ("GPTBot", "ClaudeBot", "Google-Extended")
for raw in sys.argv[1:]:
    parts = urlsplit(raw if "://" in raw else "https://" + raw)
    if parts.scheme not in ("http", "https") or not parts.hostname or parts.username:
        raise SystemExit("Use public HTTP(S) URLs without credentials")
    origin = f"{parts.scheme}://{parts.netloc}"
    target = origin + (parts.path or "/")
    if parts.query:
        target += "?" + parts.query
    robots_url = origin + "/robots.txt"
    try:
        req = Request(robots_url, headers={"User-Agent": "PolicyReview/1.0"})
        with urlopen(req, timeout=15) as response:
            if response.status != 200:
                raise ValueError("Unexpected HTTP status")
            kind = response.headers.get("Content-Type", "").lower()
            body = response.read(1_000_001)
            if len(body) > 1_000_000 or "html" in kind or b"<html" in body.lower():
                raise ValueError("Too large or HTML: inspect manually")
        parser = RobotFileParser(robots_url)
        parser.parse(body.decode("utf-8-sig").splitlines())
        for bot in BOTS:
            print(origin, bot, target, "allowed" if parser.can_fetch(bot, target)
                  else "disallowed", sep="\t")
    except (HTTPError, URLError, ValueError, UnicodeError, OSError) as error:
        print(origin, "UNKNOWN", str(error), sep="\t", file=sys.stderr)

The script fetches the body itself rather than calling RobotFileParser.read(), so HTTP errors remain visible. It never fetches the test page. It is a local demonstration, not a service for untrusted URLs: add destination and redirect validation before exposing it to other users.

Python parser limitations and Cloudflare blocks

Version matters. CPython 3.13 uses first-matching-rule behavior and lacks RFC wildcard path handling and proper duplicate named-group merging. The 3.14 branch has newer matching logic. Check the source for your runtime; the module name alone does not establish correctness. Test mixed Allow/Disallow, wildcard and duplicate-group fixtures. Sources: 3.13 source, 3.14 source and Python: urllib.robotparser.

robots.txt is voluntary and is not access authorization. A CDN or web application firewall can return HTTP 403 despite permissive robots rules. Cloudflare has separate AI bot controls that need not appear in robots.txt. If you own the site, correlate path, time, verified bot IP and security event. A 403 alone does not identify the blocking rule. See Cloudflare: AI bot controls and Cloudflare: HTTP 403.

Changing curl's User-Agent to GPTBot does not make your IP an OpenAI crawler. It can show what your client received, but cannot prove the real bot receives the same response. Google-Extended has no separate HTTP user agent: it is a use-control token. Google: common crawlers and Google-Extended explains that scope.

Check many domains and monitor policy changes

Use one evidence row per origin, token and tested path. Keep the time, final status, body hash, parser version, selected rules and unknown reason. Compare against the previous successful observation; keep failures separate from policy changes. Cache each robots body for all local token tests, and space requests.

For path coverage, choose representative URLs such as an article and a product page. Avoid treating a root-only verdict as a complete site audit. Confirm actual access in logs when you control the site. Neither an Allow nor a missing rule is a license to reuse content.

The AI Crawler Policy Auditor accepts domains, optional bots and a testPath. It reports effective bot verdicts and root/test-path decisions, llms.txt presence, TDMRep entries and selected homepage headers/meta tags. It does not fetch the test path or crawl the full site.

{
  "domains": ["example.com", "example.org"],
  "bots": ["GPTBot", "ClaudeBot", "Google-Extended"],
  "testPath": "/news/",
  "maxResults": 10,
  "monitorMode": false
}

For changes, enable monitorMode with a stable monitorKey. The first observation is silent unless emitBaselineOnFirstRun is true. Inspect errors and RUN_SUMMARY; a failed observation is not evidence of an unblock. Its score measures observed signals, not legal protection.

Frequently asked questions

Does blocking GPTBot also block ChatGPT search?

No. OpenAI documents independent GPTBot and OAI-SearchBot controls. Review both tokens and the firewall policy.

Can robots.txt prove a crawler never accessed a page?

No. Confirm actual requests with server or CDN logs and verify the requester's identity.

Why do Python and another checker disagree?

Parsers and Python versions differ in group merging, wildcards and precedence. Compare the selected rules and exact tested URL.

Does an allowed result permit content reuse?

No. A parsed crawl preference is not a license. Review content rights and applicable terms separately.