llms.txt: What It Is and How to Check if a Website Has One (2026)

What llms.txt is, and what it is not

The llms.txt proposal was introduced by Jeremy Howard of Answer.AI in September 2024. It describes a Markdown file at /llms.txt that helps tools and people discover useful site content. Its structure starts with an H1 title, can include a blockquote summary and introductory text, and uses H2 sections of links with optional descriptions. An optional /llms-full.txt can provide combined content rather than just the index.

# Example documentation

> A short guide to this project.

## Documentation

- [Getting started](https://example.com/docs/start): Installation and first steps.

## Optional

- [Release notes](https://example.com/releases): Historical changes.

llms.txt is a discovery convention, not an access-control or licensing mechanism, and grants no training permission. It is not robots.txt, does not block requests, and does not replace authentication, a content licence or other rights signals. Publishing it provides no guarantee that a particular AI company consumes it or cites your content. Missing files do not imply permission or prohibition.

llms.txt vs robots.txt vs TDMRep vs noai meta

TDMRep is a W3C Community Group report, not a W3C Recommendation. None of these files or tags proves how a crawler behaves. Keep observed instructions separate from access controls and a rights assessment.

Check one site with curl

Use the exact origin, including its subdomain. These are commands for you to run on a reviewed public site; no live requests were made to prepare this guide. Review site terms and published request policies first.

# HEAD: inspect status, Content-Type and any Location header.
curl -sI --connect-timeout 5 --max-time 20 https://example.com/llms.txt
curl -sI --connect-timeout 5 --max-time 20 https://example.com/llms-full.txt

# GET: inspect headers and actual body. -S also shows curl errors.
curl -s -S -D - --connect-timeout 5 --max-time 20 https://example.com/llms.txt
curl -s -S -D - --connect-timeout 5 --max-time 20 https://example.com/llms-full.txt

A 200 response is only a starting point. Look for a text content type such as text/plain or text/markdown, a leading # H1 after whitespace or a byte-order mark, and plausible Markdown sections and links. A text type alone proves nothing. text/html, <!doctype html> or HTML tags often reveal an HTML soft-404: the server returned its normal error page with status 200. Compare suspicious content to the site's ordinary error page manually. A Markdown error page can fool simple heuristics too.

404 or 410 indicates a missing file at that URL. 401, 403, 429, 5xx and network failures are unknown, not confirmed absence. A 301, 302, 307 or 308 needs a reviewed Location destination. These examples do not follow redirects; after review, request the final public URL directly. HEAD may return 405 or differ from GET, so inspect GET before drawing a conclusion. Check llms-full.txt independently: its absence does not invalidate llms.txt.

Check many sites with Python 3 and CSV

For a batch of sites, save the script below as check_llms.py. Put one reviewed HTTPS origin per line in domains.txt. It makes two serial requests per origin, waits at least one second before each request, uses a 15-second socket timeout, refuses redirects, and reads at most 1 MiB plus one overflow byte per response. Increase the delay to respect site limits and longer published crawl delays; stop and retry later on 429 or server failures. This discovery example does not crawl linked pages or evaluate robots rules.

# Save as check_llms.py. Python 3, standard library only.
# domains.txt: one reviewed public HTTPS origin per line, no paths.
import csv
import json
import re
import sys
import time
from datetime import datetime, timezone
from pathlib import Path
from urllib.error import HTTPError, URLError
from urllib.parse import urlsplit
from urllib.request import Request, HTTPRedirectHandler, build_opener

CAP = 1024 * 1024  # At most 1 MiB plus one overflow byte read.
DELAY = 1.0       # Increase for site-specific limits.

class NoRedirect(HTTPRedirectHandler):
    def redirect_request(self, req, fp, code, msg, headers, newurl):
        return None  # Review redirects manually; do not probe new hosts.

OPENER = build_opener(NoRedirect())

def probe(url):
    row = dict(url=url, found="unknown", status="", bytes=0,
               link_count=0, note="", checked_at=datetime.now(timezone.utc).isoformat())
    request = Request(url, headers={"User-Agent": "LlmsDiscoveryCheck/1.0",
                                   "Accept-Encoding": "identity"})
    try:
        with OPENER.open(request, timeout=15) as response:
            row["status"] = response.status
            data = response.read(CAP + 1)
            row["bytes"] = len(data)
            if len(data) > CAP:
                row["note"] = "body_cap_exceeded"
                return row
            text = data.decode("utf-8-sig", errors="replace").lstrip()
            content_type = response.headers.get("Content-Type", "").lower()
            encoded = response.headers.get("Content-Encoding", "identity").lower()
            html = "html" in content_type or re.search(r"<\s*(?:!doctype\s+html|html|head|body)\b", text, re.I)
            if response.status != 200 or encoded != "identity":
                row["note"] = "unexpected_status_or_encoding"
            elif html or not text.startswith("# "):
                row["note"] = "html_soft_404_or_not_markdown"
            else:
                row["found"] = "true"
                row["link_count"] = len(re.findall(r"\[[^]\n]+\]\([^\s)]+\)", text))
                row["note"] = "markdown_candidate"
    except HTTPError as error:
        row["status"] = error.code
        row["found"] = "false" if error.code in (404, 410) else "unknown"
        row["note"] = "missing" if row["found"] == "false" else "http_error_or_redirect"
        error.close()
    except (URLError, OSError, ValueError) as error:
        row["note"] = type(error).__name__
    return row

def main():
    origins_file, csv_file, state_file = map(Path, sys.argv[1:])
    state = json.loads(state_file.read_text()) if state_file.exists() else {}
    origins = []
    for line in origins_file.read_text().splitlines():
        origin = line.strip().rstrip("/")
        if not origin or origin.startswith("#"):
            continue
        parts = urlsplit(origin)
        if (parts.scheme != "https" or not parts.hostname or parts.username
                or parts.password or parts.path or parts.query or parts.fragment):
            raise ValueError("Expected a reviewed HTTPS origin: " + origin)
        if origin not in origins:
            origins.append(origin)
    fields = ["url", "found", "status", "bytes", "link_count", "note", "checked_at", "event"]
    with csv_file.open("w", newline="", encoding="utf-8") as output:
        writer = csv.DictWriter(output, fieldnames=fields)
        writer.writeheader()
        for origin in origins:
            for path in ("/llms.txt", "/llms-full.txt"):
                time.sleep(DELAY)  # Serial requests, including across hosts.
                row = probe(origin + path)
                row["event"] = ""
                if row["found"] != "unknown":
                    previous = state.get(row["url"])
                    if previous is not None and previous != row["found"]:
                        row["event"] = "added" if row["found"] == "true" else "removed"
                    state[row["url"]] = row["found"]
                writer.writerow(row)
    temporary = state_file.with_name(state_file.name + ".tmp")
    temporary.write_text(json.dumps(state, indent=2) + "\n")
    temporary.replace(state_file)  # Replace only after a completed pass.

if __name__ == "__main__":
    main()

python3 check_llms.py domains.txt results.csv llms-state.json

CSV includes found, HTTP status, bytes read and link_count, plus URL, UTC check time, diagnostic note and event. Found is true, false or unknown. Link counts are an approximate count of inline Markdown links in the downloaded text, not a full Markdown parse; reference links and raw URLs are not counted. A Markdown candidate is not proof of proposal conformance. The same H1 heuristic is applied to llms-full.txt, so differently structured full-content files need manual review.

The timeout bounds socket operations, not the entire batch or a total wall-clock deadline. Body caps bound memory and mark oversized responses unknown, rather than counting a partial file as complete. Use this local script with a trusted list of public origins; it is not a public URL submission service with private-network destination protection.

Monitor llms.txt added or removed over time

Start with a weekly run and retain llms-state.json between runs. Archive each dated CSV rather than overwriting historical evidence. The script stores a baseline on the first known result, then records added for false to true and removed for true to false. Each path is tracked independently, so llms-full.txt changes do not imply llms.txt changes.

Only a reviewed Markdown candidate or an explicit 404/410 updates this example's state. Timeouts, redirects, HTML responses, failed requests and oversized bodies preserve the last known value. Review unknown results separately; an HTML replacement may be a real removal, but needs confirmation. For sensitive alerts, confirm a removal with a later successful check. The state file is replaced after a completed pass; run only one process against it at a time. A weekly check establishes the observation time, not the exact publication time. Content edits that leave presence unchanged do not produce events.

The easy option: AI Crawler Checker

AI Crawler Checker: does a website block GPTBot, ClaudeBot or Google-Extended? checks a list of domains and returns robots.txt verdicts for selected AI bot tokens alongside llms.txt and TDM observations. The documented llmsTxt output includes presence, bytes read and link count; llmsFullTxt includes presence and bytes read. Titles are null to avoid retaining names. Export the dataset as JSON, CSV or Microsoft Excel.

The README price is $2.50 per 1,000 domain rows with platform compute included, plus Apify's small per-run start fee. Store discount tiers pay less and the Store price is authoritative. Normal mode bills one domain event per returned domain row, including unreachable-domain reports. Monitoring bills one domain event per changed-field row. A silent first baseline and unchanged monitor runs emit no billable rows.

{
  "domains": [
    "example.com"
  ],
  "maxResults": 100,
  "monitorMode": true,
  "monitorKey": "weekly-llms-watch",
  "requestDelayMs": 1000
}

Keep the same monitorKey and schedule weekly in Apify. State persists in the named ai-crawler-policy-auditor-state key-value store. The first successful observation normally saves a silent baseline. Later changes can emit llms_txt_added and llms_txt_removed, alongside robots.txt AI bot verdict changes and other documented policy events. Unavailable resources retain prior values. llms.txt size-bucket changes use meta_changed; do not assume every edit is detected.

Inspect resources, errors, checkedAt and RUN_SUMMARY. The documented llms.txt body cap is 1 MB and llms-full.txt cap is 64 KB; byte and link counts describe only the read prefix when truncated. HTML soft-404 checks can miss other error formats. Use one schedule at a time per namespace; interrupted delivery can repeat events. Apify schedules and your chosen integrations handle repeat runs and downstream alerts. Observed bot verdicts are published-policy signals, not proof of crawler behavior or permission to reuse content.

Frequently asked questions

Does llms.txt grant permission to train an AI model?

No. llms.txt is a discovery convention, not an access-control or licensing mechanism. Its presence grants no training permission. Review applicable rights, site terms and other policy signals separately.

Where should I look for llms.txt and llms-full.txt?

Check /llms.txt and optionally /llms-full.txt at the exact site origin. Subdomains are separate. Inspect the final status, content type and body rather than treating HTTP 200 as proof.

Does a missing llms.txt block AI crawlers?

No. A missing llms.txt says nothing about crawler access. Inspect robots.txt for published crawl rules and keep discovery, access restrictions and content permissions separate.

Will publishing llms.txt make AI companies use my content?

There is no guarantee. The proposal describes a discovery format; publishing the file does not prove that any particular AI service reads it or that rankings or citations will improve.

Why did the bulk checker report an unknown result?

Redirects, timeouts, HTTP failures, oversized bodies and responses that fail the Markdown heuristic require review. The example preserves the previous state for these results instead of reporting a removal.