# llms.txt and its very limited job

> llms.txt gives agents a curated map of a site, but current crawler evidence says it neither controls access nor improves AI search visibility.

`llms.txt` is a proposed Markdown index that helps a willing agent find a site's most useful material. It does not grant permission, block a crawler, submit pages to an index, change rankings, or make weak content more quotable. That limited job can still be useful, especially for technical documentation, but the file has been sold far beyond the available evidence.

I would not put `llms.txt` on an AI visibility roadmap as a growth project. I would treat it as cheap, experimental publishing infrastructure: generate it from sources you already maintain, keep it accurate, measure who requests it, and delete it if the maintenance cost exceeds the observed use. The distinction matters because teams have a habit of shipping the visible new file while leaving crawl access, server rendering, canonical pages, and documentation quality broken.

## llms.txt is a curated map for a willing reader

The file tells a reader what a site or project is and points that reader toward selected resources. Jeremy Howard's proposal at llmstxt.org places it at `/llms.txt` and gives it a deliberately simple Markdown shape: one required H1, an optional blockquote summary, optional explanatory prose, and H2 sections containing lists of named destinations with descriptions. A section named `Optional` marks material that a reader may skip when context is tight.

That design solves an orientation problem. A documentation site may contain tutorials, API references, migration notes, generated pages, archived versions, and thousands of navigation links. An agent that already knows to fetch the file can get the author's short list before spending requests and context on the rest. A developer can also ask a coding agent to load the file explicitly. In that workflow, the value comes from curation, not from the filename itself.

The proposal also discusses Markdown versions of full pages and larger context bundles. Those are related ideas, but they are not the same artifact. The root file is an index. It normally contains descriptions and destinations, not the entire site copied into one enormous prompt. Calling any giant text export an `llms.txt` file removes the one useful constraint in the proposal: a human owner chooses what deserves attention.

Nothing in the format tells an agent how it must process the file. The proposal says processing depends on the application. That sentence is easy to miss and hard to market, which explains much of the confusion. A conforming file can sit untouched forever because a crawler has no obligation to discover, fetch, parse, trust, or follow it.

## It does not control crawling or confer permission

`llms.txt` has no access-control semantics. A line in it cannot allow a blocked bot, deny a training crawler, opt content out of model training, set a crawl rate, or authorize reuse. If you need to express crawler access, use the mechanism that the crawler vendor documents, usually `robots.txt`, and remember that robots rules are voluntary instructions rather than authentication. Use login controls when content must stay private.

The official crawler documents make the separation concrete. OpenAI tells publishers to manage OAI-SearchBot access through `robots.txt` for ChatGPT search and to address GPTBot separately for potential training. Anthropic documents ClaudeBot, Claude-User, and Claude-SearchBot as different agents with different purposes, all controlled through their own `robots.txt` groups. Perplexity documents PerplexityBot for search indexing and Perplexity-User for user-requested retrieval. None of those instructions tells publishers to use `llms.txt` as a permission file.

Do not put legal terms, licensing language, secrets, internal endpoints, or hidden instructions in the file and assume machines will obey them. Public text at a predictable path is public text. If your policy requires enforceable controls, put the policy in human-facing terms and enforce access at the application, identity, or network layer.

There is another operational consequence. An agent may treat the file as untrusted input, ignore its prose, or accept only its destinations. That is sensible. A compromised `llms.txt` could direct an agent toward hostile material or contain prompt-shaped instructions. Keep the file boring, version-controlled, generated from an approved inventory, and limited to descriptions that help a reader choose a resource.

## Adoption has outrun actual readership

The best broad log evidence currently available says publication is common in a technical sample while requests remain rare. Ahrefs examined 137,210 domains using its Web Analytics product that received traffic in May 2026. It found valid files on 28 percent of those domains, about 38,000 sites, but warned that its customers skew technical and SEO-aware. Treat that percentage as an upper bound for the wider web, not a universal adoption rate.

Among the sites with a valid file, 97 percent received no request for it during the month. The remaining group, roughly 1,100 domains, produced about 22,000 requests. Bots made 96 percent of those requests, but named AI tools accounted for only 19.5 percent. AI retrieval bots, the category closest to the usual promise of better AI search visibility, accounted for 1.1 percent of requests to the files that got any traffic. Training crawlers accounted for 5.3 percent, assistants for 2.5 percent, and agents for 10.5 percent.

The denominator is the important part. Saying that AI tools were the largest identifiable readership sounds encouraging until you remember that 97 percent of published files got no traffic at all. Even a request is only evidence that a client fetched bytes. It does not prove the client parsed the Markdown, followed a destination, used the content in an answer, cited the site, or changed a ranking. Each step requires its own observation.

The same study examined requests for missing files that returned 404. It reported no AI-bot share in those 404 requests. Its inference was that AI systems were not systematically probing every domain root for the file. Existing files were probably discovered through a reference, a directory, a platform feature, or a user instruction. That result fits the proposal's strongest present use: a resource that an informed agent or developer requests on purpose.

This is one dataset, not a law of the web. The sample is biased, one month is short, user agents can be spoofed, and a fetch cannot reveal downstream use. Still, a measured ceiling beats a sales claim. Anyone promising an automatic citation lift from the file needs controlled evidence that separates `llms.txt` from content changes, ordinary crawl access, and the rest of the release.

## A crawler request is not an AI citation

Teams routinely blur discovery, retrieval, parsing, use, and referral into one event. They are five different events, and confusing them produces fake success. A server log can prove retrieval from your origin. It cannot prove that the named company sent the request unless you verify the source as the vendor recommends, and it cannot show what happened after delivery.

The chain looks like this in practice. A client must discover the file, request it successfully, recognize its format, select one or more listed resources, retrieve those resources, and use their content in a response or task. A referral or citation may follow, but many agent tasks have no browser click at all. Failure at any stage breaks the hoped-for outcome while leaving earlier stages looking healthy on a dashboard.

User-agent names also describe intent imperfectly. Training crawlers, search indexers, and user-triggered fetchers do different work. OpenAI and Anthropic publish separate names precisely because a site owner may want search discovery while refusing training, or may allow a user's explicit retrieval while limiting bulk crawling. Aggregating every request with `bot`, `GPT`, `Claude`, or `AI` in its name hides those choices.

I have seen teams celebrate a bot spike that came from their own validator, deployment check, or link-preview service. The Ahrefs study found that tools auditing or studying `llms.txt` generated a meaningful share of requests. A readiness scanner asking for the file proves that the scanner works. It says nothing about whether the assistant your customers use has consumed the file.

Define evidence before launch. A useful progression is: a verified crawler fetched the file, the crawler then fetched a listed destination, an assistant cited or used that destination for a stable test prompt, and qualified referral traffic or task completion changed. Most sites will only be able to measure the first two. That limitation should lower the budget, not loosen the definition of success.

## Existing web files do different jobs

`robots.txt`, a sitemap, canonical markup, structured data, and `llms.txt` are not competing versions of the same feature. Each answers a different question. Replacing one with another leaves a gap that the new file cannot fill.

A `robots.txt` group tells a crawler which paths it may request under that crawler's interpretation of the Robots Exclusion Protocol. It does not keep a URL secret, and blocking a page can prevent a crawler from seeing page-level `noindex` instructions. A sitemap supplies discoverable canonical URLs and optional metadata to search systems that support its format. Google explicitly describes a sitemap as a hint, not a guarantee of crawling, indexing, or ranking.

Canonical markup helps a search system choose among duplicate or near-duplicate URLs. Page-level robots metadata can restrict indexing or snippets for crawlers that fetch and support it. Structured data describes entities and page content using vocabularies understood by particular consumers. Ordinary internal links let both people and crawlers move through the site and remain the least exotic discovery mechanism.

The proposed file answers a narrower editorial question: if a capable reader has limited context and wants the authoritative material, which resources should it read first? It can complement the other mechanisms because it carries selection and short descriptions. It cannot repair their configuration.

That distinction gives you a sane order of operations. Make public pages reachable, return useful server-rendered HTML, choose canonical URLs, maintain internal links and a sitemap where appropriate, set crawler policy intentionally, and publish accurate content. Add the curated agent index after those basics work. An extra file at the root cannot compensate for an application shell that returns empty HTML to a client without JavaScript or a firewall that serves the relevant bot a 403.

## Documentation sites have the clearest use case

A documentation owner should bother when agents are already part of the product's user workflow and the authoritative corpus is hard to select. Libraries, APIs, developer tools, data catalogs, and technical standards often qualify. Their users ask coding agents to read docs, compare versions, find a migration path, or assemble context for a change. A short maintained index can save those users manual selection.

The file earns its keep when four conditions hold. You know which resources are authoritative, those resources have stable destinations, an agent can retrieve useful text from them, and your publishing system can update the index with the docs. If a release changes a versioned API reference, the same release should update or regenerate the file. Stale curation is worse than no curation because it gives an agent confident directions to the wrong material.

A second credible use is a controlled agent workflow that you own. Suppose your support assistant, coding agent, or internal research tool checks a supplier's `llms.txt` by design. The standard path and predictable structure reduce integration work. In that case you have an actual consumer, so you can test parsing behavior, error handling, destination selection, and the effect of file changes.

The benefit is weaker for a small brochure site with a dozen well-linked pages. A crawler can already discover the pages, and an agent presented with a specific page does not need a second index. The file may still cost almost nothing if the CMS generates it, but almost nothing is the correct budget. Do not schedule a content project to hand-summarize pages that already have clear titles, headings, and navigation.

E-commerce presents a harder case. A hand-curated file cannot keep pace with a large, changing catalog, and it has no agreed schema for inventory, price, availability, or variants. Product feeds and supported structured data address those facts more directly. An index may point to buying guides or policy pages, but calling it a product-discovery solution invites stale prices and missing products.

## Skip it when the file becomes ceremonial work

You should skip `llms.txt` when nobody can name its consumer, owner, update trigger, and success signal. That combination usually means the team wants an AI-readiness badge, not functioning infrastructure. A file generated once during a marketing sprint will age quietly until an agent finally reads the wrong version.

Skip the project if it competes with crawl failures you already know about. Fix repeated 403 responses, accidental bot blocks, broken canonical tags, thin documentation, unstable URLs, client-only content, and missing internal links first. Official crawler guidance from OpenAI, Anthropic, and Perplexity focuses on access through `robots.txt`, reachable pages, and bot protection because those controls sit on the actual retrieval path.

Do not buy a large consulting package whose main deliverable is this file. The syntax is too small to justify it. Judgment about the authoritative corpus can require work, but that work should improve navigation, documentation ownership, and content quality for humans too. If the only output is a new index no confirmed consumer requests, the return is speculative.

I would also skip hand-maintenance on a news site or busy blog unless analytics show agent demand for a stable subset. A chronological feed, sitemap, topic pages, and clean article HTML already expose the corpus. Selecting every article again in Markdown creates another editorial queue with no clear reader.

A Team & AI Audit should treat `llms.txt` as a minor distribution experiment, never as evidence that a company has transformed how it builds or sells. The expensive questions are whether your team has useful agent workflows, reliable source material, appropriate access controls, and measurements tied to business outcomes.

## A correct file should be short enough to inspect

A conforming file needs an H1 and may add a blockquote, explanatory prose, and H2 lists. The example below uses replacement tokens instead of live destinations. Replace every token with the canonical destination you own, then serve the result as UTF-8 plain text at the site root.

```markdown
# Example Cloud API

> An API for processing queued jobs. Use the current v2 documentation unless a project explicitly targets v1.

The API reference defines request fields. Tutorials explain workflows but do not override the reference.

 ## Start here

- [Quickstart](QUICKSTART_URL): Create credentials and submit a first job
- [API reference](API_REFERENCE_URL): Current v2 endpoints, request fields, responses, and errors
- [Authentication](AUTHENTICATION_URL): Credential scope, rotation, and failure behavior

 ## Operations

- [Rate limits](RATE_LIMITS_URL): Limit headers, retry rules, and quota behavior
- [Status and errors](ERRORS_URL): Error codes and recovery guidance

 ## Optional

- [Changelog](CHANGELOG_URL): Release history and migration notices
- [Examples](EXAMPLES_URL): Complete sample applications
```

This file is intentionally not a sitemap. It omits the company page, press posts, every SDK version, and duplicate tutorials because they do not help an agent answer the primary documentation questions. It also avoids commands aimed at the model. The descriptions state scope and authority in ordinary editorial language.

Do not copy the example's content blindly. Copy its shape. Give the project an unambiguous name, write a summary that prevents the most likely category mistake, and list the smallest set that covers the common tasks. Put secondary resources under `Optional`. If you need hundreds of destinations, improve the hierarchy or create scoped files for separate documentation areas.

Validate the response as well as the Markdown. The root destination should return `200`, a text content type, and the expected body without a login page, redirect loop, HTML error shell, or bot challenge. A pretty source file that your CDN converts into a 403 has no reader.

## Production ownership matters more than syntax

The safe implementation is generated, reviewed, and observable. Store the source beside the documentation configuration or derive it from the same approved navigation data. Make one team responsible for selection. Run a deployment check that fetches the public response and rejects missing destinations, duplicate entries, unexpected hosts, and an absent H1.

Treat changes like code changes. A destination added by a compromised CMS plugin can send agents toward an attacker. Restrict edits, review diffs, and alert on an unexpected hash or destination host. Keep executable-looking instructions out of descriptions. Consumers should treat the whole file as untrusted content, but publishers should not make that job harder.

Cache behavior deserves attention. A long CDN lifetime can preserve old version guidance after a release, while disabling caching creates needless origin traffic. Tie invalidation to documentation releases and use the same rollback process as the rest of the site. Check subdomains separately because the root file on the marketing host does not describe or control a documentation host automatically.

Do not expose private documentation through curation. Listing an unlinked but publicly reachable destination makes it easier to find. If a resource requires authorization, keep authorization on the resource and usually leave it out of the public index. Security through an obscure destination was already broken before `llms.txt`; the new index merely makes the mistake obvious.

Set a deletion rule when you set the launch rule. For example, keep the generated file while its maintenance stays automatic and quarterly log review shows either verified agent requests or direct use in a workflow you support. If generation breaks repeatedly and no consumer appears, remove it. Experimental infrastructure should be allowed to fail without becoming permanent ceremony.

## Measure behavior in your own access logs

Your origin or edge logs are the first place to check whether anyone requests the file. Search the normalized request path, then group by status, user agent, verified network identity when available, and time. Do not search only for known AI names because that hides unknown clients and does not separate your monitoring from outside traffic.

For a common combined access log copied to a local analysis machine, this command gives a rough first pass:

```bash
awk '$7 == "/llms.txt" {print $9, $1, substr($0, index($0, $12))}' access.log | sort | uniq -c | sort -nr
```

The output shape is a count followed by status, client address, and user-agent tail:

```text
18 200 192.0.2.40 "Mozilla/5.0 (compatible; ExampleBot/1.0)"
4 304 192.0.2.40 "Mozilla/5.0 (compatible; ExampleBot/1.0)"
2 404 198.51.100.7 "Mozilla/5.0"
```

This is triage, not attribution. The example addresses are documentation ranges, and the example bot is fictional. In real analysis, remove health checks and your own validators, retain status and cache data, and verify claimed crawler identity using the vendor's published method. A user-agent header alone can be copied by anyone.

Then look for follow-through. Within a reasonable session window, did the same verified client request destinations listed only or prominently in the file? Did those requests succeed? Did the pattern begin after publication? A canary destination can help, but it must contain useful public material rather than a trap or hidden instruction. Never infer use in an answer merely because two requests occurred close together.

Keep the reporting plain: unique verified clients, successful file fetches, subsequent destination fetches, errors, and any referrals associated with tested assistant queries. Compare against a pre-publication period if you have it. If all observed traffic comes from your deploy check and an SEO scanner, record zero confirmed consumer use. That is a result, and it may save the next quarter from more speculative work.

## Spend effort in proportion to the evidence

Publishing `llms.txt` is reasonable when it is a small output of documentation you already maintain or an input to an agent workflow you can name. It is unreasonable when it becomes a substitute for accessible pages, intentional crawler policy, clear information architecture, or proof that customers use AI tools to reach you.

The current evidence does not show a ranking benefit, a citation benefit, or routine discovery by major AI systems. It shows occasional fetching concentrated among a small share of published files, with agents and training crawlers more visible than AI search retrieval bots. That pattern may change. Build the file so a change in demand costs you little: automate it, keep it accurate, and retain logs.

There is a useful asymmetry here. A ten-minute generated index for clean technical docs can be a sensible bet because the downside is tiny and a known agent user may benefit. A multiweek AI visibility program centered on the same file is a bad bet because the evidence cannot support the expense. The bytes are identical; ownership, audience, and opportunity cost decide whether the work makes sense.

Revisit the decision when the consumer changes, not when a new checklist circulates. A vendor may document support, your own product may add an agent that reads the convention, or logs may show verified clients following listed resources. Any of those events justifies a fresh test. A competitor publishing a file does not. Neither does a site scanner changing a score from red to green. Those are observations about supply, while the business case depends on demand.

Set a maintenance budget in advance. For an automatically generated file tied to an existing docs build, that budget might be a small review during each release. For a hand-curated file spanning several teams, count the meetings, ownership disputes, broken-destination fixes, and security review. Those costs recur even when crawler traffic stays at zero. A cheap technical option can become expensive organizational work surprisingly fast.

Also separate a user benefit from a publisher benefit. A developer who tells an agent to load a curated documentation index may save time even if the site gains no referral, ranking change, or citation. That is a legitimate reason to publish. Record it as support for an agent-assisted customer workflow, not as search growth. Honest labels make later investment decisions much easier.

If you publish it, write down the consumer you expect and the observation that would prove you right. Review the logs after a fixed interval. Until a real reader appears, call the file what it is: a maintained experiment with a small, specific job.