Do AI agents read your llms.txt? We asked Claude Code, Codex and OpenCode about our own site
llms.txt is easy to publish, and most advice says to publish it. What's harder to find is whether an agent, asked a question about your product, actually goes there. Server logs tell you who fetched the file. We wanted to know what agents do when a user asks.
The test
We used our own site, spiritroom.app, because we know exactly what it serves. The homepage has <link rel="alternate" type="text/plain" href="/llms.txt"> and a rel="llms-txt" link in its head and links /llms.txt in its body, and our articles declare a Markdown copy with rel="alternate" type="text/markdown".
Three agent apps, each with only web access: Claude Code (WebFetch and WebSearch), Codex (its web search), and OpenCode (webfetch, with a small free model). Two questions, three runs each per app:
- A, general: “What does spiritroom.app offer, who is it for, and how do I get in touch with them? Check the site itself and answer briefly.”
- B, specific: “According to spiritroom.app, how many MCP tools is too many, and what do they recommend? Use the site as your source and answer briefly.”
We recorded every URL each app requested.
What each app fetched
| App | Question A | Question B | Markdown copy | First request |
|---|---|---|---|---|
| Claude Code 2.1.304 | 0/3 | 0/3 | 0/6 | Homepage, 6/6 |
| Codex 0.161.0 | 0/3 | 1/3 | 0/6 | Homepage (A), site: search (B) |
| OpenCode 1.18.34 | 0/3 | 1/3 | 0/6 | Homepage, 6/6 |
| All | 0/9 (0–30%) | 2/9 (6–55%) | 0/18 (0–18%) | — |
On question B, OpenCode read llms.txt third, after the homepage and a different article; Codex read it after a site: search and the homepage. Claude Code and OpenCode reached the right article in all of their runs on question B, by following links from the homepage, and once for Claude Code by a web search.
Find out what agents read when users ask about your product.
We run your users' real questions through Claude Code, Codex and the other apps they use, against your site, docs and MCP server, and show what each agent reads, what it misses, and what it gets wrong.
What it means
This matches what server-log studies find from the other side: coding agents are among the few clients that ever request llms.txt, but they use it when they happen to land on it, not as a first stop. See Ahrefs' llms.txt study and a nine-day log of agent data-seeking patterns. Our runs add the controlled view: same site, same questions, three apps, and the file was a fallback 2 times out of 18.
The rel="alternate" tags did nothing visible. Agents' fetch tools take a URL and convert the HTML themselves; none of them looked for the declared Markdown copy.
What to publish
- A homepage that answers the basic questions in plain HTML: what the product is, who it's for, how to get in touch. On question A, Claude Code and OpenCode fetched nothing but the homepage in every run, and Codex opened only the homepage.
- Descriptive links to key pages. Agents navigated by link text. “How many MCP tools is too many” as anchor text got them there.
- llms.txt and Markdown copies as extras. They're cheap, and an agent that lands on llms.txt gets a map. Just don't put anything only there.
- For product use, not docs, put the guidance in your MCP server. Agents using your API never visit your docs site; they read tool descriptions and instructions. See how to onboard AI agents through your MCP server.
Go deeper: Do agents read your MCP server instructions?, Which MCP channels agents use, Agent experience: how to measure it.
How we measured
- Apps: Claude Code 2.1.304 (only WebFetch and WebSearch allowed), Codex CLI 0.161.0 (live web search, read-only sandbox, MCP servers off), OpenCode 1.18.34 with the free GLM-5.3 Flash model (webfetch on, shell and edits off). Headless, 21 Oct 2026, about 21:00–21:30 (UTC+3).
- Evidence: every tool call each app made, from its own JSON transcript: the URL fetched or the search run.
- Site: spiritroom.app as served that day, with /llms.txt, head and body links to it on the homepage, and Markdown alternates on articles.
- Limits: one site, two questions, 3 runs per question and app, default models except OpenCode. AGENTS.md wasn't part of this test. Ranges are 95% Wilson intervals.
FAQ
Do AI agents read llms.txt?
Sometimes, and not first. In 18 runs where we asked Claude Code, Codex and OpenCode about our site, they fetched /llms.txt in 2, both times as a fallback after the homepage. Claude Code didn't fetch it in any of its 6 runs.
Does Claude Code use llms.txt?
Not in our runs. Asked about spiritroom.app, Claude Code's WebFetch went to the homepage in 6 of 6 runs and then followed links to the right article; it never requested /llms.txt, though the homepage links to it.
Do agents read Markdown versions of pages?
Not unprompted in our runs: 0 of 18 fetched a .md copy, although our articles declare one with a link rel=alternate tag. Agents fetched the HTML pages and let their fetch tool convert them.
Should I still publish llms.txt?
Yes, it's cheap and coding agents do use it when they land on it or are pointed to it. But don't rely on it: put the important answers on the homepage and link pages clearly, because that's the path agents took.
How do I make my docs readable by AI agents?
Make the homepage say what the product is and link to the key pages with descriptive anchor text, keep pages readable as HTML without scripts, and publish llms.txt and Markdown copies as extras. If agents use your product through an API, put the guidance in the MCP server too.
Find out what agents read when users ask about your product.
We run your users' real questions through Claude Code, Codex and the other apps they use, against your site, docs and MCP server, and show what each agent reads, what it misses, and what it gets wrong.