GlossaryAI search
What is llms.txt?
Also called: llms txt, llms-full.txt
Definition
llms.txt is a proposed markdown file at a site's root, /llms.txt, that summarizes the site and links to its most useful pages for AI agents. It's voluntary, grants no access, and Google Search ignores it.
llms.txt, explained
Jeremy Howard proposed the format in September 2024 at llmstxt.org. The problem it targets is real: web pages are built for people, wrapped in navigation, scripts and ads, and hard for a model to read cleanly within a limited context. A short markdown file pointing to the important pages, ideally with markdown versions, is easier for an agent to use.
The format is simple. An H1 with the project or site name is the only required part. Then a blockquote with a short summary, optional paragraphs of detail, and H2 sections containing lists of markdown links, each with an optional note. Version 2 of the proposal, published in August 2026, also lets a file sit at a subpath and cover the pages under it.
The honest part: it's not a ranking file. Google's guide to optimizing for its generative AI features says you don't need new machine-readable files, AI text files, markup or Markdown to appear in Google Search, including its AI features, because Google Search doesn't use them. It doesn't block or allow anything either; that's robots.txt's job.
Where it helps is agent-driven use: coding assistants reading your docs, tools that fetch a site's llms.txt when a user points them at it, and documentation platforms that generate the file automatically. For developer tools and anything with an API or docs, it's a cheap, sensible addition.
It takes about an hour to write a good one. Keep it short, link to canonical pages, describe each link in a few words, and update it when your key pages change. Don't stuff it with marketing copy; an agent reading it wants facts.
Why it matters for founders
You'll hear llms.txt pitched as essential for AI visibility. It isn't, especially for Google. It's worth an hour if agents or developers use your docs, and not worth more than that.
Example
A developer tool publishes /llms.txt with its name, a one-line summary, and sections linking to the quickstart, API reference and pricing. A coding assistant asked to integrate the tool reads that file first.
Common mistakes
- Expecting it to improve Google rankings or AI Overviews.
- Treating it as an access control. It blocks nothing.
- Dumping the whole site's text into it instead of linking.
- Letting it go stale after pages move.
Sources
- llmstxt.org: The /llms.txt file
- Google Search Central: Optimizing your website for generative AI features
Checked
Related terms
- robots.txtrobots.txt is a plain-text file at the root of a domain that tells crawlers which URLs they may and may not fetch. It controls crawling, not indexing, and well-behaved bots follow it voluntarily.
- AI crawlersAI crawlers are bots run by AI companies that fetch web pages, either to collect training data, to build an index for AI search answers, or to read a page a user asked about. Each has its own robots.txt name.
- Generative engine optimization (GEO)Generative engine optimization (GEO) is the practice of making your content more likely to be found, used and cited in answers from AI systems like ChatGPT, Perplexity, Claude, Gemini and Google's AI Overviews.
- Structured data (JSON-LD)Structured data is machine-readable information about a page, usually written as JSON-LD using the schema.org vocabulary. It helps search engines understand what a page is about and can make it eligible for rich results.