robots.txt, llms.txt and AI crawlers: a practical setup

Every robots.txt decision about AI is really three decisions: do you want your content used to train models, do you want to be cited in AI search, and do you want assistants to fetch your pages when a user asks?

Those are different crawlers with different names, and blocking the wrong one is an easy mistake. Here's a practical setup, based on each vendor's own documentation. You can check your current file with our free AI crawler checker.

Short answer: Sort AI crawlers into three groups: training crawlers (GPTBot, ClaudeBot, Google-Extended, Applebot-Extended, CCBot), search crawlers (OAI-SearchBot, Claude-SearchBot, PerplexityBot) and user-triggered fetchers (ChatGPT-User, Claude-User, Perplexity-User). Allow search crawlers if you want AI citations. Decide on training separately. Add llms.txt if it takes five minutes; it's optional.

First: what robots.txt can and can't do

Google's robots.txt introduction says it "tells search engine crawlers which URLs the crawler can access on your site," mainly "to avoid overloading your site with requests." Then the warning everyone forgets: "it is not a mechanism for keeping a web page out of Google." A blocked URL can still be indexed from links elsewhere, just without a description. To keep a page out of search, use noindex.

A few technical details from Google's robots.txt spec that matter in practice:

• Crawlers use "the group with the most specific user agent that matches." A User-agent: GPTBot group overrides your User-agent: * rules for GPTBot.
• Paths are case-sensitive.
• Google caches robots.txt "for up to 24 hours." Other vendors are similar: OpenAI says it can take "~24 hours" for its systems to adjust after you change robots.txt.
• If robots.txt returns a 5xx error, Google pauses crawling. A 4xx (other than 429) is treated as if no robots.txt exists. So a broken robots.txt can do more damage than a strict one.

The three kinds of AI crawler

Here's how the major vendors describe their own bots. We only include what each company documents.

Training crawlers

These collect content that may be used to train models.

AI search crawlers

These index your pages so an AI product can cite and link to them.

User-triggered fetchers

These visit a page because a person asked an assistant to.

The main takeaway: training and search are separate switches at OpenAI, Anthropic and Perplexity. You can say no to training and yes to being cited.

Google is different. Google Search, including AI Overviews and AI Mode, uses Googlebot. There's no separate "AI search" crawler to allow. AI features are controlled by snippet rules and the Search generative AI setting in Search Console, which we cover in AI Overviews and AI Mode for founders.

Three setups, depending on what you want

Setup A: allow everything (most startups)

If you want the most visibility and don't mind training use, you may not need AI-specific rules at all:

User-agent: *
Disallow: /admin/
Disallow: /api/

Sitemap: https://example.com/sitemap.xml

For a new product, this is usually right. Being known by models and cited by AI search is visibility you'd otherwise have to earn slowly.

Setup B: be cited, don't be trained on

If you want AI search citations but not training use:

User-agent: GPTBot
Disallow: /

User-agent: ClaudeBot
Disallow: /

User-agent: Google-Extended
Disallow: /

User-agent: Applebot-Extended
Disallow: /

User-agent: CCBot
Disallow: /

User-agent: *
Disallow: /admin/
Disallow: /api/

Sitemap: https://example.com/sitemap.xml

OAI-SearchBot, Claude-SearchBot and PerplexityBot fall through to User-agent: * and are allowed. Googlebot is unaffected.

One caveat on Google-Extended: Google's docs say it covers grounding for Gemini, "providing content from the Google Search index to the model at prompt time," as well as training. That grounding applies in Gemini Apps and in Grounding with Google Search on Vertex AI. Blocking it doesn't touch Google Search or AI Overviews, but it can keep your content out of those grounded answers. If Gemini app citations matter to you, leave it allowed.

Setup C: keep AI out of one section

Maybe you're fine with AI everywhere except your paid docs or a members area:

User-agent: GPTBot
User-agent: ClaudeBot
User-agent: Google-Extended
Disallow: /members/

User-agent: *
Disallow: /admin/

Remember robots.txt is a request, not a lock. Anything truly private belongs behind a login.

Our robots.txt generator builds any of these with the AI crawlers as checkboxes, so you don't have to remember the tokens.

The mistakes we see most

Blocking the search crawler while trying to block training. A "block all AI bots" list copied from a blog post can include OAI-SearchBot or PerplexityBot. The site then never shows up in ChatGPT or Perplexity search, and nobody knows why.

A CDN or firewall doing it for you. Some CDNs and hosting providers offer one-click AI bot blocking. It may block at the network level, before robots.txt is ever read. If AI crawlers can't reach you, check your CDN dashboard, not just the file.

A robots.txt that errors. As above, a 5xx on robots.txt can stop Googlebot crawling your site. Check that /robots.txt returns a 200.

Using robots.txt to hide pages. It stops crawling, not indexing. Use noindex.

Our AI crawler checker reads your live robots.txt and tells you, crawler by crawler, what's allowed.

Where llms.txt fits

llms.txt is a different kind of file. It grants or blocks nothing. It's a markdown summary at /llms.txt that points AI agents to your most useful pages.

Google's AI optimization guide is clear that Google Search doesn't use it, and that having one "will neither harm nor help" your visibility in Google. It can still help agents and developer tools that look for it, especially for documentation.

Our view: add one if it takes five minutes, generate it from the same source as your sitemap so it never goes stale, and don't pay anyone to "optimize" it. The llms.txt generator does it in a minute. The full explanation is in what is llms.txt, and do you need one?

A 15-minute setup checklist

1. Decide your policy: all in (A), cited but not trained (B), or partial (C).
2. Write robots.txt with the robots.txt generator.
3. Deploy, then confirm /robots.txt returns a 200 and plain text.
4. Check your CDN or host for AI bot blocking that contradicts your file.
5. Run the AI crawler checker to confirm each crawler's status.
6. Optionally add /llms.txt.
7. Wait about a day for vendors to pick up changes before testing.

If you want to see your crawler setup through a glossary lens, we have short entries for AI crawlers, Google-Extended, robots.txt and llms.txt.

Once crawlers can reach you, the next question is whether AI products actually cite you. That's what how to get cited by ChatGPT and Perplexity is about. And if you're launching, a free launch on LaunchRanked gives your product a permanent, crawlable page and a followed link after review.