GlossaryAI search
What are AI crawlers?
Also called: AI bots, GPTBot, OAI-SearchBot, ClaudeBot, PerplexityBot, LLM crawlers
Definition
AI crawlers are bots run by AI companies that fetch web pages, either to collect training data, to build an index for AI search answers, or to read a page a user asked about. Each has its own robots.txt name.
AI crawlers, explained
The key thing to know is that AI companies now run separate bots for separate jobs, and you can treat them differently in robots.txt.
OpenAI: GPTBot crawls content that may be used to train its foundation models. OAI-SearchBot surfaces websites in ChatGPT's search features. ChatGPT-User fetches pages when a user's request needs them; OpenAI says robots.txt rules may not apply to it because a user initiated the action. OpenAI notes robots.txt changes can take about 24 hours to take effect.
Anthropic: ClaudeBot collects content that may contribute to training. Claude-SearchBot indexes content to improve Claude's search results. Claude-User fetches pages when a Claude user asks a question that needs them. Anthropic says its bots respect robots.txt and supports the non-standard Crawl-delay rule.
Perplexity: PerplexityBot surfaces and links websites in Perplexity's search results and, Perplexity says, isn't used to crawl content for AI foundation models. Perplexity-User fetches pages for user questions and, per Perplexity, generally ignores robots.txt because a user requested the fetch.
Google is different: it doesn't have a separate AI crawler. Googlebot crawls for Search, including AI Overviews and AI Mode. The Google-Extended token only controls whether that content is used for Gemini training and grounding.
The common setup for a startup that wants visibility: allow the search bots (OAI-SearchBot, Claude-SearchBot, PerplexityBot) and user fetchers, and decide separately whether you're comfortable with training bots. Blocking a training bot doesn't remove you from that company's search answers. Blocking a search bot does.
Why it matters for founders
Some CDNs, hosts and site builders now block AI bots with a single toggle, and some do it by default on new sites. If you want to be cited in ChatGPT or Perplexity, check that their search crawlers aren't blocked, because it's easy to be invisible without knowing.
Example
A site's robots.txt has User-agent: * / Disallow: plus a block for GPTBot. It stays out of OpenAI's training data but remains eligible for ChatGPT search, because OAI-SearchBot is still allowed.
Common mistakes
- Blocking every AI user agent and expecting to be cited in AI answers.
- Assuming blocking
GPTBotalso removes you from ChatGPT search. It doesn't. - Assuming blocking a bot in robots.txt stops user-triggered fetchers.
- Never checking what your CDN's bot settings add on top of robots.txt.
Sources
- OpenAI: Overview of OpenAI crawlers
- Anthropic: Does Anthropic crawl data from the web?
- Perplexity: Perplexity crawlers
Checked
Related terms
- robots.txtrobots.txt is a plain-text file at the root of a domain that tells crawlers which URLs they may and may not fetch. It controls crawling, not indexing, and well-behaved bots follow it voluntarily.
- Google-ExtendedGoogle-Extended is a robots.txt token that controls whether content Google crawls from your site may be used to train Gemini models and for grounding. It doesn't affect Google Search, including AI Overviews.
- LLM citationAn LLM citation is a link or named source that an AI assistant shows alongside its answer to say where information came from. Being cited sends referral traffic and signals that your page was trusted.
- llms.txtllms.txt is a proposed markdown file at a site's root, /llms.txt, that summarizes the site and links to its most useful pages for AI agents. It's voluntary, grants no access, and Google Search ignores it.