GlossarySEO
What is robots.txt?
Also called: robots file, robots exclusion protocol
Definition
robots.txt is a plain-text file at the root of a domain that tells crawlers which URLs they may and may not fetch. It controls crawling, not indexing, and well-behaved bots follow it voluntarily.
robots.txt, explained
The file lives at exactly one place: https://yourdomain.com/robots.txt. It is a list of groups. Each group starts with one or more User-agent lines naming a crawler (or * for all), followed by Allow and Disallow rules for URL paths. You can also add a Sitemap: line pointing to your XML sitemap.
Google describes the purpose plainly: robots.txt tells crawlers which URLs they can access, mainly to avoid overloading your site with requests. It also warns that it is not a mechanism for keeping a page out of Google. A disallowed URL can still be indexed if other sites link to it, and then it shows up with no description. If you want a page out of search, use noindex and let it be crawled.
In 2026 the file does a second job: it's where you decide which AI crawlers can read your site. Each AI company publishes its own user-agent tokens. OpenAI uses GPTBot for training and OAI-SearchBot for ChatGPT search. Anthropic uses ClaudeBot for training and Claude-SearchBot for search. Perplexity uses PerplexityBot. Google uses the Google-Extended token to control Gemini training and grounding without affecting Google Search. You can block training and still allow search, or the reverse.
Two practical notes. First, rules match by path prefix, so Disallow: /app also blocks /apple. Add a trailing slash when you mean a folder. Second, blocking CSS and JavaScript can stop Google from rendering your pages properly, so leave assets crawlable.
Most small sites need very little here: allow everything, block admin and internal search paths, list the sitemap, and make a deliberate choice about AI bots.
Why it matters for founders
One wrong line can hide your whole site from Google or from ChatGPT search. Some site builders and CDNs now add AI-bot rules for you. Check what yours says before you assume AI assistants can see you.
Example
User-agent: GPTBot / Disallow: / blocks OpenAI's training crawler, while a separate User-agent: OAI-SearchBot / Allow: / group keeps your pages eligible for ChatGPT search results.
Common mistakes
- Using
Disallowto remove a page from Google. Use noindex instead, and leave the page crawlable. - Leaving
Disallow: /from a staging environment in production. - Blocking every AI bot by default and then wondering why AI assistants never cite you.
- Forgetting that paths are case-sensitive prefixes.
Sources
Checked
Related terms
- AI crawlersAI crawlers are bots run by AI companies that fetch web pages, either to collect training data, to build an index for AI search answers, or to read a page a user asked about. Each has its own robots.txt name.
- NoindexNoindex is a rule, set in a robots meta tag or an X-Robots-Tag HTTP header, that tells search engines not to show a page in their results even though they can crawl it.
- XML sitemapAn XML sitemap is a file that lists the URLs you want search engines to crawl and index, optionally with the date each one last changed. It helps discovery; it doesn't guarantee indexing.
- Google-ExtendedGoogle-Extended is a robots.txt token that controls whether content Google crawls from your site may be used to train Gemini models and for grounding. It doesn't affect Google Search, including AI Overviews.