robots.txt generator

Pick a preset or choose bot by bot. Add the paths you want kept out and your sitemap. Copy or download the file.

robots.txt

# Generated with https://launchranked.com/tools/robots-txt-generator

User-agent: *
Allow: /

Save it as robots.txt at your site root. Then check it with the AI crawler checker.

What the presets do

Allow all. One User-agent: * group. Every crawler can read everything except the paths you add.
Block AI training, allow AI search. Adds one group that blocks the training crawlers and tokens (GPTBot, ClaudeBot, Google-Extended, Applebot-Extended, CCBot, Bytespider, Meta-ExternalAgent). AI search bots (OAI-SearchBot, Claude-SearchBot, PerplexityBot) and user-triggered agents fall back to your general rules, so assistants can still find, cite and link your pages.
Block all AI bots. Also blocks the AI search and user-triggered agents. The trade-off: those assistants can’t read fresh copies of your pages, so they’re less likely to cite you. Google and Bing search stay open.

Two entries need a note. Google-Extended and Applebot-Extended aren’t crawlers: they tell Google and Apple whether pages their regular crawlers already fetch may be used for AI models. Blocking them leaves Google Search, AI Overviews and Apple’s search features alone. And user-triggered agents fetch a page because a person asked; some operators say those fetches may not follow robots.txt, so blocking them is a request, not a guarantee.

How the file is structured

A crawler reads only the group that names it, or the * group if none does. That’s why the blocked bots get their own group with Disallow: /, and everyone else shares the * group with your custom paths. A typical result:

User-agent: *
Disallow: /admin/

User-agent: GPTBot
User-agent: ClaudeBot
User-agent: Google-Extended
Disallow: /

Sitemap: https://example.com/sitemap.xml

Several User-agent lines in a row form one group, which is valid under RFC 9309, the robots.txt standard.

Mistakes to avoid

Using robots.txt to hide a page from Google. A disallowed URL can still be indexed from links, without its content. To keep a page out of results, allow crawling and add noindex.
Blocking CSS and JavaScript. Google needs them to render the page. Don’t disallow asset folders.
Disallow: / under *. That blocks every crawler, Google included. It’s the classic staging setting that ships to production.
A huge file. Google reads only the first 500 KiB. Use wildcards instead of listing thousands of URLs.

After you upload the file, confirm each bot’s access with the AI crawler checker.

Questions

Which bots count as AI training bots?

In this generator: GPTBot, ClaudeBot, Google-Extended, Applebot-Extended, CCBot, Bytespider, Meta-ExternalAgent. Google-Extended and Applebot-Extended aren't crawlers; they're tokens that control whether content Google and Apple already crawl is used for their AI models.

Will blocking AI bots hurt my Google rankings?

Not if you block the AI-specific tokens only. Googlebot is never blocked by these presets, and Google says Google-Extended doesn't affect Search. Blocking AI search bots does affect whether ChatGPT search, Claude or Perplexity can cite you.

Does robots.txt stop AI companies using my content?

It asks crawlers not to fetch pages from now on, and the major AI companies document that their crawlers honour it. It doesn't remove content already collected, and it can't stop bots that ignore it.

Where does robots.txt go?

At the root of each host: https://example.com/robots.txt. Subdomains need their own file. In Next.js, add app/robots.ts or a static public/robots.txt. In WordPress, most SEO plugins let you edit it.

Autopilot · early access

Want this fixed automatically?

Autopilot checks your site, publishes researched articles at a safe pace, and tracks whether ChatGPT, Perplexity, Gemini and Claude cite you. $49/mo.

See Autopilot Launch your product free