AI crawler checker

What this checks

We fetch https://yourdomain.com/robots.txt the way a crawler does: following up to five redirects and reading the first 500 KB, which is where Google stops. We split the file into user-agent groups, work out which group each bot follows, and apply that group’s rules to the path you entered. Enter a bare domain and we test the homepage.

How robots.txt decides

• A bot uses the group that names it. Only when no group names it does it fall back to User-agent: *. Groups don’t stack: add a User-agent: GPTBot group with one line in it and GPTBot stops reading every rule under *.
• The longest matching rule wins. Allow: /blog/ beats Disallow: / for /blog/post. On a tie, Allow wins.
• Status codes matter. A 404 means no restrictions. A 5xx error means crawlers such as Googlebot treat the whole site as blocked until the file comes back.

Training, search and user-triggered bots

AI companies now run separate agents for separate jobs. Training crawlers (GPTBot, ClaudeBot, CCBot, Meta-ExternalAgent) collect pages that may end up in model training data. Search crawlers (OAI-SearchBot, Claude-SearchBot, PerplexityBot) build the index those assistants use to cite and link sources. User-triggered agents (ChatGPT-User, Claude-User, Perplexity-User) fetch a single page because someone asked about it.

That split makes a middle path possible: opt out of training and still get cited. If you want to show up in ChatGPT search or Perplexity answers, their search bots need access. OpenAI says pages that block OAI-SearchBot won’t be shown in ChatGPT search answers; blocking GPTBot alone doesn’t do that.

Google-Extended and Applebot-Extended are different. They aren’t crawlers. They’re tokens that tell Google and Apple whether content their main crawlers already fetched can be used for AI models. Blocking Google-Extended doesn’t touch Google Search or AI Overviews.

Common fixes

• Blocked by accident. A CMS setting, security plugin or CDN toggle added a blanket AI-bot block. Remove the groups for the search bots you want to keep.
• A named group that forgot your other rules. Repeat your Disallow lines inside it, since the bot no longer reads the * group.
• robots.txt served as an HTML page. Some hosts return a styled 404 page with status 200. Serve a real plain-text file, or a true 404.

Want a clean file? The robots.txt generator writes one with the presets above. Then check the rest of the page with the meta tag checker.