robots.txt tester

How the decision is made

robots.txt is a list of groups. Each group starts with one or more User-agent lines, followed by Allow and Disallow rules. The tester follows RFC 9309, the standard Google and Bing implement:

1. Pick the group. The crawler looks for groups naming its product token (Googlebot, GPTBot…), case insensitive. If several groups name it, their rules are merged. If none do, it uses the User-agent: * group. If there’s no * group either, everything is allowed.
2. Find the longest match. Among that group’s rules, the one whose path matches the most characters of the URL wins. * matches any run of characters and a trailing $ anchors the end.
3. Break ties with Allow. If an Allow and a Disallow match with the same length, Allow wins.

The result shows the verdict, the group that applied and the exact line that decided, highlighted in the file. You can paste a full user-agent string: like Google, we match on its product token.

Things that trip people up

• Disallow: /blog also blocks /blog-post and /blogroll. Use /blog/ to block the folder only.
• Paths are case sensitive: /Admin and /admin are different rules.
• Query strings count. Disallow: /*? blocks every URL with a query string, including tracking parameters.
• An empty Disallow: allows everything; Disallow: / blocks everything.
• Google reads only the first 500 KiB of the file.

Test the URLs that matter before you ship a new robots.txt: your homepage, a product page, a blog post, your sitemap, and any CSS or JavaScript files pages need to render. Blocking resources Google needs to render a page can hurt how it understands the page.

To see every AI crawler’s status in one go, use the AI crawler checker. To write a new file, start from the robots.txt generator.