GlossarySEO
What does "Indexed, though blocked by robots.txt" mean?
Also called: indexed though blocked by robots.txt, blocked by robots.txt but indexed
Definition
This Search Console status means Google indexed a URL even though robots.txt blocks crawling it. Google can't read the page, so it indexes the bare URL from links pointing to it.
Indexed, though blocked by robots.txt, explained
Google's Page indexing report explains that the page was indexed despite being blocked by your robots.txt file: Google won't request and crawl the page, but it can still index it. It learns about the URL from links, internal or external, and may show it in results with no description or a note that no information is available.
The root cause is a common misunderstanding. robots.txt controls crawling, not indexing. Blocking a URL stops Google from reading it, which also stops Google from seeing a noindex tag on it. So the one tool that would remove the page from results can't work.
The fix depends on what you want. If the page should not appear in Google: remove the robots.txt block, add noindex (meta tag or X-Robots-Tag header), let Google recrawl and drop it, and only then consider blocking again, though usually you won't need to. If the page should appear, just remove the block. For private pages, use authentication; neither robots.txt nor noindex protects content.
Typical offenders are internal search results, filtered listing URLs, admin paths with public links pointing at them, and old staging paths. Also check for broad rules: Disallow: /app also blocks /apple-integration, because rules match by prefix. A handful of URLs in this status is common and often harmless; hundreds usually point to one rule that's doing more than intended.
Why it matters for founders
Blocked URLs showing in Google with no description look broken to searchers. Fixing it means using the right tool for the job: noindex for indexing, robots.txt for crawling.
Example
Your /search?q= pages are disallowed in robots.txt, but spam sites link to them and they appear in Google. You lift the block, add noindex to search result pages, and they drop out over the next few weeks.
Common mistakes
- Adding
noindexwhile keeping the robots.txt block. - Using robots.txt to hide sensitive pages.
- Broad prefix rules that block more than you meant.
- Re-blocking before Google has recrawled and seen the noindex.
Sources
Checked
Related terms
- robots.txtrobots.txt is a plain-text file at the root of a domain that tells crawlers which URLs they may and may not fetch. It controls crawling, not indexing, and well-behaved bots follow it voluntarily.
- NoindexNoindex is a rule, set in a robots meta tag or an X-Robots-Tag HTTP header, that tells search engines not to show a page in their results even though they can crawl it.
- Meta robots tagThe meta robots tag is an HTML tag in a page's head, like
<meta name="robots" content="noindex">, that tells search engines whether to index the page, follow its links and how to show snippets. - Page indexing reportThe Page indexing report in Search Console shows how many of your known URLs are indexed and groups the rest by the reason they aren't, like noindex, redirect, duplicate, 404 or crawled but not indexed.