GlossarySEO
What is the X-Robots-Tag header?
Also called: X-Robots-Tag header, robots HTTP header
Definition
X-Robots-Tag is an HTTP response header that carries the same rules as the meta robots tag, like noindex or nosnippet. It works on any file type, including PDFs, images and other non-HTML files.
X-Robots-Tag, explained
The meta robots tag only works in HTML, because it lives in the page head. The X-Robots-Tag header is set by your server or CDN and travels with any response. Google's documentation shows the syntax: X-Robots-Tag: noindex, several rules separated by commas, and optional user-agent targeting such as X-Robots-Tag: googlebot: noindex. Google supports the same rules in both places.
Where it's the right tool: keeping PDFs, internal documents or downloadable files out of search; noindexing images you don't want in Google Images; applying noindex to a whole path or file type with one server rule instead of editing templates; and noindexing API responses or feeds that were getting indexed. It's also handy on platforms where you can set headers but can't edit the HTML.
The same caveat as meta robots applies. Crawlers must be allowed to fetch the URL to see the header, so don't also block it in robots.txt. Header rules are also easy to forget because nobody sees them in the page source. A broad rule at the CDN can silently noindex a whole section, and the page will look perfectly normal in a browser.
When auditing, check headers as well as HTML. Many "why did my page disappear" cases come from a header rule added for a different path.
Why it matters for founders
If you host PDFs, pitch decks or docs on your domain, this is how you keep them out of results. It's also a place where accidental noindex rules hide.
Example
Your investor update PDFs under /files/ keep appearing in Google. You add a CDN rule that sends X-Robots-Tag: noindex for that path, and they drop out after the next crawl.
Common mistakes
- Blocking the files in robots.txt so the header is never seen.
- A broad header rule that noindexes HTML pages too.
- Forgetting headers when auditing why a page isn't indexed.
- Relying on noindex to protect private documents instead of authentication.
Sources
Checked
Related terms
- Meta robots tagThe meta robots tag is an HTML tag in a page's head, like
<meta name="robots" content="noindex">, that tells search engines whether to index the page, follow its links and how to show snippets. - NoindexNoindex is a rule, set in a robots meta tag or an X-Robots-Tag HTTP header, that tells search engines not to show a page in their results even though they can crawl it.
- robots.txtrobots.txt is a plain-text file at the root of a domain that tells crawlers which URLs they may and may not fetch. It controls crawling, not indexing, and well-behaved bots follow it voluntarily.
- IndexingIndexing is when a search engine analyzes a crawled page and stores it in its index, the database it serves results from. A page that isn't indexed can't appear in search results.