GlossarySEO
What is the meta robots tag?
Also called: robots meta tag, meta name robots, index follow, noindex nofollow
Definition
The meta robots tag is an HTML tag in a page's head, like <meta name="robots" content="noindex">, that tells search engines whether to index the page, follow its links and how to show snippets.
Meta robots tag, explained
It lives in the <head> and applies to that page only. The name can be robots for all crawlers or a specific one, like googlebot. The content holds one or more comma-separated rules. Without the tag, the default is index, follow, so there's no need to add that explicitly.
Google's documentation lists the rules it supports: noindex (don't show the page in results), nofollow (don't follow links on the page), none (same as noindex, nofollow), nosnippet, max-snippet:[number], max-image-preview:[setting], max-video-preview:[number], notranslate, noimageindex, indexifembedded and unavailable_after:[date]. When rules conflict, the more restrictive one applies. The same rules can be sent as an X-Robots-Tag HTTP header, which also works for PDFs and images.
Two things break it. If robots.txt blocks the page, crawlers never see the tag. And if the tag is added or changed by JavaScript, Google's JavaScript SEO guide warns that when Google sees noindex it may skip rendering, so using script to remove a noindex may not work. Put the final value in the server-rendered HTML.
Most pages need no tag at all. Use it for thin, private-ish or duplicate pages you want out of results, and for snippet controls when you have a specific reason. After any change to shared layouts, fetch a few live URLs and read the tag in the raw HTML, not the browser's inspector, which shows the page after scripts have run.
Why it matters for founders
One stray noindex in a shared layout can remove your whole site from Google. It's the first thing to check when pages vanish after a deploy.
Example
Your staging environment sets <meta name="robots" content="noindex"> from an environment variable. After launch you check the production HTML and confirm the tag is gone from every template.
Common mistakes
- A layout-level
noindexleft over from staging. - Blocking the page in robots.txt so the tag is never read.
- Toggling
noindexwith client-side JavaScript. - Adding
index, followeverywhere and thinking it boosts rankings.
Sources
- Google Search Central: Robots meta tag, data-nosnippet and X-Robots-Tag
- Google Search Central: Understand JavaScript SEO basics
Checked
Related terms
- NoindexNoindex is a rule, set in a robots meta tag or an X-Robots-Tag HTTP header, that tells search engines not to show a page in their results even though they can crawl it.
- X-Robots-TagX-Robots-Tag is an HTTP response header that carries the same rules as the meta robots tag, like
noindexornosnippet. It works on any file type, including PDFs, images and other non-HTML files. - nosnippet
nosnippetis a robots rule that stops Google showing a text snippet or video preview for a page. Google says it also keeps the page's content from being used in AI Overviews and AI Mode. - robots.txtrobots.txt is a plain-text file at the root of a domain that tells crawlers which URLs they may and may not fetch. It controls crawling, not indexing, and well-behaved bots follow it voluntarily.