What is noindex?

Noindex, explained

There are two ways to set it. In the page's HTML head: <meta name="robots" content="noindex">. Or as an HTTP response header: X-Robots-Tag: noindex, which also works for files like PDFs. When Googlebot next crawls the page and sees the rule, it drops the page from Google Search, even if other sites link to it.

The catch trips up a lot of people. Google has to crawl the page to see the noindex. Its documentation is direct: for noindex to work, the page must not be blocked by robots.txt. If you disallow a URL in robots.txt and also add noindex, the crawler never reads the noindex, and the URL can still show up in results with no description. robots.txt controls crawling. Noindex controls indexing. They are not interchangeable.

Noindex is also slow to take effect. Nothing happens until Google recrawls the page, which can take days or weeks for pages it visits rarely. You can request a recrawl with the URL Inspection tool in Search Console to speed up important ones.

Used well, noindex is a quality tool. Thin tag pages, internal search results, near-duplicate variants, thank-you pages and staging copies usually don't belong in the index. Taking them out lets the pages that matter carry your site's reputation. Search Console lists these pages as "Excluded by 'noindex' tag", which is fine when it's on purpose.

A noindexed page can still pass link signals through its links if you leave it followable (noindex, follow is the usual combination). On this site, pages that don't meet a quality bar are noindexed but still followed, so they don't break internal linking.

Why it matters for founders

Founders often ship staging sites, duplicate landing pages or programmatic pages that are too thin. Noindex keeps those out of Google without deleting them. The opposite mistake is worse: a leftover noindex on a production page silently removes it from search.

Example

You generate a page for every integration you support, but a few have only a sentence of content. You add noindex to those until they have real detail, and leave them out of your sitemap.

Common mistakes

• Blocking a page in robots.txt and expecting its noindex to work. Google can't read a rule on a page it can't crawl.
• Shipping a staging noindex to production. Check the live HTML after every launch.
• Listing noindexed URLs in your XML sitemap, which sends Google mixed signals.
• Noindexing pages you want to rank because they "look thin" instead of improving them.