What is crawl budget?

Crawl budget, explained

Google defines crawl budget as the combination of two things. The crawl capacity limit is how hard Googlebot can crawl without straining your server; it rises when your site responds quickly and falls when it slows down or returns errors. Crawl demand is how much Google wants to crawl, which depends on a site's size, update frequency, page quality and relevance compared with other sites.

Here's the part most founders need: Google says its crawl budget guide is for large sites, roughly 1 million or more unique pages that change weekly, or 10,000 or more pages that change daily. If your site has a few hundred pages, crawl budget is almost never why pages aren't indexed. Quality and discovery are.

It becomes relevant for smaller sites in one situation: when you create a lot of URLs quickly. Programmatic pages, faceted filters, calendar pages and URL parameters can multiply your URL count faster than Google's interest in your site grows. The Page indexing report then fills up with "Discovered – currently not indexed", which Google describes as a URL it found but rescheduled because crawling it was expected to overload the site.

The fixes are mostly about removing waste: keep infinite filter and parameter combinations out of crawl paths, return real 404s for dead pages, avoid long redirect chains, keep the sitemap limited to pages you want indexed, and make the server fast. Then raise demand the honest way, with pages people actually link to and search for.

Quality appears in Google's definition of crawl demand for a reason. A site that ships lots of similar pages can see Google slow down across the whole site, not only on the weak pages.

Why it matters for founders

If you're planning programmatic SEO, crawl budget is the constraint you'll hit after launch, not before. Publishing faster than Google wants to crawl leaves pages sitting in "Discovered" or "Crawled – currently not indexed" for weeks.

Example

An e-commerce site's color and size filters generate 200,000 parameter URLs from 2,000 products. Googlebot spends its visits on filter pages. Blocking the parameter paths and linking only to canonical product pages lets new products get crawled sooner.

Common mistakes

• Blaming crawl budget for indexing problems on a 200-page site.
• Letting faceted navigation or session IDs create endless crawlable URLs.
• Publishing thousands of pages at once on a young domain.
• Using noindex to save crawl budget. Google still has to crawl a page to see noindex.