GlossarySEO
What is Googlebot?
Also called: Googlebot Smartphone, Googlebot Desktop, Google crawler
Definition
Googlebot is Google's main web crawler for Search. It comes in a smartphone and a desktop version, and since mobile-first indexing most sites are crawled by Googlebot Smartphone.
Googlebot, explained
Googlebot fetches pages, renders them with an up-to-date version of Chromium, and hands the result to Google's indexing systems. Google's documentation says Googlebot Smartphone simulates a user on a mobile device and does most of the crawling, because Google primarily indexes the mobile version of content. It also says Googlebot shouldn't access a site more than once every few seconds on average, though short bursts happen.
There's a size limit worth knowing. Google's Googlebot page says it crawls the first 2MB of a supported file type and the first 64MB of a PDF, measured on uncompressed data, and that once the limit is reached it stops the fetch and only sends what it downloaded for indexing. CSS and JavaScript files are fetched separately with the same limits. Very large HTML pages, often bloated with inline data, can lose content at the bottom.
Anyone can send a request with a Googlebot user-agent string, so Google recommends verifying real Googlebot traffic with a reverse DNS lookup on the IP (the host should end in googlebot.com, google.com or googleusercontent.com, confirmed with a forward lookup) or by matching Google's published IP ranges.
Googlebot obeys robots.txt. Blocking it hides your pages from Google Search, Images, Video, News and Discover. Its rules are separate from Google-Extended, which only controls use of your content for Gemini.
Why it matters for founders
Googlebot's view of your page is the only one Google indexes. If it can't fetch, render or fit your page in its limits, what users see doesn't matter for Search.
Example
Your log file shows heavy traffic from "Googlebot" hitting login URLs. A reverse DNS lookup shows the IPs belong to a hosting company, not Google, so you rate-limit them without touching real Googlebot.
Common mistakes
- Blocking Googlebot from CSS or JavaScript it needs to render the page.
- Trusting the user-agent string without IP verification.
- Shipping huge HTML documents with inline JSON that pushes content past the fetch limit.
- Testing only the desktop version when Googlebot Smartphone does the crawling.
Sources
- Google Search Central: Googlebot
- Google Search Central: Verifying Googlebot and other Google crawlers
Checked
Related terms
- Web crawlerA web crawler is a program that fetches web pages automatically, follows their links to find more pages, and passes what it finds to a search engine or AI system for indexing or analysis.
- Mobile-first indexingMobile-first indexing means Google uses the mobile version of a page, crawled with its smartphone crawler, for indexing and ranking. If content is missing on mobile, Google doesn't see it.
- RenderingRendering is when a search engine runs a page's JavaScript and CSS in a headless browser to see the final content, as a user would. Google renders pages before indexing what it finds.
- robots.txtrobots.txt is a plain-text file at the root of a domain that tells crawlers which URLs they may and may not fetch. It controls crawling, not indexing, and well-behaved bots follow it voluntarily.