What is a web crawler?

Web crawler, explained

Crawling is the first step of search. Google describes it as automated programs, called crawlers, downloading text, images and videos from pages they find on the internet. Crawlers start from URLs they already know, like sitemaps and pages indexed before, fetch them, extract the links, and add new URLs to a queue. Google calls this URL discovery.

Well-behaved crawlers identify themselves with a user-agent string, check robots.txt before fetching, and pace their requests so they don't overload your server. Google's crawlers slow down when a site returns errors or responds slowly. Googlebot and Bingbot are the best known, but many others visit: SEO tools like Ahrefs and Semrush, social media link previewers, uptime monitors, and a growing set of AI crawlers that collect training data or fetch pages to answer questions.

Not every visitor claiming to be a crawler is one. User-agent strings are easy to fake, so Google and Bing publish ways to verify their bots, usually a reverse DNS lookup on the requesting IP address.

For a new site, the crawler's view is what matters. If important content only appears after a click, if links are JavaScript handlers instead of <a href> elements, or if a page is buried five clicks deep, crawlers may never find or understand it.

Why it matters for founders

Every page that ranks, and every page an AI assistant cites, was fetched by a crawler first. Making pages easy to discover and fetch is the base layer of SEO.

Example

You launch a new comparison page and link it from your homepage. Googlebot recrawls the homepage, finds the new link, fetches the page, and it appears in Search Console's crawl data a few days later.

Common mistakes

• Blocking crawlers in robots.txt without knowing which bot does what.
• Trusting user-agent strings without verifying the IP.
• Links that only work with JavaScript click handlers.
• Assuming a published page has been crawled.