GlossaryAI search
How does Claude search the web?
Also called: Claude search, Claude-SearchBot, Anthropic web search
Definition
Claude web search is Anthropic's feature that lets Claude search the internet during a conversation and answer with up-to-date information and direct citations to the pages it used.
Claude web search, explained
Anthropic announced web search for Claude on March 20, 2025, and later made it available globally on all Claude plans. Its announcement says that when Claude incorporates information from the web, it provides direct citations so users can fact-check sources. Developers can also give Claude a web search tool through the API, so the same behavior shows up inside other products built on Claude.
Three Anthropic user agents matter, and Anthropic documents each. ClaudeBot collects web content that may contribute to training. Claude-SearchBot indexes content to improve the quality of Claude's search results. Claude-User fetches pages when a Claude user asks something that needs them. Anthropic says its bots respect robots.txt, and it supports the non-standard Crawl-delay directive.
That split gives you the same choice as with OpenAI: block ClaudeBot if you don't want to contribute to training, and keep Claude-SearchBot and Claude-User allowed if you want to be found and cited. Blocking all three keeps Claude from reading your pages when users ask about you.
Claude is used heavily for research and technical work, including by developers evaluating tools. For B2B and developer products especially, being citable in Claude matters. The same page qualities help: a direct answer, specific facts, versions and dates where relevant, and docs that load as readable HTML.
Referral traffic from Claude typically shows claude.ai as the referrer. Add it to your analytics segment for AI assistants, alongside ChatGPT, Perplexity, Gemini and Copilot, so the channel is visible as it grows.
Why it matters for founders
Developers and technical buyers use Claude for research. Allowing Claude-SearchBot while deciding separately about ClaudeBot keeps you citable without giving up your training choice.
Example
A developer asks Claude which open-source queue libraries support delayed jobs. Claude searches, cites the docs pages that state it plainly, and the developer clicks through to one of them.
Common mistakes
- Blocking every Anthropic user agent when you only meant to block training.
- Serving docs as client-rendered shells that show little text to crawlers.
- Forgetting
claude.aiin your AI referral segment.
Sources
- Claude blog: Claude can now search the web (March 2025)
- Anthropic: Does Anthropic crawl data from the web?
Checked
Related terms
- AI crawlersAI crawlers are bots run by AI companies that fetch web pages, either to collect training data, to build an index for AI search answers, or to read a page a user asked about. Each has its own robots.txt name.
- LLM citationAn LLM citation is a link or named source that an AI assistant shows alongside its answer to say where information came from. Being cited sends referral traffic and signals that your page was trusted.
- robots.txtrobots.txt is a plain-text file at the root of a domain that tells crawlers which URLs they may and may not fetch. It controls crawling, not indexing, and well-behaved bots follow it voluntarily.
- AI referral trafficAI referral traffic is the visits your site gets when people click links in AI assistants' answers, such as citations in ChatGPT, Perplexity, Claude, Gemini or Copilot.