GlossaryAI search
What is Common Crawl?
Also called: CCBot, Common Crawl dataset
Definition
Common Crawl is a non-profit that crawls the web with its bot, CCBot, and publishes the results as a free, open repository of web data that researchers and AI companies widely use.
Common Crawl, explained
Common Crawl describes itself as maintaining a free, open repository of web crawl data that can be used by anyone. It's a 501(c)(3) non-profit founded in 2007, and its site cites over 300 billion pages spanning 15 years. The data is released in regular crawls that anyone can download.
It matters for AI because many language models were trained partly on it. The GPT-3 paper, for example, describes filtering Common Crawl as one of its training data sources. Many other published models and datasets build on it too. So Common Crawl is one path by which your pages can end up in model training, separate from any AI company's own crawler.
The crawler is CCBot. Common Crawl's documentation shows the opt-out: User-agent: CCBot with Disallow: / in robots.txt. It also warns that other crawlers sometimes falsely identify themselves as CCBot, and says CCBot now runs on dedicated IP ranges with reverse DNS so you can verify real requests.
Blocking it is a training decision, not a search decision. It doesn't power ChatGPT search, Perplexity or Google's AI features. It only affects future crawls, not data already collected. If you want to stay out of training datasets broadly, CCBot belongs on the same list as GPTBot, ClaudeBot and Google-Extended. If you want future models to know your product exists, you may prefer to allow it.
There's no right answer for everyone. Publishers with valuable content often block it; early-stage startups that want to be known by future models often don't. Make the choice deliberately rather than inheriting it from a template.
Why it matters for founders
Common Crawl is a quiet route into AI training data that many founders don't know about. Deciding on CCBot is part of the same training-vs-visibility choice you make for AI companies' own bots.
Example
A startup allows AI search bots, blocks GPTBot and ClaudeBot, and forgets CCBot. Its pages keep flowing into an open dataset used for training. It adds one robots.txt group to make its policy consistent.
Common mistakes
- Blocking named AI training bots but not
CCBot, then assuming you're out of training data. - Thinking a block removes data already collected in past crawls.
- Trusting every request that claims to be CCBot without checking.
Sources
- Common Crawl: Overview
- Common Crawl: CCBot
- Brown et al., Language Models are Few-Shot Learners (arXiv, 2020)
Checked
Related terms
- AI crawlersAI crawlers are bots run by AI companies that fetch web pages, either to collect training data, to build an index for AI search answers, or to read a page a user asked about. Each has its own robots.txt name.
- robots.txtrobots.txt is a plain-text file at the root of a domain that tells crawlers which URLs they may and may not fetch. It controls crawling, not indexing, and well-behaved bots follow it voluntarily.
- Knowledge cutoffA knowledge cutoff is the point in time after which a language model has no training data. Anything that happened or launched later is unknown to the model unless it's retrieved at answer time.
- Google-ExtendedGoogle-Extended is a robots.txt token that controls whether content Google crawls from your site may be used to train Gemini models and for grounding. It doesn't affect Google Search, including AI Overviews.