CCBot: Should You Allow or Block Common Crawl?

Most guides tell you to block CCBot. Almost none tell you what blocking it actually does, or what it does not do.

CCBot is Common Crawl's web crawler. It collects pages for a free, open dataset that most large language models train on, so blocking CCBot in robots.txt opts your content out of that pipeline.

Here is the part most guides skip: CCBot does not fetch live answers, so blocking it does nothing to your ChatGPT or Perplexity visibility. It is purely a training and bandwidth decision.

What is CCBot?

CCBot is the crawler run by Common Crawl, a nonprofit that has archived the public web since 2008. It fetches around 2 billion pages a month into a public dataset now larger than 10 petabytes, which anyone can download for free. Per Common Crawl's own docs, CCBot identifies itself as CCBot/2.0 and checks robots.txt before fetching a page. It is a batch archiver, not a live search bot.

That scale is the reason CCBot matters at all.

Common Crawl has been running for more than 15 years, and its archive has been cited in over 12,000 research papers, according to Common Crawl. When you publish a page, CCBot is often the first crawler that pulls it into a dataset an AI lab will later train on.

But it works on a monthly cycle, not in real time.

CCBot crawls in batches and publishes a new archive roughly once a month. Nothing about that process answers a live user question. It builds the raw material that model training runs on, which is a completely different job from the search crawlers that go fetch a page while a user sits waiting on an answer.

One detail changes how you should think about a block. Most AI labs never touch CCBot directly.

The Mozilla research found that most builders do not crawl or filter Common Crawl themselves, and instead train on filtered versions someone else published. So your page can reach a model through several hands you never see. Blocking CCBot cuts that supply at the source, but only for crawls that happen after you add the rule.

CCBot feeds training, not live answers. Blocking it is a data choice, not a visibility choice.The one line to remember

CCBot vs GPTBot: do you need to block both?

They differ in one way that matters: GPTBot feeds one company, OpenAI, while CCBot feeds an open dataset that hundreds of models and researchers draw from. Blocking GPTBot opts you out of OpenAI training only. Blocking CCBot opts you out of the shared corpus behind GPT, Llama, Mistral, and many other models at once. You control each with its own robots.txt token.

So do you need to block both? Only if your goal is a full training opt-out.

Blocking GPTBot and Google-Extended covers the two biggest vendor-specific trainers. Adding CCBot covers the open corpus that everyone else, including smaller labs and researchers, builds on. Our GPTBot allow-or-block guide and Google-Extended guide cover those two tokens in detail.

Does blocking CCBot remove you from ChatGPT and other AI models?

No. Blocking CCBot stops future crawls, but it does not delete pages Common Crawl already archived, and it cannot pull your content out of models that were already trained on older snapshots. A robots.txt block is forward-looking. Common Crawl runs a separate opt-out registry for existing archives, and past model weights cannot be un-trained at all.

This trips people up constantly.

They block CCBot in 2026 and assume they have scrubbed themselves from ChatGPT. They have not. A model trained on a 2023 crawl already learned whatever it learned, and no robots.txt edit reaches back into those weights.

The scale of that back catalog is easy to underestimate.

A 2024 Mozilla Foundation study found that at least 64% of 47 large language models published between 2019 and October 2023 were trained on a filtered version of Common Crawl, and that over 80% of GPT-3's tokens came from it. If your site was live in those years, it is likely already baked into models that shipped before you touched your robots.txt.

Does blocking CCBot hurt your AI search visibility?

No, and this is the key point. CCBot is not the bot that cites you in AI answers. Live citations come from retrieval crawlers like OAI-SearchBot and the search fetches other assistants run, which visit you the moment a user asks. CCBot only feeds training, so blocking it leaves your ChatGPT and Perplexity search visibility fully intact.

That makes CCBot the mirror image of a search crawler.

Blocking a search bot like OAI-SearchBot drops you out of live AI answers, which is a direct hit to visibility. Blocking CCBot does no such thing. Your pages can still be crawled, retrieved, and cited in real time exactly as before.

There is one subtler trade-off, though, and it is worth naming.

Being in the training corpus is part of how a model knows your brand exists at all. That baseline familiarity, the stuff a model can say about you with no live search, comes from training data like Common Crawl.

Block CCBot for years and you slowly reduce how much new material about you reaches future models. That is a real but slow cost, separate from live citations, and our guide on where ChatGPT gets its information unpacks the training-versus-retrieval split.

Should you allow or block CCBot?

Allow it unless you have a specific reason not to. For most sites, being in the Common Crawl corpus is how models learn your brand and content exist in the first place, which helps you get recognized later. Block CCBot only if you hold a firm training-opt-out stance, or the monthly crawl is straining your server. There is no AI-search visibility cost either way.

The decision comes down to one question: do you want to be in AI training data?

If you sell content, run a paywall, or object to your work training models you have no stake in, blocking CCBot is a reasonable stand. Publishers are increasingly taking it. A 2026 analysis of robots.txt files across Cloudflare's network found CCBot sitting at roughly a 4-to-1 block-to-allow ratio, with 604 of 4,223 sampled domains disallowing it outright.

But notice what those same publishers do next.

The same 2026 data shows training-bot traffic falling from 47% to 40% of AI crawling between June and August while search-bot traffic climbed from 11% to 16%. Sites are blocking training crawlers like CCBot and deliberately keeping search crawlers open, because they have worked out that the two do completely different jobs.

There is also a plain operational reason some sites block it. A full batch crawl of a large site can land as a burst of requests.

If CCBot fetching thousands of pages in a short window strains your server or inflates your bandwidth bill, that alone can justify a block, with no training argument at all. Just verify the traffic is really CCBot first, since spoofed crawlers create the same load while pretending to be Common Crawl.

How do you block CCBot and verify it is really CCBot?

Add a two-line rule to robots.txt: a User-agent line for CCBot, then Disallow slash. Common Crawl documents this exact block and rechecks your file on its next pass. To confirm a crawl is genuinely CCBot and not an impostor, verify the request against Common Crawl's published IP ranges, which it lists at index.commoncrawl.org/ccbot.json, rather than trusting the user-agent string alone.

A couple of details keep this from backfiring. Spell the token exactly as CCBot, since a typo in a user-agent line does nothing at all.

And do not pair a CCBot block with a blanket disallow that also catches your search crawlers. If you block everything, you lose live citations along with the training opt-out, which is almost never what you want. For a fuller way to manage all of this in one place, see our guide on llms.txt.

The verification step matters more than it sounds.

Plenty of scrapers set their user-agent to CCBot to look legitimate and dodge blocks. Because Common Crawl publishes its real IP ranges, you can check any suspicious request against that list and tell a genuine crawl from a spoof. If you are going to make blocking decisions off your server logs, verify the IP first, or you are reacting to traffic that was never Common Crawl.

The mistakes people make with CCBot

Most CCBot mistakes come from treating it like a live AI-answer crawler. It is not one. It is a training archiver, and confusing the two leads to blocks that cost you brand familiarity while protecting nothing you actually care about. Here are the ones that show up most often.

• Blocking CCBot to stop ChatGPT from citing you. Citations come from OAI-SearchBot, not CCBot, so this does nothing to live answers.
• Assuming a block deletes you from existing models. Past training snapshots and archived pages stay put; robots.txt only affects future crawls.
• Blocking CCBot while leaving GPTBot and Google-Extended allowed, so you opt out of the open corpus but stay in the vendor-specific ones.
• Trusting the user-agent string. Scrapers spoof CCBot, so verify against the published IP ranges before acting on log data.
• Copying a blanket anti-AI snippet that blocks CCBot without deciding whether you actually want out of the training corpus.

Notice that most of those make you less familiar to future models, not more protected today.

The instinct to block every AI crawler is understandable, but CCBot is the one where a blunt block has the least upside. It gives up your place in the open corpus without buying back any live visibility, because there was never any live visibility running through it to begin with.

Frequently asked questions

Decide CCBot on training, not on visibility

Do this next: open your robots.txt and decide CCBot on one question only, whether you want to be in the open training corpus. Your AI-search visibility does not depend on it, so do not block CCBot hoping to protect citations.

Then measure what actually drives visibility. Run your buyer prompts through ChatGPT, Claude, Gemini, and Perplexity with MentionsAPI and watch who gets cited, because that is decided by live search bots and your content, not by CCBot.