picked
Start tracking

CCBot: Common Crawl's training crawler

CCBot is the crawler of Common Crawl, a non-profit that publishes an open archive of the web. Many AI models were trained on that archive.

See if it reads your siteCommon Crawl's docs

Picked is an AI visibility tool for agencies and B2B SaaS teams. It asks ChatGPT, Claude, Gemini, Perplexity and Google's AI answers the prompts your buyers ask, records which brands and pages they name, and writes the article built to become the source they quote.

At a glance

CCBot in one table

Company
Common Crawl
What it's for
Training: collected to train future models.
robots.txt token
CCBot
Full user agent
CCBot/2.0 (https://commoncrawl.org/faq/)
Follows robots.txt
Follows it. Blocking it keeps the site's future pages out of Common Crawl's public archive.
How to verify
Check the visitor's IP against the list Common Crawl publishes. A request outside it isn't CCBot, whatever its user agent says. The IP list
Source
Common Crawl's documentation, read on October 12, 2026.

Real data

What CCBot read on a real site

LaunchOn.it, our own launch platform, tracked in Picked through Cloudflare from September 12 to October 11, 2026. One site, so read it as an example, not an average.

  • 166

    Requests in 30 days

  • 77

    Different pages read

  • 3 of 30

    Days it came

  • 0%

    Verified by Cloudflare as the real bot

Requests per dayBusiest day: 128
Sep 12Sep 19Sep 27Oct 4Oct 11

48% of its requests got a page back; 0% hit an address that doesn't exist. The rest were redirects or refusals.

What it read most
  • /_next/52%
  • /products/11%
  • /launch-platforms/5%
  • /categories/4%
  • /tech/3%

By the first folder of the address. /_next/ holds the site's scripts and images.

robots.txt

Allow CCBot

Allowing it lets your future pages into what the model learns, which is how it knows your brand before it searches. Opting out is a choice you can make if your content is your product: search and live-answer bots run under other names, so it doesn't take you out of today's answers.

robots.txt· Allow it
User-agent: CCBotAllow: /

Needed only if a wider rule blocks it: robots.txt lets every bot in by default. A bot named in its own group ignores the User-agent: * group, so repeat your private paths there as Disallow lines. A block in your CDN or firewall has to be lifted there.

robots.txt· Or opt out of training
User-agent: CCBotDisallow: /

A rule only changes what the bot reads next. To see whether CCBot reads your pages now, and which ones, track AI crawlers on your server or see the full AI crawler list.

Questions

Frequently asked questions

From Common Crawl's documentation, read on October 12, 2026.

CCBot is the crawler of Common Crawl, a non-profit that publishes an open archive of the web. Many AI models were trained on that archive.

Blocking it keeps the site's future pages out of Common Crawl's public archive.

Check the visitor's IP against the list Common Crawl publishes. A request outside it isn't CCBot, whatever its user agent says.

Allowing it lets your future pages into what the model learns, which is how it knows your brand before it searches. Opting out is a choice you can make if your content is your product: search and live-answer bots run under other names, so it doesn't take you out of today's answers.

Find out what ChatGPT and Google AI say about you

Add your domain and the prompts your buyers ask. Unlimited seats for your whole team, and no promises we can't keep.

Start trackingCompare plans