ClaudeBot: Anthropic's training crawler
ClaudeBot is Anthropic's crawler for public web content that may be used to train Claude.
Picked is an AI visibility tool for agencies and B2B SaaS teams. It asks ChatGPT, Claude, Gemini, Perplexity and Google's AI answers the prompts your buyers ask, records which brands and pages they name, and writes the article built to become the source they quote.
At a glance
ClaudeBot in one table
- Company
- Anthropic
- What it's for
- Training: collected to train future models.
- robots.txt token
ClaudeBot- Full user agent
- Anthropic publishes the token only, not the full string.
- Follows robots.txt
- Follows it. Blocking it tells Anthropic to leave the site's future content out of its training data.
- How to verify
- Check the visitor's IP against the list Anthropic publishes. A request outside it isn't ClaudeBot, whatever its user agent says. The IP list
- Source
- Anthropic's documentation, read on October 12, 2026.
What to know
ClaudeBot, in practice
ClaudeBot collects pages that may train Claude. Anthropic runs two other agents for answers, Claude-SearchBot for the search index and Claude-User for live fetches, and blocking ClaudeBot touches neither.
Older guides list anthropic-ai and Claude-Web. Anthropic's page names only ClaudeBot, Claude-User and Claude-SearchBot, so a robots.txt that names the old tokens doesn't speak to the bots that visit now.
It rereads robots.txt and sitemap.xml often: on our own site both were among its most requested addresses. A sitemap that lists only live pages saves it, and you, a lot of 404s.
Real data
What ClaudeBot read on a real site
LaunchOn.it, our own launch platform, tracked in Picked through Cloudflare from September 12 to October 11, 2026. One site, so read it as an example, not an average.
2,676
Requests in 30 days
1,428
Different pages read
24 of 30
Days it came
100%
Verified by Cloudflare as the real bot
95% of its requests got a page back; 1% hit an address that doesn't exist. The rest were redirects or refusals.
- /compare/15%
- /tech/15%
- /products/11%
- /use-cases/10%
- /robots.txt9%
By the first folder of the address. /_next/ holds the site's scripts and images.
robots.txt
Allow ClaudeBot
Allowing it lets your future pages into what the model learns, which is how it knows your brand before it searches. Opting out is a choice you can make if your content is your product: search and live-answer bots run under other names, so it doesn't take you out of today's answers.
User-agent: ClaudeBotAllow: /Needed only if a wider rule blocks it: robots.txt lets every bot in by default. A bot named in its own group ignores the User-agent: * group, so repeat your private paths there as Disallow lines. A block in your CDN or firewall has to be lifted there.
User-agent: ClaudeBotDisallow: /A rule only changes what the bot reads next. To see whether ClaudeBot reads your pages now, and which ones, track AI crawlers on your server or see the full AI crawler list.
Questions
Frequently asked questions
From Anthropic's documentation, read on October 12, 2026.
ClaudeBot is Anthropic's crawler for public web content that may be used to train Claude.
Blocking it tells Anthropic to leave the site's future content out of its training data.
Check the visitor's IP against the list Anthropic publishes. A request outside it isn't ClaudeBot, whatever its user agent says.
Allowing it lets your future pages into what the model learns, which is how it knows your brand before it searches. Opting out is a choice you can make if your content is your product: search and live-answer bots run under other names, so it doesn't take you out of today's answers.
Find out what ChatGPT and Google AI say about you
Add your domain and the prompts your buyers ask. Unlimited seats for your whole team, and no promises we can't keep.