Claude-SearchBot: Anthropic's search crawler
Claude-SearchBot is the crawler Anthropic uses to index pages for Claude's search results.
Picked is an AI visibility tool for agencies and B2B SaaS teams. It asks ChatGPT, Claude, Gemini, Perplexity and Google's AI answers the prompts your buyers ask, records which brands and pages they name, and writes the article built to become the source they quote.
At a glance
Claude-SearchBot in one table
- Company
- Anthropic
- What it's for
- Search index: read into the index an ai assistant searches.
- robots.txt token
Claude-SearchBot- Full user agent
- Anthropic publishes the token only, not the full string.
- Follows robots.txt
- Follows it. Anthropic says blocking it keeps the site out of the index Claude searches, which may lower its visibility and accuracy in Claude's search answers.
- How to verify
- Check the visitor's IP against the list Anthropic publishes. A request outside it isn't Claude-SearchBot, whatever its user agent says. The IP list
- Source
- Anthropic's documentation, read on October 12, 2026.
robots.txt
Allow Claude-SearchBot
Allow it. It builds the index Anthropic's answers are drawn from, so blocking it means those answers can't cite your pages.
User-agent: Claude-SearchBotAllow: /Needed only if a wider rule blocks it: robots.txt lets every bot in by default. A bot named in its own group ignores the User-agent: * group, so repeat your private paths there as Disallow lines. A block in your CDN or firewall has to be lifted there.
A rule only changes what the bot reads next. To see whether Claude-SearchBot reads your pages now, and which ones, track AI crawlers on your server or see the full AI crawler list.
Questions
Frequently asked questions
From Anthropic's documentation, read on October 12, 2026.
Claude-SearchBot is the crawler Anthropic uses to index pages for Claude's search results.
Anthropic says blocking it keeps the site out of the index Claude searches, which may lower its visibility and accuracy in Claude's search answers.
Check the visitor's IP against the list Anthropic publishes. A request outside it isn't Claude-SearchBot, whatever its user agent says.
Allow it. It builds the index Anthropic's answers are drawn from, so blocking it means those answers can't cite your pages.
Find out what ChatGPT and Google AI say about you
Add your domain and the prompts your buyers ask. Unlimited seats for your whole team, and no promises we can't keep.