picked
Start tracking

AI crawler list: every AI bot and why it reads your site

An AI crawler is a bot an AI company sends to read web pages: to train its models, to build the index its assistant searches, or to open a page live because someone asked a question. These are the 34 we know by user agent, each checked against its owner's documentation on October 12, 2026.

See which ones read your siteDetect them with npm

Picked is an AI visibility tool for agencies and B2B SaaS teams. It asks ChatGPT, Claude, Gemini, Perplexity and Google's AI answers the prompts your buyers ask, records which brands and pages they name, and writes the article built to become the source they quote.

Live answer · 10

Live answers

Sent because a person asked an assistant a question just now and it opened your page to answer. Each one is a reader, not a crawl.

  • ChatGPT-UserOpenAI

    robots.txt: may ignore it · Verify by: published ips

  • Claude-UserAnthropic

    robots.txt: follows it · Verify by: published ips

  • Perplexity-UserPerplexity

    robots.txt: may ignore it · Verify by: published ips

  • meta-externalfetcherMeta

    robots.txt: may ignore it · Verify by: not published

  • MistralAI-UserMistral

    robots.txt: not stated · Verify by: published ips

  • DuckAssistBotDuckDuckGo

    robots.txt: follows it · Verify by: published ips

  • Amzn-UserAmazon

    robots.txt: may ignore it · Verify by: published ips

  • Google-GeminiNotebookGoogle

    robots.txt: may ignore it · Verify by: not published

  • Grok-DeepSearchxAI

    robots.txt: no official page · Verify by: no official page

  • Shap-UserParallel

    robots.txt: not stated · Verify by: not published

AI agent · 3

AI agents

An agent browsing for someone: comparing products, filling a form, reading docs for a developer.

  • Google-AgentGoogle

    robots.txt: may ignore it · Verify by: published ips

  • CursorCursor

    robots.txt: no official page · Verify by: no official page

  • opencodeOpenCode

    robots.txt: no official page · Verify by: no official page

Training · 10

Training crawlers

Collect pages that may train future models. Blocking them doesn't take you out of today's answers.

  • GPTBotOpenAI

    robots.txt: follows it · Verify by: published ips

  • ClaudeBotAnthropic

    robots.txt: follows it · Verify by: published ips

  • meta-externalagentMeta

    robots.txt: follows it · Verify by: not published

  • MistralAI-TrainingMistral

    robots.txt: follows it · Verify by: not published

  • AmazonbotAmazon

    robots.txt: follows it · Verify by: published ips

  • GoogleOtherGoogle

    robots.txt: follows it · Verify by: published ips

  • BytespiderByteDance

    robots.txt: no official page · Verify by: no official page

  • CCBotCommon Crawl

    robots.txt: follows it · Verify by: published ips

  • GrokBotxAI

    robots.txt: no official page · Verify by: no official page

  • AI2BotAi2

    robots.txt: not stated · Verify by: not published

robots.txt

Allow the bots that put you in answers

Search crawlers and live-answer fetchers are how your pages reach ChatGPT, Claude, Perplexity and the rest. Block them and you're out of those answers.

  • Buyers ask AI first

    Someone who asks ChatGPT or Google's AI for the best tool for their job is shopping. The brands the answer names get the visit, and ChatGPT tags those clicks with utm_source=chatgpt.com, so you can count them.

  • Search crawlers build the index

    Block OAI-SearchBot and ChatGPT search can't show your pages. Block Claude-SearchBot and Anthropic says your visibility in Claude's answers may drop.

  • Live answers are readers

    ChatGPT-User, Claude-User and Perplexity-User come because a person asked a question your page helps answer, right now. Where they honour robots.txt, a block costs you that answer.

robots.txt· Allow AI search and live answers
User-agent: OAI-SearchBotUser-agent: ChatGPT-UserUser-agent: Claude-SearchBotUser-agent: Claude-UserUser-agent: PerplexityBotUser-agent: Perplexity-UserUser-agent: ApplebotUser-agent: Amzn-SearchBotUser-agent: Amzn-UserUser-agent: meta-webindexerUser-agent: meta-externalfetcherUser-agent: MistralAI-IndexUser-agent: MistralAI-UserUser-agent: DuckAssistBotUser-agent: YouBotAllow: /# Repeat the private paths from your User-agent: * group:Disallow: /admin/

Before you paste it

  • You may not need it

    robots.txt lets every bot in by default. Add the group only if a wider rule blocks them, like User-agent: * with Disallow: /.

  • Repeat your private paths

    A bot named in its own group ignores the User-agent: * group. Copy your Disallow lines into it; /admin/ stands for yours.

  • Never block Googlebot or Bingbot

    Google's AI Overviews and AI Mode read pages as Googlebot, and Copilot answers from Bing's index.

  • Give it a day

    Bots reread robots.txt on their own schedule. Meta and Perplexity say up to 24 hours, DuckDuckGo 72.

Optional

What about training crawlers?

Most robots.txt guides tell you to block them. That suits publishers, whose pages are the product. If you sell software or a service, we'd let them in: what a model learned is how it knows your brand before it searches anything.

They don't decide today's answers. OpenAI, Anthropic, Meta and Mistral run search and live answers under other names, and Google and Apple give training its own token. So if your content is what you sell, opting out costs you nothing in the answers. It only stops future crawls; a model keeps what it already learned.

robots.txt· Opt out of AI training only
User-agent: GPTBotUser-agent: ClaudeBotUser-agent: Google-ExtendedUser-agent: Applebot-ExtendedUser-agent: meta-externalagentUser-agent: MistralAI-TrainingUser-agent: CCBotDisallow: /

Beyond robots.txt

Make sure they can read you

A bot you allow can still leave with nothing. Four things decide whether the visit counts.

  1. 1

    Put the content in the HTML

    OpenAI's and Anthropic's crawlers download JavaScript but don't run it, while Gemini (through Googlebot) and Applebot do. A page that builds its text in the browser is blank to ChatGPT and Claude. Render prices, features and answers on the server.

  2. 2

    Check your CDN and firewall

    A robots.txt that allows everyone doesn't help if Cloudflare's AI Crawl Control, Bot Fight Mode or a WAF rule turns the bot away first. Open those settings and look for AI search and agent bots in the blocked list.

  3. 3

    Fix 404s and redirect chains

    In Vercel and MERJ's logs, 34.8% of ChatGPT's fetches and 34.2% of Claude's hit a 404, against 8.2% for Googlebot, and ChatGPT spent 14.4% of its fetches on redirects. Keep the sitemap current and redirect old URLs in one hop.

  4. 4

    Give every subdomain its own robots.txt

    Rules apply per host: docs.example.com doesn't read example.com/robots.txt. And robots.txt only controls crawling. To keep a page out of answers and search, use noindex; to keep it private, put it behind a login.

Rendering, 404 and redirect figures: Vercel and MERJ, The rise of the AI crawler, December 2024. Crawlers may have changed since.

How we checked

From each company's own documentation

Names, user agents, robots.txt rules and verification come from these pages, read on October 12, 2026. A bot whose owner publishes nothing stays on the list without a page of its own.

The same list powers Picked's crawler analytics and the open-source @picked-so/crawler-detect package. To see which of these bots read your pages, connect Cloudflare, Vercel or your own server to the AI visibility tool.

Questions

Frequently asked questions

Short answers, each one checked against the owners' docs linked above.

A bot an AI company sends to read web pages. Some collect pages to train models (GPTBot, ClaudeBot), some build the index an assistant searches (OAI-SearchBot, PerplexityBot), and some open a page live because a person asked a question (ChatGPT-User, Claude-User).

Allow the search crawlers and live-answer fetchers: they're how your pages get into ChatGPT, Claude, Perplexity and other AI answers, and blocking them takes you out. Training crawlers are a separate choice; opting out of them doesn't remove you from today's answers.

Check the visitor's IP address against the ranges the company publishes (OpenAI's are at openai.com/gptbot.json), or by reverse DNS where the owner documents it. Anyone can send GPTBot as a user agent, and many scanners do.

Googlebot. Google's AI answers read the pages Google Search has already crawled, so there's no separate user agent to allow or block. Google-Extended is a robots.txt token that controls Gemini training and grounding, not a crawler.

No. robots.txt only stops future crawls. Pages a model was already trained on stay in it, and pages already in a search index can stay there until the owner drops them.

No. robots.txt applies to one host, so blog.example.com and docs.example.com each need their own file. A bot reads the one at the root of the host it's on.

xAI publishes no crawler documentation, so GrokBot, xAI-Grok and Grok-DeepSearch are listed by the names bot directories agree on, without pages of their own. DeepSeek publishes no user agent at all, and Copilot reads pages through Bingbot.

Find out what ChatGPT and Google AI say about you

Add your domain and the prompts your buyers ask. Unlimited seats for your whole team, and no promises we can't keep.

Start trackingCompare plans