# AI crawlers

> See which of your pages GPTBot, ChatGPT-User, ClaudeBot and other AI crawlers open, and how to connect Cloudflare, Vercel or your own server.

Canonical: https://picked.so/docs/ai-crawlers
Last updated: 2026-10-16

The AI models don't only answer from memory. When someone asks a question, ChatGPT or Claude may open pages on the web to read them. Those visits appear in your logs under names like `ChatGPT-User`. Picked reads them so you can see which of your pages the AI opens, and why.

It is optional. Tracking prompts and writing articles work without it.

## Four kinds of visit

Every AI crawler hit gets a purpose, so a training crawl doesn't look like a lead:

| Purpose | Bots | What it means |
| --- | --- | --- |
| **Live answer** | `ChatGPT-User`, `Claude-User`, `Perplexity-User`, `MistralAI-User`, `Amzn-User`, `DuckAssistBot`, `Shap-User`, `Grok-DeepSearch`, Meta's fetcher | A person asked a question just now and the assistant opened your page to answer it. Any question, not only your prompts |
| **AI agent** | `Google-Agent`, coding agents (`Cursor`, `OpenCode`) and other AI agents | An AI agent browsing your site on someone's behalf, or a coding agent reading your docs for a developer |
| **Search index** | `OAI-SearchBot`, `Claude-SearchBot`, `PerplexityBot`, `MistralAI-Index`, `meta-webindexer`, `ShapBot`, `Applebot`, `Amzn-SearchBot`, `YouBot`, `Google-CloudVertexBot`, `xAI-Grok` | Read into the index an AI assistant searches, so it can show your page in answers later |
| **Training** | `GPTBot`, `ClaudeBot`, `MistralAI-Training`, `Amazonbot`, `Bytespider`, `CCBot`, `AI2Bot`, `GrokBot`, `GoogleOther`, Meta's crawler | Collected to train future models. Your page may end up in an answer someday, or not |

Live answers are the ones that count most: each is someone asking about your subject right now. Every bot, with its user agent, robots.txt rules and how to verify it, is on the [AI crawler list](https://picked.so/ai-crawlers).

## Connect your site

You connect once per project, and a project reads from one source at a time. Two would count the same visit twice, so connecting one removes the other.

| Your site runs on | How to connect | What to know |
| --- | --- | --- |
| Cloudflare | Sign in to Cloudflare and approve | Reads the last 30 days when you connect, then every hour. No code |
| Vercel | Install the Picked integration, which adds a log drain | Pro or Enterprise teams. Counts from the day you connect |
| Anything else | The open-source package, or a plain HTTP call | Next.js, Express, Workers, Hono, Bun and any server that can make a request |

Open **Analytics** in the dashboard and pick the option that matches. Each one walks you through it.

### Cloudflare

1. Choose **Connect Cloudflare** and sign in.
2. Approve read access: zone, analytics and offline access, so the hourly sync runs without you. Picked can't change anything on your account.
3. Pick the site that serves your domain, if the account has several.

Picked reads the last 30 days in a few seconds, then syncs every hour. Cloudflare samples busy sites, so counts there are close, not exact. More on the [Cloudflare integration](https://picked.so/integrations/cloudflare).

### Vercel

1. Choose **Connect Vercel** and install the Picked integration on the team that owns your site.
2. Pick the project that serves your domain. Picked picks it for you when the domain matches.

Vercel sends every request as it happens and keeps no history, so the screen fills in from the first AI crawler visit after you connect. Drains are billed by Vercel on your team's bill, by volume. More on the [Vercel integration](https://picked.so/integrations/vercel).

### Your server

1. Choose **Your server**. The dialog shows your project's private address and the code for your stack.
2. Install the package: `npm install @picked-so/crawler-detect`, and set `PICKED_INGEST_URL` to the address.
3. Add one line: in `proxy.ts` on Next.js 16 (`middleware.ts` on 13 to 15), `app.use(aiCrawlers())` on Express, or `trackAiCrawler` in any Fetch API handler.
4. Deploy.

The package sends only AI bot hits, after the response, so it never slows a page down or breaks one. Working with an AI assistant like Cursor or Claude Code? The dialog includes a ready-made prompt that installs the package, sets your environment variable and wires the middleware automatically. From any other language, POST the hits as JSON to the same address. The address holds your project's key: keep it in server code, never in the browser. More on the [server integration](https://picked.so/integrations/server).

## What you see

**Analytics** is about what AI does on your site. What AI answers say and link is on **Prompts**, under Cited sources.

- **AI visits:** every AI crawler request in the range, with the AI models behind them ranked (ChatGPT, Claude, Meta AI, Petal Search and the rest). Click one to chart it alone.
- **AI visit activity:** visits by day, or by the hour on a one-day range. Today is dashed while it's still counting. Click a day to open it hour by hour, with every crawler that came and what it reads pages for.
- **Pages AI read:** which of your pages the bots opened, how often against the days before, which AI models read each, the live answers among them, and how many answers to your prompts cite it. Above the table, **What AI reads** splits every page read by type from its URL (homepage, docs and help, pricing, comparisons, blog, product pages, `llms.txt`, sign-in, legal); click a type to filter. Filter by page or AI model, and sort by visits, change, live answers or last visit.
- **Recent visits:** every hit in the range, newest first, a page at a time, with the crawler, page, purpose and country when your source sends it. **Pause live updates** holds the list still while you read.
- **Visits from AI answers:** people, not bots, who clicked through to your site from ChatGPT, Perplexity, Claude, Gemini or another assistant, counted by the referrer or ChatGPT's `utm_source=chatgpt.com`, and the pages they landed on. It comes from Vercel, your server (package 0.2 or later) or Cloudflare on a plan whose analytics include referrers; Cloudflare's Free plan doesn't. Some AI apps send no referrer, and Google's AI Overviews and AI Mode arrive as Google search, so read it as a floor.
- **Turned away:** requests a bot made that got a 401, 403 or 429 back, a firewall rule or rate limit between the bot and your page. It opens by itself, with a note at the top of the page, while a public page is still being blocked.

**All purposes** at the top narrows everything to one reason a bot came: live answers, AI agents, search index or training. Visits are counted per hour, in UTC.

While Analytics is open on a range that ends today, Picked reads Cloudflare again about once a minute, so new visits show up within a few minutes and the total moves on its own. Vercel and your server send visits as they happen.

### Verified only

Anyone can send `GPTBot` as a user agent. **Verified only** keeps just the hits you can trust:

- **Vercel and your server:** hits whose IP address is in the ranges OpenAI, Anthropic, Perplexity, Google, Apple, Mistral, DuckDuckGo, Common Crawl and Parallel publish for their crawlers. Amazonbot and YouBot are checked by reverse DNS.
- **Cloudflare:** the requests Cloudflare itself confirmed came from the company named.

A few bots stay unverified on Vercel and your server. For most, their owners publish no IP ranges we can read: Meta's, `Amzn-User`, `Shap-User`, `AI2Bot`, `MistralAI-Training`, `Bytespider`, xAI's three and the coding agents (Cursor, OpenCode). `PetalBot` and `Reflectionbot` do publish a way to check them, which Picked doesn't use yet. Their hits show, just not as verified.

It's on when you open Analytics. Turn it off to see every hit, the unverified ones included.

## Check "Turned away" first

If you block `OAI-SearchBot` by accident, ChatGPT search can't show your pages. If you block `Claude-SearchBot` or `Claude-User`, Anthropic says your visibility in Claude's search answers may drop. Blocking training crawlers is your call. Blocking the others takes you out of the answers.

Picked shows what the bots do; it doesn't change your settings. Allow or block them in `robots.txt`, your CDN's bot settings or your firewall.

## Use it to check an article landed

After you publish a page, a crawler's first visit often comes before any answer changes. Watch the page in **Pages**: a search index bot means it can be found, a live answer means someone's question just opened it. Then [Watch](https://picked.so/docs/articles#watching-it-work) on the article shows whether the answers start naming you.

## Disconnect

From the same Analytics screen. Picked removes its access (the Cloudflare grant, or the Vercel drain) and deletes the visits it stored for that project. For your server, disconnecting issues a new key, and the old address stops working.

## Next

[Write the article](https://picked.so/docs/articles) for a prompt you lose.
