Documentation
AI crawlers
The AI models don't only answer from memory. When someone asks a question, ChatGPT or Claude may open pages on the web to read them. Those visits appear in your logs under names like ChatGPT-User. Picked reads them so you can see which of your pages the AI opens, and why.
It is optional. Tracking prompts and writing articles work without it.
Four kinds of visit
Every AI crawler hit gets a purpose, so a training crawl doesn't look like a lead:
| Purpose | Bots | What it means |
|---|---|---|
| Live answer | ChatGPT-User, Claude-User, Perplexity-User, MistralAI-User, DuckAssistBot, Meta's fetcher | A person asked a question just now and the assistant opened your page to answer it. Any question, not only your prompts |
| AI agent | Google-Agent and other AI agents | An AI agent browsing your site on someone's behalf |
| Search index | OAI-SearchBot, Claude-SearchBot, PerplexityBot, Applebot, Amzn-SearchBot | Read into the index an AI assistant searches, so it can show your page in answers later |
| Training | GPTBot, ClaudeBot, Amazonbot, Bytespider, CCBot, GoogleOther, Meta's crawler | Collected to train future models. Your page may end up in an answer someday, or not |
Live answers are the ones that count most: each is someone asking about your subject right now.
Connect your site
You connect once per project, and a project reads from one source at a time. Two would count the same visit twice, so connecting one removes the other.
| Your site runs on | How to connect | What to know |
|---|---|---|
| Cloudflare | Sign in to Cloudflare and approve | Reads the last 30 days when you connect, then every hour. No code |
| Vercel | Install the Picked integration, which adds a log drain | Pro or Enterprise teams. Counts from the day you connect |
| Anything else | The open-source package, or a plain HTTP call | Next.js, Express, Workers, Hono, Bun and any server that can make a request |
Open Analytics in the dashboard and pick the option that matches. Each one walks you through it.
Cloudflare
- Choose Connect Cloudflare and sign in.
- Approve read access: zone, analytics and offline access, so the hourly sync runs without you. Picked can't change anything on your account.
- Pick the site that serves your domain, if the account has several.
Picked reads the last 30 days in a few seconds, then syncs every hour. Cloudflare samples busy sites, so counts there are close, not exact. More on the Cloudflare integration.
Vercel
- Choose Connect Vercel and install the Picked integration on the team that owns your site.
- Pick the project that serves your domain. Picked picks it for you when the domain matches.
Vercel sends every request as it happens and keeps no history, so the screen fills in from the first AI crawler visit after you connect. Drains are billed by Vercel on your team's bill, by volume. More on the Vercel integration.
Your server
- Choose Your server. The dialog shows your project's private address and the code for your stack.
- Install the package:
npm install @picked-so/crawler-detect, and setPICKED_INGEST_URLto the address. - Add one line: in
proxy.tson Next.js 16 (middleware.tson 13 to 15),app.use(aiCrawlers())on Express, ortrackAiCrawlerin any Fetch API handler. - Deploy.
The package sends only AI bot hits, after the response, so it never slows a page down or breaks one. From any other language, POST the hits as JSON to the same address. The address holds your project's key: keep it in server code, never in the browser. More on the server integration.
What you see
On Analytics, under your citations:
- Visits by day, split by purpose. Click a purpose to see it alone, and again for all.
- Pages: which of your pages the AI bots opened, how often against the days before, and how many answers to your prompts cite each one. A page no crawler read in the range says so.
- Crawlers: which bots came, the company behind each, and why.
- Turned away: requests a bot made that got a 401, 403 or 429 back, a firewall rule or rate limit between the bot and your page.
- Latest visits: the most recent hits, with the page, bot, purpose and country, when your source sends it.
- Visits from AI answers: people, not bots, who clicked through to your site from ChatGPT, Perplexity, Claude, Gemini or another assistant, counted by the referrer or ChatGPT's
utm_source=chatgpt.com, and the pages they landed on. It comes from Vercel, your server (package 0.2 or later) or Cloudflare on a plan whose analytics include referrers; Cloudflare's Free plan doesn't. Some AI apps send no referrer, and Google's AI Overviews and AI Mode arrive as Google search, so read it as a floor.
Visits are counted per hour, in UTC. The date range and topic filters at the top work here too.
Verified only
Anyone can send GPTBot as a user agent. Verified only keeps just the hits you can trust:
- Vercel and your server: hits whose IP address is in the ranges OpenAI, Anthropic, Perplexity, Google and Apple publish for their crawlers. Amazonbot is checked by reverse DNS.
- Cloudflare: the requests Cloudflare itself confirmed came from the company named.
A few bots, like Meta's, stay unverified because their owners publish no IP ranges. Their hits show, just not as verified.
Check "Turned away" first
If you block OAI-SearchBot by accident, ChatGPT search can't show your pages. If you block Claude-SearchBot or Claude-User, Anthropic says your visibility in Claude's search answers may drop. Blocking training crawlers is your call. Blocking the others takes you out of the answers.
Picked shows what the bots do; it doesn't change your settings. Allow or block them in robots.txt, your CDN's bot settings or your firewall.
Use it to check an article landed
After you publish a page, a crawler's first visit often comes before any answer changes. Watch the page in Pages: a search index bot means it can be found, a live answer means someone's question just opened it. Then Watch on the article shows whether the answers start naming you.
Disconnect
From the same Analytics screen. Picked removes its access (the Cloudflare grant, or the Vercel drain) and deletes the visits it stored for that project. For your server, disconnecting issues a new key, and the old address stops working.
Next
Write the article for a prompt you lose.
Updated
