picked
Start tracking

meta-webindexer: Meta's search crawler

meta-webindexer is the crawler Meta uses to improve Meta AI's search results.

See if it reads your siteMeta's docs

Picked is an AI visibility tool for agencies and B2B SaaS teams. It asks ChatGPT, Claude, Gemini, Perplexity and Google's AI answers the prompts your buyers ask, records which brands and pages they name, and writes the article built to become the source they quote.

At a glance

meta-webindexer in one table

Company
Meta
What it's for
Search index: read into the index an ai assistant searches.
robots.txt token
meta-webindexer
Full user agent
meta-webindexer/1.1
Follows robots.txt
Follows it. Blocking it stops Meta AI from citing and linking the site's pages in its answers.
How to verify
Meta publishes no IP list or other way to check, so a visit with this user agent can't be verified.
Source
Meta's documentation, read on October 12, 2026.

Real data

What meta-webindexer read on a real site

LaunchOn.it, our own launch platform, tracked in Picked through Cloudflare from September 12 to October 11, 2026. One site, so read it as an example, not an average.

  • 928

    Requests in 30 days

  • 158

    Different pages read

  • 6 of 30

    Days it came

  • 99%

    Verified by Cloudflare as the real bot

Requests per dayBusiest day: 214
Sep 12Sep 19Sep 27Oct 4Oct 11

100% of its requests got a page back; 0% hit an address that doesn't exist.

What it read most
  • /products/19%
  • /_next/image17%
  • /categories/7%
  • /tech/7%
  • /_next/6%

By the first folder of the address. /_next/ holds the site's scripts and images.

robots.txt

Allow meta-webindexer

Allow it. It builds the index Meta's answers are drawn from, so blocking it means those answers can't cite your pages.

robots.txt· Allow it
User-agent: meta-webindexerAllow: /

Needed only if a wider rule blocks it: robots.txt lets every bot in by default. A bot named in its own group ignores the User-agent: * group, so repeat your private paths there as Disallow lines. A block in your CDN or firewall has to be lifted there.

A rule only changes what the bot reads next. To see whether meta-webindexer reads your pages now, and which ones, track AI crawlers on your server or see the full AI crawler list.

Questions

Frequently asked questions

From Meta's documentation, read on October 12, 2026.

meta-webindexer is the crawler Meta uses to improve Meta AI's search results.

Blocking it stops Meta AI from citing and linking the site's pages in its answers.

Meta publishes no IP list or other way to check, so a visit with this user agent can't be verified.

Allow it. It builds the index Meta's answers are drawn from, so blocking it means those answers can't cite your pages.

Find out what ChatGPT and Google AI say about you

Add your domain and the prompts your buyers ask. Unlimited seats for your whole team, and no promises we can't keep.

Start trackingCompare plans