Meta-ExternalAgent
Meta-ExternalAgent is the crawler Meta uses for uses such as training its foundation AI models and improving its products by indexing content. Meta says it follows robots.txt; it doesn’t currently publish the addresses it crawls from.
- Operator
- Meta
- Purpose
- Model training. Crawlers that collect pages to train models. Blocking them does not affect search or AI answers.
- robots.txt token
Meta-ExternalAgent- We test with
meta-externalagent/1.1 (+https://developers.facebook.com/docs/sharing/webmasters/crawler)
Meta’s string with the documentation link it carried before Meta’s 2026 documentation move.
What Meta-ExternalAgent does
The Meta-ExternalAgent crawler crawls the web for use cases such as training foundation AI models or improving products by indexing content directly.
developers.facebook.com
| Bot | What it does | Model training |
|---|---|---|
Meta-ExternalAgent | Crawls for training foundation models and indexing | Yes |
Meta-ExternalFetcher | Fetches links at a user’s request, including for agentic AI tasks | Not described as training |
Meta-WebIndexer | Indexes pages for citations in Meta AI search | Not described as training |
The Meta-ExternalAgent user agent
| Documented | User agent string |
|---|---|
| Now | meta-externalagent/1.1meta-externalagent/1.1 (+/documentation/sharing/webmasters/web-crawlers) |
| Until May 2026 | meta-externalagent/1.1 (+https://developers.facebook.com/docs/sharing/webmasters/crawler) |
Meta writes it in lower case. The link in its current form is a relative path, which looks like a slip from Meta’s 2026 documentation move, so it is unclear which form requests carry today. Match on meta-externalagent, case-insensitively, rather than any full string.
How Meta-ExternalAgent reads robots.txt
- The token is
meta-externalagent(robots.txt matching ignores case). - Allow a day. Meta says its crawlers may cache robots.txt for up to 24 hours.
- It doesn’t cover Meta-ExternalFetcher, which Meta says may bypass robots.txt because a user requested the fetch.
In order to block these crawlers, add a disallow for the relevant crawler to robots.txt.
developers.facebook.com
Meta-ExternalAgent IP addresses
Meta’s current documentation (updated May 2026) publishes no IP list or verification method for its crawlers, though it still suggests allowlisting by IP address. Earlier versions pointed to the addresses announced by Meta’s network, AS32934, which also carries Facebook, Instagram and WhatsApp traffic, so that isn’t a test for this crawler. Without an official list, the user agent is all a firewall can go on.
Block Meta-ExternalAgent at your firewall or CDN
robots.txt asks; a firewall enforces. A rule on the user agent refuses Meta-ExternalAgent even if it doesn’t honour robots.txt. Match the name case-insensitively and anywhere in the string, never a full string or a version.
Cloudflare
In Security → WAF → Custom rules, create a rule with this expression and the action Block. Cloudflare’s contains is case-sensitive, hence lower().
(lower(http.user_agent) contains "meta-externalagent")
Cloudflare also has one-click controls: AI bot policies, which block AI crawlers by what they do (search, agent or training), and AI Crawl Control, which lets you allow or block crawlers one by one and shows which of them fetch your pages.
nginx
# in the http block
map $http_user_agent $blocked_bot {
default 0;
~*meta-externalagent 1;
}
# in the server block
if ($blocked_bot) { return 403; }
Block Meta-ExternalAgent
Add a group naming it to /robots.txt. Compliant crawlers stop before requesting any page. A firewall rule on the user agent is the only way to enforce it against crawlers that ignore robots.txt. This keeps your pages out of Meta’s AI training crawls. Pages Meta AI opens on a user’s request come from Meta-ExternalFetcher, which may bypass robots.txt.
User-agent: Meta-ExternalAgent
Disallow: /
Allow Meta-ExternalAgent when everything else is blocked
A group naming the bot wins over User-agent: *, so an explicit allow lets it in while your catch-all rules stay in place.
User-agent: Meta-ExternalAgent
Allow: /
User-agent: *
Disallow: /
Does Meta-ExternalAgent reach your site?
The checker applies your robots.txt the way Meta-ExternalAgent does, then requests your page with the user agent above, alongside 29 other bots, and opens the report on Meta-ExternalAgent.
Questions about Meta-ExternalAgent
What is Meta-ExternalAgent?
Meta’s web crawler for use cases such as training foundation AI models and improving products by indexing content directly.
Does Meta-ExternalAgent respect robots.txt?
Meta says to block its crawlers by adding a disallow for the relevant crawler to robots.txt, and to allow up to 24 hours for changes to take effect.
What is the Meta-ExternalAgent user agent?
Meta documents it in lower case as meta-externalagent/1.1, with and without a documentation link. Match on meta-externalagent, case-insensitively.
What IP addresses does Meta-ExternalAgent use?
Meta’s current documentation doesn’t publish them. Its network carries all of Meta’s traffic, so an address in Meta’s ranges doesn’t prove a request came from this crawler.
Also run by Meta: Meta-ExternalFetcher.
Other ai training: GPTBot, ClaudeBot, Google-Extended, CCBot, Applebot-Extended, GoogleOther, Bytespider.
Build a complete file with the robots.txt generator.