Quick answer: Meta-ExternalAgent is the crawler Meta uses for purposes such as training AI models and improving products by indexing content directly. It follows robots.txt. It is separate from facebookexternalhit, which fetches link previews when someone shares your URL, so blocking one does not block the other.
- Operator
- Meta
- Type
- Training crawler
- robots.txt token
meta-externalagent- Follows robots.txt
- Yes
- Used for
- Crawling for uses such as training AI models or improving products by indexing content
- User agent
meta-externalagent/1.1 (+https://developers.facebook.com/docs/sharing/webmasters/crawler)- How to verify
- Check that the IP belongs to Meta's network (AS32934)
What is Meta-ExternalAgent?
For years the only Meta agent most site owners saw was facebookexternalhit, which builds the preview card when a link is shared. Meta-ExternalAgent is newer and has a different purpose: collecting content for Meta's AI work, including model training.
Meta also documents a user-triggered fetcher, meta-externalfetcher, for requests made on behalf of users. Like other user-initiated agents, it may not follow robots.txt.
What happens if you block Meta-ExternalAgent?
Disallowing meta-externalagent tells Meta not to crawl your site for these AI and indexing uses. Link previews on Facebook, Instagram, WhatsApp and Messenger keep working because they use facebookexternalhit, which you should leave allowed.
Should you allow Meta-ExternalAgent?
This is mainly a training decision. Meta AI is widely used inside Meta's apps, so if you want those assistants to know your brand, allow it. If you opt out of training elsewhere, blocking it keeps your policy consistent. Never block facebookexternalhit by accident, or shared links lose their previews.
Check whether Meta-ExternalAgent can reach your site
Enter your domain or a specific page. The checker reads your live robots.txt, picks the group Meta-ExternalAgent would use, and applies the longest matching rule, the same way major crawlers resolve conflicts.
robots.txt rules for Meta-ExternalAgent
Put each group in the robots.txt file at the root of your domain. User agent names are matched without regard to case, and the most specific matching group wins.
User-agent: meta-externalagent
Allow: /User-agent: meta-externalagent
Disallow: /User-agent: meta-externalagent
Disallow: /account/
Disallow: /checkout/
Allow: /The rule most people get wrong: once Meta-ExternalAgent has its own group, it ignores everything under User-agent: *. If your wildcard group disallows paths such as /admin/, repeat those lines inside the Meta-ExternalAgent group too.
How to verify real Meta-ExternalAgent traffic
Meta's crawlers operate from Meta's own network, AS32934. A WHOIS or ASN lookup on the source IP should show Meta as the owner. Requests that use this name from other networks are not Meta.
To see how often it visits, count requests by user agent in your access log. On a typical Nginx server:
grep -i "meta-externalagent" /var/log/nginx/access.log | awk '{print $4}' | cut -d: -f1 | sort | uniq -cMeta-ExternalAgent compared with similar agents
| Agent | Operator | Type | Follows robots.txt |
|---|---|---|---|
| Meta-ExternalAgent | Meta | Training crawler | Yes |
| GPTBot | OpenAI | Training crawler | Yes |
| ClaudeBot | Anthropic | Training crawler | Yes |
| Bytespider | ByteDance | Training crawler | Widely reported as inconsistent |
See all 20 agents side by side in the AI crawler directory.
Meta-ExternalAgent FAQ
See every AI crawler signal at once
The free AI SEO audit checks robots.txt access for 12 AI crawlers, plus llms.txt, schema, content and metadata, in a few seconds.
Run the free AI SEO audit