Meta-ExternalAgent: Should You Allow or Block It?

Most AI-crawler decisions are simple. This one has a catch.

With a crawler like Applebot-Extended, blocking it opts you out of AI training and costs you nothing else. Meta-ExternalAgent is not that clean.

Meta says it crawls both to train models and to index content for its products, so the same block that opts you out of training also opts you out of being seen inside Meta AI. That is the whole decision, and it is worth getting right.

And Meta AI is not a small surface. It is built into Facebook, Instagram, WhatsApp, and Messenger, apps that a large part of the planet opens every day.

So the visibility you would give up by blocking is not hypothetical. It is placement in front of one of the biggest audiences any assistant can reach, which is why this call deserves more thought than a reflexive block.

What is Meta-ExternalAgent?

Meta-ExternalAgent is Meta's web crawler. In its own developer docs, Meta says it crawls the web for use cases such as training foundation AI models like Llama or improving products by indexing content directly. That content feeds Meta AI, the assistant built into Facebook, Instagram, WhatsApp, and Messenger. It identifies itself as meta-externalagent/1.1 and honors robots.txt.

Meta launched it in mid-2024, and it scaled fast.

Its share of AI crawling climbed from about 0.9% to 7.5% in its first year, per research into Meta's crawlers, putting it among the highest-volume AI crawlers alongside GPTBot, ClaudeBot, Bytespider, and Amazonbot. When Meta wants training data, this is the bot that gets it.

The dual purpose is the detail everyone misses. Read Meta's own description again: it crawls to train models or to index content for products.

That little word, indexing, is why this bot is different. Most AI-training crawlers only feed training. This one also decides whether you show up when a Meta AI user asks a question, so it sits on the visibility side of the ledger, not just the training side.

Meta-ExternalAgent is not just a training bot. It also indexes you for Meta AI, so blocking it has a visibility cost.The one line that changes the decision

Meta-ExternalAgent vs Meta-ExternalFetcher vs facebookexternalhit

Meta runs several crawlers, and you control each one separately in robots.txt. The three that matter for this decision are the training-and-indexing agent, the live fetcher, and the link-preview bot. Blocking the wrong one breaks something you wanted to keep.

Meta also documents two more tokens, meta-webindexer and meta-externalads, so its full family is broader than three. For the block-or-allow question, the three above are the ones that change your outcome.

The reason this table matters is that people reach for one blunt rule. They see a Meta bot, decide they are anti-AI, and block everything with meta in the name.

That is how sites accidentally break their own Facebook and Instagram link previews, which run on facebookexternalhit, a bot that has nothing to do with AI training. Precision here is not pedantry. It is the difference between opting out of Llama and quietly making every shared link look broken.

Does Meta-ExternalAgent respect robots.txt?

Yes, Meta-ExternalAgent honors robots.txt. Meta states it in the crawler docs, and a disallow rule on the meta-externalagent token is the supported way to opt out of its crawl. That single rule covers both the training and the indexing it does, since one bot handles both.

If you want to confirm the traffic hitting your logs is really Meta, check the source. Meta publishes the details that let you verify it, rather than trusting the user-agent string alone.

That matters because a user-agent is trivial to fake, and plenty of scrapers pose as meta-externalagent to look legitimate. Verify by the published network before you decide a real Meta bot ignored your block, or you may be chasing an impostor instead.

Two caveats keep this from being airtight.

First, Meta-ExternalFetcher, the live-request bot, may bypass robots.txt because a person triggered it, so a user who pastes your link into Meta AI can still pull the page. Second, some webmasters have reported meta-externalagent crawling after a block, the same stealth pattern seen with Bytespider and PerplexityBot.

If a block must truly hold, back it with a firewall or WAF rule that verifies the bot by its published autonomous system, not just a robots.txt line. For more on how AI crawlers ignore robots.txt, see our guide on Bytespider.

The practical read: robots.txt is enough for a policy signal, not for a hard wall. If you just want to tell Meta you opt out, the token does that.

If you need certainty that not one page is fetched, robots.txt alone will not get you there, because the fetcher and the occasional misbehaving crawl route around it. That gap matters for legal or compliance blocks, and far less for a simple preference.

Should you allow or block Meta-ExternalAgent?

For most brands, allow it. Because Meta-ExternalAgent both trains models and indexes content for Meta AI, blocking it opts you out of visibility across Facebook, Instagram, WhatsApp, and Messenger, not just out of training. That is a real cost, and it is the key difference between this bot and a training-only switch like Applebot-Extended.

The honest version: this is a values-and-reach tradeoff, not a pure technical one.

Block it if you have a firm policy against your content training AI models and you accept losing Meta AI product visibility to enforce that. Allow it if you want to be findable inside Meta's apps, which reach billions of people, and you are comfortable with the training that comes bundled in.

Picture two brands making the call. A stock-photo company blocks, because its images are its product and it does not want them training a model that could imitate them.

A project-management SaaS allows, because it wants Meta AI to name it when someone in a WhatsApp chat asks for the best tool for their team. Same crawler, opposite decision, and both are right for their own reasons. The mistake is not choosing; it is copying someone else's choice without checking whether your reasons match.

How do you block Meta-ExternalAgent without breaking anything?

You block Meta-ExternalAgent by disallowing the meta-externalagent token in robots.txt, while leaving facebookexternalhit allowed so link previews keep working. Give the agent its own user-agent block with a disallow, and do not touch the preview bot. Changes can take a while to propagate, so do not expect instant results.

A few details trip people up. Spell the token exactly as meta-externalagent, since a typo silently does nothing.

Give each bot its own user-agent block rather than stacking names, and remember that Meta-ExternalFetcher is a separate token, so blocking the agent does not stop the live fetcher. If you want to limit that one too, add a rule for meta-externalfetcher, knowing it may still act on direct user requests.

For the wider picture of controlling every AI crawler in one place, our guide on llms.txt covers the emerging standard, and the GPTBot guide walks the same decision for OpenAI.

Who should block Meta-ExternalAgent?

Block it if protecting your content from AI training outweighs being found inside Meta AI. That is a real, defensible position for some publishers and rights-holders, but it is a minority of cases, because the block also costs product visibility. Most brands are better served allowing it.

A few clear profiles help you place yourself.

• News publishers and stock-content owners with a firm no-AI-training stance often block, accepting the reach they give up.
• Brands and SaaS companies that want to be recommended inside Meta AI almost always allow it, since blocking hides them from billions of app users.
• Regulated or privacy-sensitive organizations sometimes block on principle, then verify the block holds with a firewall rule.
• Most small and mid-size businesses should allow it and spend their energy on being worth citing, not on blocking a crawler.
• Ecommerce and local brands almost never benefit from blocking, since product and location answers inside Meta AI are exactly where they want to appear.

If you are unsure, default to allowing it. Being invisible inside Meta AI is a quiet, permanent cost, while you can always add a block later if your policy changes.

Reversibility is the quiet argument for allowing. Blocking today and unblocking next month does not instantly restore your presence, since Meta has to recrawl and reindex you, and that takes time you cannot get back.

How is Meta-ExternalAgent different from GPTBot and Applebot-Extended?

The difference is what each block costs you. GPTBot is OpenAI's crawler and Applebot-Extended is Apple's training switch; blocking either mostly affects AI training. Meta-ExternalAgent bundles training and product indexing, so blocking it also cuts your visibility inside Meta AI. Same action, a bigger consequence.

That makes the Meta decision less obvious than the others.

With Applebot-Extended, a privacy-first brand can block and lose almost nothing. With Meta-ExternalAgent, the same brand trades away real reach across the most-used social apps in the world. Neither answer is wrong, but the stakes are higher, so decide on purpose rather than copying a rule from another bot.

There is a second difference worth knowing: enforcement. GPTBot and Applebot-Extended have a fairly clean reputation for honoring blocks.

Meta's picture is messier, because the fetcher may bypass robots.txt on user requests and some crawls have been reported after a block. So even the act of blocking is less reliable here, which is another reason to treat the Meta decision as its own thing rather than reusing a one-size rule across every AI crawler.

The full allow-or-block logic for the other major crawlers is in our guides on Amazonbot, Applebot-Extended, and ClaudeBot.

Frequently asked questions

Decide on the tradeoff, then write one clean rule

Do this next: decide whether Meta AI visibility across Facebook, Instagram, and WhatsApp is worth allowing the training that comes with it. If yes, leave Meta-ExternalAgent allowed. If no, disallow the meta-externalagent token and keep facebookexternalhit open.

Whichever way you go, keep facebookexternalhit allowed so your shared links stay clean, and spell every token exactly. A single typo in a user-agent line silently does nothing, which is the most common way these rules fail.

Then do not guess whether it is working. Run your buyer prompts through Meta AI and the other assistants with MentionsAPI, and watch whether blocking or allowing actually changed whether you get named. A crawler rule you never measure is just a hope in a text file.