Anthropic is the one AI company whose crawlers actually do what your robots.txt says.
That is the surprising part, and it changes how you should think about ClaudeBot. ClaudeBot is Anthropic's training crawler, and blocking it in robots.txt opts your content out of Claude model training and nothing else. It does not remove you from Claude, because Claude reads and cites pages through two other bots. So the real question is not block or allow. It is which of Anthropic's three crawlers to allow, and for what.
What is ClaudeBot?
ClaudeBot is Anthropic's web crawler for collecting public content to train and improve its Claude models. It fetches pages the way a search crawler does, then adds them to the data future Claude versions learn from. Per Anthropic's own documentation, ClaudeBot respects robots.txt, so you can allow or block it. It is one of three separate Anthropic crawlers, each with a different job.
That last part trips people up. When someone says "block the Claude bot," they usually mean ClaudeBot. But ClaudeBot is only the training bot.
What are Anthropic's three crawlers?
ClaudeBot trains models. Claude-User fetches a page when a person asks Claude to look something up. Claude-SearchBot indexes content for Claude's search results. Each has its own user-agent, so you can allow the ones that make you visible and block the one that trains, all in the same robots.txt file.
| ClaudeBot | Claude-User | Claude-SearchBot | |
|---|---|---|---|
| Job | Trains models | Fetches for a user | Indexes for search |
| Respects robots.txt | Yes | Yes | Yes |
| Block to | Opt out of training | Stop user fetches | Leave Claude search |
| Affects visibility | No | Yes | Yes |
Does ClaudeBot respect robots.txt?
Yes, and this is where Anthropic stands apart. It says all three of its bots, including the user-directed Claude-User, honor robots.txt. OpenAI and Perplexity draw a sharper line: ChatGPT-User and Perplexity-User generally do not apply robots.txt to user-initiated fetches. With Claude, your rules actually hold across the board, which makes control predictable.
With most AI crawlers, a user-triggered fetch slips past your robots.txt. With Claude, the rules hold across all three bots. That is rare, and worth knowing.The Claude difference
Should you allow or block ClaudeBot?
Block ClaudeBot if you would rather your content stay out of Anthropic model training. Allow it if you want to contribute to training. For most brands the better move is to block ClaudeBot but allow Claude-User and Claude-SearchBot, so you keep Claude visibility while opting out of the training set. The choice comes down to what you are trying to protect.
Claude matters more for B2B than its market share suggests, because of heavy enterprise adoption. Business buyers research vendors inside Claude, so being readable by Claude-User and Claude-SearchBot shapes what they hear about your category. We compare that enterprise angle in Copilot vs ChatGPT.
Does blocking ClaudeBot hurt your Claude visibility?
No. Blocking ClaudeBot opts you out of training only. Claude reads and cites your pages through Claude-User and Claude-SearchBot, which are separate bots you can still allow. You can disallow ClaudeBot and stay fully visible in Claude answers. The bots share a name but not a job.
How do you block, allow, and verify ClaudeBot?
Add a block at yourdomain.com/robots.txt that names each Claude user-agent. To keep Claude visibility while opting out of training, allow Claude-User and Claude-SearchBot and disallow ClaudeBot. Then verify bots by matching their IP against Anthropic's published list at claude.com/crawling/bots.json, rather than trusting the user-agent string, which scrapers fake.
# Stay citable in Claude
User-agent: Claude-User
Allow: /
User-agent: Claude-SearchBot
Allow: /
# Opt out of model training
User-agent: ClaudeBot
Disallow: /
Frequently asked questions
What is ClaudeBot?
What are Anthropic's three crawlers?
Does ClaudeBot respect robots.txt?
Does blocking ClaudeBot remove me from Claude?
How do I block or allow ClaudeBot in robots.txt?
How do I verify a request is really ClaudeBot?
Set the config, then measure the result
Do this today: allow Claude-User and Claude-SearchBot, block ClaudeBot if you want out of training, and verify bots by IP. Because Claude honors robots.txt across all three, the config does exactly what it says, which is a nice change.
Then check the part that matters: whether Claude actually cites you. Pull that baseline with MentionsAPI, keep your crawler access clean across engines, and watch whether allowing the visibility bots turns into real mentions in Claude.