Quick answer: ClaudeBot is the crawler Anthropic uses to collect public web content that may be used to train its Claude models. It respects robots.txt. Blocking it keeps future content out of training, but Claude's live search and user fetches use two other agents, Claude-SearchBot and Claude-User, which you control separately.
- Operator
- Anthropic
- Type
- Training crawler
- robots.txt token
ClaudeBot- Follows robots.txt
- Yes
- Used for
- Collecting public web content that may be used to train Anthropic's Claude models
- User agent
Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; ClaudeBot/1.0; [email protected])- How to verify
- Match the user agent and check that the source network is not a residential or unrelated hosting range
What is ClaudeBot?
Anthropic splits its web activity into three agents, and ClaudeBot is the one tied to training. It crawls public pages, and Anthropic states it tries not to access password protected or login gated content.
The other two agents cover what users experience in Claude. Claude-SearchBot crawls to improve search results, and Claude-User opens pages when a person asks Claude a question that needs the web. That separation means you can decide on training and on search visibility independently.
What happens if you block ClaudeBot?
A Disallow rule for ClaudeBot tells Anthropic to exclude your site from future training crawls. It does not remove content already collected, and it does not block Claude-SearchBot or Claude-User, so Claude can still find and read your pages in answers unless you block those as well.
Should you allow ClaudeBot?
The same logic as GPTBot applies. If you want Claude to know your brand and describe your product accurately by default, allow ClaudeBot. If your content is your product, for example paid research or licensed data, block ClaudeBot and keep Claude-SearchBot and Claude-User allowed so you still appear when users search.
Check whether ClaudeBot can reach your site
Enter your domain or a specific page. The checker reads your live robots.txt, picks the group ClaudeBot would use, and applies the longest matching rule, the same way major crawlers resolve conflicts.
robots.txt rules for ClaudeBot
Put each group in the robots.txt file at the root of your domain. User agent names are matched without regard to case, and the most specific matching group wins.
User-agent: ClaudeBot
Allow: /User-agent: ClaudeBot
Disallow: /User-agent: ClaudeBot
Disallow: /account/
Disallow: /checkout/
Allow: /The rule most people get wrong: once ClaudeBot has its own group, it ignores everything under User-agent: *. If your wildcard group disallows paths such as /admin/, repeat those lines inside the ClaudeBot group too.
Common Anthropic setup: appear in Claude answers, opt out of training
User-agent: Claude-SearchBot
Allow: /
User-agent: Claude-User
Allow: /
User-agent: ClaudeBot
Disallow: /How to verify real ClaudeBot traffic
Start with the user agent, then confirm the behaviour matches a well-behaved crawler: it fetches robots.txt, respects your rules, and does not hammer the server. Requests that claim to be ClaudeBot but ignore your Disallow rules, or come from residential connections, are almost certainly spoofed and can be blocked at the firewall.
To see how often it visits, count requests by user agent in your access log. On a typical Nginx server:
grep -i "ClaudeBot" /var/log/nginx/access.log | awk '{print $4}' | cut -d: -f1 | sort | uniq -cClaudeBot compared with other Anthropic agents
| Agent | Operator | Type | Follows robots.txt |
|---|---|---|---|
| ClaudeBot | Anthropic | Training crawler | Yes |
| Claude-User | Anthropic | User-triggered fetcher | Yes, per Anthropic's documentation |
| Claude-SearchBot | Anthropic | Search crawler | Yes, per Anthropic's documentation |
See all 20 agents side by side in the AI crawler directory.
ClaudeBot FAQ
See every AI crawler signal at once
The free AI SEO audit checks robots.txt access for 12 AI crawlers, plus llms.txt, schema, content and metadata, in a few seconds.
Run the free AI SEO audit