Quick answer: Claude-SearchBot is the crawler Anthropic uses to navigate the web and improve the search results Claude shows its users. It respects robots.txt and it is separate from ClaudeBot, the training crawler. Anthropic says blocking it may reduce your visibility and accuracy in Claude's search results.
- Operator
- Anthropic
- Type
- Search crawler
- robots.txt token
Claude-SearchBot- Follows robots.txt
- Yes, per Anthropic's documentation
- Used for
- Crawling to improve the quality of search results shown to Claude users
- User agent
User agent contains Claude-SearchBot- How to verify
- Match the user agent and confirm it follows your robots.txt rules
What is Claude-SearchBot?
Claude can search the web while it answers. To return good results it needs an up to date picture of which pages are relevant, and Claude-SearchBot is the agent that builds that picture for Anthropic.
It sits between the other two Anthropic agents. ClaudeBot is about training, Claude-User is a live fetch for one user, and Claude-SearchBot is ongoing crawling for search quality.
What happens if you block Claude-SearchBot?
Disallowing Claude-SearchBot means Anthropic will not crawl your pages to improve search, so Claude is less likely to surface them, and less likely to describe them accurately, when users search. It does not remove your pages from any other engine.
Should you allow Claude-SearchBot?
Allow it if you want Claude users to find you. It is not a training crawler, so allowing it does not change your position on training, which you control with ClaudeBot.
Check whether Claude-SearchBot can reach your site
Enter your domain or a specific page. The checker reads your live robots.txt, picks the group Claude-SearchBot would use, and applies the longest matching rule, the same way major crawlers resolve conflicts.
robots.txt rules for Claude-SearchBot
Put each group in the robots.txt file at the root of your domain. User agent names are matched without regard to case, and the most specific matching group wins.
User-agent: Claude-SearchBot
Allow: /User-agent: Claude-SearchBot
Disallow: /User-agent: Claude-SearchBot
Disallow: /account/
Disallow: /checkout/
Allow: /The rule most people get wrong: once Claude-SearchBot has its own group, it ignores everything under User-agent: *. If your wildcard group disallows paths such as /admin/, repeat those lines inside the Claude-SearchBot group too.
Common Anthropic setup: appear in Claude answers, opt out of training
User-agent: Claude-SearchBot
Allow: /
User-agent: Claude-User
Allow: /
User-agent: ClaudeBot
Disallow: /How to verify real Claude-SearchBot traffic
Match the user agent and confirm the crawler requests robots.txt and respects it. A crawler using this name that ignores your rules is not behaving like Anthropic's agent and can be blocked at the network level.
To see how often it visits, count requests by user agent in your access log. On a typical Nginx server:
grep -i "Claude-SearchBot" /var/log/nginx/access.log | awk '{print $4}' | cut -d: -f1 | sort | uniq -cClaude-SearchBot compared with other Anthropic agents
| Agent | Operator | Type | Follows robots.txt |
|---|---|---|---|
| ClaudeBot | Anthropic | Training crawler | Yes |
| Claude-User | Anthropic | User-triggered fetcher | Yes, per Anthropic's documentation |
| Claude-SearchBot | Anthropic | Search crawler | Yes, per Anthropic's documentation |
See all 20 agents side by side in the AI crawler directory.
Claude-SearchBot FAQ
See every AI crawler signal at once
The free AI SEO audit checks robots.txt access for 12 AI crawlers, plus llms.txt, schema, content and metadata, in a few seconds.
Run the free AI SEO audit