Quick answer: Claude-User is the agent Anthropic uses when a person asks Claude a question and Claude needs to open a web page to answer it. It does not crawl on its own and it is not a training crawler. Anthropic says disallowing it in robots.txt prevents Claude from retrieving your content for those user requests.
- Operator
- Anthropic
- Type
- User-triggered fetcher
- robots.txt token
Claude-User- Follows robots.txt
- Yes, per Anthropic's documentation
- Used for
- Fetching a page when a Claude user asks a question that needs it
- User agent
User agent contains Claude-User- How to verify
- Match the user agent; traffic follows real conversations, so it arrives in irregular bursts
What is Claude-User?
Claude-User is tied directly to user demand. Every request represents a conversation where Claude decided your page was worth reading. That makes it one of the more valuable agents to watch in your logs, because it shows which pages people are actually asking about.
It is independent of ClaudeBot, the training crawler, and of Claude-SearchBot, which improves search quality. Allowing Claude-User while blocking ClaudeBot is a common and sensible setup.
What happens if you block Claude-User?
If you disallow Claude-User, Claude cannot pull your page into an answer when a user asks, and it will rely on other sources. Anthropic notes that this can reduce your visibility in user-directed web search. Your content is not removed from anything, it is simply not fetched live.
Should you allow Claude-User?
Allow it unless the content must never be quoted. These visits happen because a real person wanted information your page has, which is exactly the audience you want to reach through AI assistants.
Check whether Claude-User can reach your site
Enter your domain or a specific page. The checker reads your live robots.txt, picks the group Claude-User would use, and applies the longest matching rule, the same way major crawlers resolve conflicts.
robots.txt rules for Claude-User
Put each group in the robots.txt file at the root of your domain. User agent names are matched without regard to case, and the most specific matching group wins.
User-agent: Claude-User
Allow: /User-agent: Claude-User
Disallow: /User-agent: Claude-User
Disallow: /account/
Disallow: /checkout/
Allow: /The rule most people get wrong: once Claude-User has its own group, it ignores everything under User-agent: *. If your wildcard group disallows paths such as /admin/, repeat those lines inside the Claude-User group too.
Common Anthropic setup: appear in Claude answers, opt out of training
User-agent: Claude-SearchBot
Allow: /
User-agent: Claude-User
Allow: /
User-agent: ClaudeBot
Disallow: /How to verify real Claude-User traffic
Match the user agent and look at the pattern. User-triggered fetchers produce short bursts on specific pages, often your most practical ones. Sustained crawling across the whole site under this name is not normal and is worth investigating as spoofed traffic.
To see how often it visits, count requests by user agent in your access log. On a typical Nginx server:
grep -i "Claude-User" /var/log/nginx/access.log | awk '{print $4}' | cut -d: -f1 | sort | uniq -cClaude-User compared with other Anthropic agents
| Agent | Operator | Type | Follows robots.txt |
|---|---|---|---|
| ClaudeBot | Anthropic | Training crawler | Yes |
| Claude-User | Anthropic | User-triggered fetcher | Yes, per Anthropic's documentation |
| Claude-SearchBot | Anthropic | Search crawler | Yes, per Anthropic's documentation |
See all 20 agents side by side in the AI crawler directory.
Claude-User FAQ
See every AI crawler signal at once
The free AI SEO audit checks robots.txt access for 12 AI crawlers, plus llms.txt, schema, content and metadata, in a few seconds.
Run the free AI SEO audit