Amazonbot: how Amazon crawls for Alexa and its AI

Operator

Amazon

Type

Search and AI crawler

robots.txt token

Amazonbot

Follows robots.txt

Yes

Used for

Improving Amazon products and services, including answers from Alexa

User agent

Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; Amazonbot/0.1; +https://developer.amazon.com/support/amazonbot) Chrome/ Safari/537.36

How to verify

Reverse DNS must end in crawl.amazonbot.amazon, confirmed by a forward lookup

What is Amazonbot?

Amazonbot is frequently among the highest-volume automated agents on content sites. Its stated purpose is to improve Amazon's services, with Alexa answers the most visible example, and as Amazon's assistants become more capable the same data becomes more relevant to AI answers.

Unlike training-only crawlers, Amazonbot sits closer to a search crawler in spirit: it collects content so Amazon's assistants can use accurate, current information.

What happens if you block Amazonbot?

Disallowing Amazonbot stops Amazon from crawling your site for these uses. That can reduce the chance Alexa or other Amazon assistants use your content in answers. It has no effect on Google, Bing, or other AI products.

Should you allow Amazonbot?

Allow it unless load is a problem. If it is, prefer narrowing what it can crawl, for example disallowing faceted navigation or search result URLs, over blocking the whole site.

Check whether Amazonbot can reach your site

Enter your domain or a specific page. The checker reads your live robots.txt, picks the group Amazonbot would use, and applies the longest matching rule, the same way major crawlers resolve conflicts.

robots.txt rules for Amazonbot

Put each group in the robots.txt file at the root of your domain. User agent names are matched without regard to case, and the most specific matching group wins.

User-agent: Amazonbot
Allow: /

User-agent: Amazonbot
Disallow: /

User-agent: Amazonbot
Disallow: /account/
Disallow: /checkout/
Allow: /

The rule most people get wrong: once Amazonbot has its own group, it ignores everything under User-agent: *. If your wildcard group disallows paths such as /admin/, repeat those lines inside the Amazonbot group too.

How to verify real Amazonbot traffic

Amazon documents that real Amazonbot requests come from IPs whose reverse DNS ends in crawl.amazonbot.amazon, and that a forward lookup of that hostname returns the same IP. Requests that fail this check are not Amazon.

To see how often it visits, count requests by user agent in your access log. On a typical Nginx server:

grep -i "Amazonbot" /var/log/nginx/access.log | awk '{print $4}' | cut -d: -f1 | sort | uniq -c

Amazonbot compared with similar agents

See all 20 agents side by side in the AI crawler directory.

Amazonbot FAQ

Abd Shanti

Co-founder and strategy lead at Outline Technologies, the team behind FreeGPTSEO and AI Citation Monitor. Abd works on how brands get found and cited by search engines and AI assistants.