Free Tool
AI Robots.txt Generator
Build a robots.txt that explicitly welcomes AI crawlers. Select the bots you want to allow, add your sitemap URL, and copy the output.
Why Your robots.txt Matters for AI
If your robots.txt blocks AI crawlers (many default configs do), ChatGPT and Perplexity literally cannot access your site. No access means zero citations.
26%
of the top 1000 websites block GPTBot in their robots.txt (source: originality.ai 2025)
40+
known AI crawler user-agents that may visit your site looking for content to cite
0 sec
is how long it takes to fix. Update your robots.txt and AI crawlers can find you immediately.
robots.txt Questions
No. AI crawlers are read-only. They visit your pages the same way Google does. Allowing them means your content can be cited by ChatGPT, Perplexity, and other AI models. Blocking them means you are invisible to AI search.
Allow tells a crawler it can access your site. Disallow tells it to stay away. By default, if there is no robots.txt, crawlers assume they can access everything. But explicitly allowing AI bots makes your intent clear.
For most websites, yes. If your content is publicly available, there is no reason to block AI crawlers. The more bots that can access your content, the more AI models can cite you. Only block bots if you have a specific reason (e.g., private content, paid content).