Robots.txt for AI bots

Check if your robots.txt is blocking AI bots

Paste a URL and Aeopro will cross-check your robots.txt with the 11 bots from ChatGPT, Claude, Perplexity, and Bing, showing you the exact line that closes the door to them.

A misplaced Disallow can cost you the only thing that brings you visits from an AI today: the citation.

Aeopro downloads your /robots.txt, parses it by User-agent groups, and evaluates, bot by bot, whether the URL you are auditing is allowed or blocked. It doesn't look for your bot name 'by eye' in the file: it applies the rules of truth: the specific group wins over the wildcard *, and within the group, the longest pattern rule prevails, with Allow breaking ties against Disallow. It supports wildcards * and $ in paths, so a Disallow: /*?* or an Allow: /blog/$ is interpreted as the crawler interprets it, not as it may seem at first glance.

The result is a matrix of 11 bots with their class alongside: OAI-SearchBot, ChatGPT-User, Claude-User, Claude-SearchBot, PerplexityBot, and Bingbot are response bots —those that search, cite, and link—; GPTBot, ClaudeBot, Google-Extended, Applebot-Extended, and CCBot are training bots. That distinction is the key to the entire report: if you block a response bot, the finding is classified as blocking and caps your AEO score because without access, no citation is possible. If you block a training bot, the finding is informative: not wanting your content to enter a dataset is a legitimate decision, and Aeopro does not dispute it.

Each blocked bot comes with proof: the red traffic light, the User-agent group that applies to it, and the exact line from the file that causes it, copied as is. You don't have to re-read your robots.txt looking for the culprit; it's right in front of you.

Aeopro also covers issues that are not a Disallow. If your /robots.txt returns an HTML page —typical of a styled 404 or a SPA router that captures everything— or responds with a 500 error, it detects this and raises it as a finding: bots treat it as 'unrestricted', but it's a sign of neglect. And if there is a Crawl-delay of more than 10 seconds, it warns you because at that pace, some crawlers may give up reading you entirely. If you simply do not have a robots.txt, the report tells you without penalizing you: the absence does not block anyone.

Benefits of Robots.txt for AI bots

Separates blocking from being invisible

blocking a training bot is your decision and is marked as informative. Blocking a response bot means disappearing from the answers of ChatGPT or Perplexity, and that is a blocking finding that caps your AEO score.

Real precedence, not text search

specific group over wildcard *, longer pattern over shorter, and Allow winning ties, with support for * and $ in paths. It’s the same logic that the crawler applies, so the verdict is what you will truly have.

The exact line that blocks you

each bot in red comes with the User-agent group it applies to and the literal text of the offending line. No more reading 200 lines of robots.txt by hand to find the unnecessary Disallow.

The 11 bots that matter now, with their class

the six response bots (OAI-SearchBot, ChatGPT-User, Claude-User, Claude-SearchBot, PerplexityBot, Bingbot) and the five training bots (GPTBot, ClaudeBot, Google-Extended, Applebot-Extended, CCBot), evaluated against the specific URL you are auditing, not just against the root of the domain.

Hunt down silent failures

a /robots.txt serving HTML instead of plain text, one that responds with a 500 error, or an absurd Crawl-delay over 10 seconds. Failures that do not appear in any classic SEO tool and indicate a misconfigured server.

The fix already written, without using AI

from the very finding, you generate the robots.txt patch with a switch for each bot. You choose whom to open up to, the final file regenerates live with your current robots.txt plus the new block, and you copy it. It’s deterministic: it doesn’t consume tokens.

Free to repeat

the robots.txt audit is part of the deterministic checks in the report. You change the file, deploy it, re-analyze, and the delta tells you if the finding was resolved. As many times as you want.

Stephen Ramos

Find out in a minute which AI bots you are blocking

Paste a URL and receive the matrix of the 11 bots, the exact line blocking each one, and the patch ready to copy.

Check robots.txt for AI bots: Are you blocking GPTBot or ChatGPT?