# Check robots.txt for AI bots: Are you blocking GPTBot or ChatGPT?

> Analyze your robots.txt against 11 AI bots (GPTBot, OAI-SearchBot, ClaudeBot, PerplexityBot...). See which bots you block, the exact line causing it, and the patch ready to copy.

- **Source (source):** https://aistohire.com/aeopro-aeo-audit-ai-visibility-198/f/robots-txt-audit-ai-bots
- **Platform (platform):** AIstoHire
- **Language (language):** en

## In summary

- Check if your robots.txt is blocking AI bots
- Paste a URL and Aeopro will cross-check your robots.txt with the 11 bots from ChatGPT, Claude, Perplexity, and Bing, showing you the exact line that closes the door to them.
- Separates blocking from being invisible
- Real precedence, not text search
- The exact line that blocks you

## Check if your robots.txt is blocking AI bots

Aeopro downloads your /robots.txt, parses it by User-agent groups, and evaluates, bot by bot, whether the URL you are auditing is allowed or blocked. It doesn't look for your bot name 'by eye' in the file: it applies the rules of truth: the specific group wins over the wildcard *, and within the group, the longest pattern rule prevails, with Allow breaking ties against Disallow. It supports wildcards * and $ in paths, so a Disallow: /*?* or an Allow: /blog/$ is interpreted as the crawler interprets it, not as it may seem at first glance.

The result is a matrix of 11 bots with their class alongside: OAI-SearchBot, ChatGPT-User, Claude-User, Claude-SearchBot, PerplexityBot, and Bingbot are response bots —those that search, cite, and link—; GPTBot, ClaudeBot, Google-Extended, Applebot-Extended, and CCBot are training bots. That distinction is the key to the entire report: if you block a response bot, the finding is classified as blocking and caps your AEO score because without access, no citation is possible. If you block a training bot, the finding is informative: not wanting your content to enter a dataset is a legitimate decision, and Aeopro does not dispute it.

Each blocked bot comes with proof: the red traffic light, the User-agent group that applies to it, and the exact line from the file that causes it, copied as is. You don't have to re-read your robots.txt looking for the culprit; it's right in front of you.

Aeopro also covers issues that are not a Disallow. If your /robots.txt returns an HTML page —typical of a styled 404 or a SPA router that captures everything— or responds with a 500 error, it detects this and raises it as a finding: bots treat it as 'unrestricted', but it's a sign of neglect. And if there is a Crawl-delay of more than 10 seconds, it warns you because at that pace, some crawlers may give up reading you entirely. If you simply do not have a robots.txt, the report tells you without penalizing you: the absence does not block anyone.

## What's included

- **Separates blocking from being invisible** — blocking a training bot is your decision and is marked as informative. Blocking a response bot means disappearing from the answers of ChatGPT or Perplexity, and that is a blocking finding that caps your AEO score.
- **Real precedence, not text search** — specific group over wildcard *, longer pattern over shorter, and Allow winning ties, with support for * and $ in paths. It’s the same logic that the crawler applies, so the verdict is what you will truly have.
- **The exact line that blocks you** — each bot in red comes with the User-agent group it applies to and the literal text of the offending line. No more reading 200 lines of robots.txt by hand to find the unnecessary Disallow.
- **The 11 bots that matter now, with their class** — the six response bots (OAI-SearchBot, ChatGPT-User, Claude-User, Claude-SearchBot, PerplexityBot, Bingbot) and the five training bots (GPTBot, ClaudeBot, Google-Extended, Applebot-Extended, CCBot), evaluated against the specific URL you are auditing, not just against the root of the domain.
- **Hunt down silent failures** — a /robots.txt serving HTML instead of plain text, one that responds with a 500 error, or an absurd Crawl-delay over 10 seconds. Failures that do not appear in any classic SEO tool and indicate a misconfigured server.
- **The fix already written, without using AI** — from the very finding, you generate the robots.txt patch with a switch for each bot. You choose whom to open up to, the final file regenerates live with your current robots.txt plus the new block, and you copy it. It’s deterministic: it doesn’t consume tokens.
- **Free to repeat** — the robots.txt audit is part of the deterministic checks in the report. You change the file, deploy it, re-analyze, and the delta tells you if the finding was resolved. As many times as you want.

## Frequently Asked Questions

### How do I know if my robots.txt is blocking ChatGPT?

Paste the URL into Aeopro: download your /robots.txt, parse it by User-agent groups, and evaluate that specific URL against the 11 AI bots. In the 'AI Bots Access' card, you will see each one with a green or red light. For ChatGPT, there are two distinct bots, and it's important not to confuse them: OAI-SearchBot and ChatGPT-User are the response bots that search and cite sources in a conversation; GPTBot is for training. If you want to be cited, you cannot block the first two.

### What is the difference between a response bot and a training bot?

A training bot (GPTBot, ClaudeBot, Google-Extended, Applebot-Extended, CCBot) collects content to train models. Blocking it is a legitimate decision, and Aeopro marks it as informational, without penalizing your score. A response bot (OAI-SearchBot, ChatGPT-User, Claude-User, Claude-SearchBot, PerplexityBot, Bingbot) reads your page when someone asks something, and it is the one that cites and links to you in the response. Blocking it means not existing for that AI: that’s why this finding is considered blocking and caps the AEO score of the report.

### Why does my robots.txt seem permissive, yet the tool says I block bots?

It's almost always a precedence issue. If you have a specific group for a bot, that group completely overrides the wildcard *: the permissive rules of * no longer apply. And within a group, the longest pattern wins, with Allow breaking ties against Disallow in case of equality. Add the wildcards * and $ and it's easy for a Disallow intended for search parameters to end up blocking half the website. Aeopro applies these rules just like the crawler and shows you the exact line that decides the verdict, so you can see where the block is coming from.

### What happens if I don't have a robots.txt, or if it returns an error?

Not having a robots.txt doesn't block anyone and Aeopro doesn't penalize you for it: it indicates it and moves on. What does raise a finding is if the file is broken: if your /robots.txt returns an HTML page instead of plain text —very common in single-page websites, where the router captures all routes— or if it responds with a 500 error. Bots interpret it as 'no restrictions', so it doesn't close the door on you, but it's a sign of a misconfigured server. It also alerts you to a Crawl-delay over 10 seconds, because at that rate some crawlers abandon before reading your entire site.

### Does Aeopro give me the corrected robots.txt or just the diagnosis?

It gives you the corrected version. From the finding, you open the robots.txt patch, which has a switch for each blocked bot: you decide who to allow and who to exclude, and the complete file is regenerated live by combining your current robots.txt with the new block of permissions, taking advantage of the fact that the specific group wins over the wildcard. You copy it and upload it. This patch is composed without AI: it is deterministic and does not consume tokens, just like the audit itself, so you can reanalyze later to confirm that the finding has been resolved.

- [Aeopro — AEO Audit: Make AI able to read and cite you](https://aistohire.com/aeopro-aeo-audit-ai-visibility-198)

## About the platform

**Aeopro — AEO Audit: Make AI able to read and cite you** — an app by AIstoHire. Available at: https://aistohire.com/aeopro-aeo-audit-ai-visibility-198/f/robots-txt-audit-ai-bots

### Other languages

- `es`: https://aistohire.com/es/aeopro-auditoria-aeo-visibilidad-en-ia-198/f/auditoria-robots-txt-bots-ia
- `fr`: https://aistohire.com/fr/aeopro-audit-aeo-visibilite-ia-198/f/audit-robots-txt-bots-ia
- `it`: https://aistohire.com/it/aeopro-audit-aeo-visibilita-ia-198/f/audit-robots-txt-bot-ia
- `pt`: https://aistohire.com/pt/aeopro-auditoria-aeo-visibilidade-ia-198/f/auditoria-robots-txt-bots-ia
- `de`: https://aistohire.com/de/aeopro-aeo-audit-ki-sichtbarkeit-198/f/robots-txt-audit-ki-bots
- `bg`: https://aistohire.com/bg/aeopro-proverete-dali-chatgpt-i-perplexity-mogat-da-tsitirat-sayta-vi-198/f/odit-na-robots-txt-za-ai-botove
- `ja`: https://aistohire.com/ja/aeo-kansa-ai-inyo-taisaku-198/f/robots-txt-ai-bot-kansa
- `nl`: https://aistohire.com/nl/aeopro-controleer-of-chatgpt-en-perplexity-je-site-kunnen-citeren-198/f/robots-txt-audit-ai-bots
- `ko`: https://aistohire.com/ko/aeopro-aeo-audit-ai-inyong-198/f/robots-txt-ai-bot-jeomgeom
- `sv`: https://aistohire.com/sv/aeopro-granska-om-chatgpt-och-perplexity-kan-citera-din-webbplats-198/f/granskning-robots-txt-ai-botar
- `da`: https://aistohire.com/da/aeopro-tjek-om-chatgpt-og-perplexity-kan-citere-dit-websted-198/f/audit-af-robots-txt-for-ai-bots
- `ar`: https://aistohire.com/ar/aeopro-dqq-hl-ysttia-chatgpt-wperplexity-alastshhad-bmuqak-198/f/tadqiq-robots-txt-botat-althaka2-alistinaa3i
- `no`: https://aistohire.com/no/aeopro-sjekk-om-chatgpt-og-perplexity-kan-sitere-nettstedet-ditt-198/f/revisjon-robots-txt-ki-roboter
- `fi`: https://aistohire.com/fi/aeopro-auditoi-voivatko-chatgpt-ja-perplexity-lainata-sivustoasi-198/f/robots-txt-tekoalybotit-auditointi
- `zh`: https://aistohire.com/zh/aeopro-aeo-geo-audit-198/f/robots-txt-ai-bots-audit
- `pl`: https://aistohire.com/pl/aeopro-sprawdz-czy-chatgpt-i-perplexity-zacytuja-twoja-strone-198/f/audyt-robots-txt-boty-ai
- `he`: https://aistohire.com/he/aeopro-bdko-am-chatgpt-v-perplexity-ykolim-ltstt-at-hatr-shlkm-198/f/bdikat-robots-txt-botim-bina-melachutit
- `cs`: https://aistohire.com/cs/aeopro-zjisti-zda-chatgpt-a-perplexity-mohou-citovat-tvuj-web-198/f/audit-robots-txt-ai-boti
- `id`: https://aistohire.com/id/aeopro-audit-apakah-chatgpt-dan-perplexity-bisa-kutip-situs-anda-198/f/audit-robots-txt-bot-ai
