AeoproAeopro

Why ChatGPT Doesn't Cite Your Website: The AEO Audit That Almost Nobody Does Right

Team at AeoproWritten with AI assistance18 min read

In summary

  • There are three families of AI bots, not one: those for training (GPTBot, ClaudeBot, Google-Extended), those for search and citation (OAI-SearchBot, Claude-SearchBot, PerplexityBot, Bingbot), and those triggered by the user (ChatGPT-User, Claude-User, Perplexity-User). Blocking the first is a legitimate decision; blocking the second is disappearing from the responses.
  • Blocking GPTBot does not block OAI-SearchBot, and blocking Google-Extended does not remove you from AI Overviews: Google documents that this token does not affect inclusion in Google Search (Google Search Central).
  • If your content only appears after executing JavaScript, there are bots that will see an almost empty page. The diagnosis is made by comparing raw HTML against rendered DOM.
  • llms.txt is not the solution: Google says you do not need to create text files for AI or special schema to appear in AI Overviews or AI Mode (Google, AI features), and John Mueller stated in January 2026 that Google does not endorse llms.txt (Search Engine Roundtable, Jan 20, 2026).
  • The traffic returned by AI engines is little and expensive to obtain: Cloudflare estimated in July 2025 that with OpenAI it is "750 times harder" to get a visit than with the old Google, and with Anthropic "30,000 times" (Cloudflare, Jul 1, 2025).

If you arrived here wondering why ChatGPT does not cite your website, the answer is rarely "my content is not good". In most audited cases, the problem is technical and boring: a line in robots.txt, a WAF that returns 403 to any strange user-agent, or a website made as a single-page application whose text does not exist until the browser executes JavaScript. This article explains how to check each thing manually and with what sources, and in the end, you will see how Aeopro, the AEO auditing tool from AIsToHire, automates the same checks on a URL.

Illustration of a web page in front of three artificial intelligence crawlers: one passes through an open door and another is blocked by a barrier
Illustration of a web page in front of three artificial intelligence crawlers: one passes through an open door and another is blocked by a barrier

What is AEO and how does it differ from SEO?

AEO (Answer Engine Optimization) is the work of preparing a website so that response engines —ChatGPT, Claude, Perplexity, Gemini, Copilot— can read, understand, and cite it when generating a response. Classic SEO optimizes for a list of ten blue links; AEO optimizes to be one of the three or four sources that the model summarizes and links. They share a lot (indexability, performance, useful content), but they have their own shortcomings that traditional SEO has never measured.

The practical difference: a flawless website for Google can be invisible to an AI engine if it blocks the citing bot, if it responds 403 to that bot, or if its text is only rendered on the client. And vice versa: Google claims that there are no additional requirements to appear in AI Overviews or AI Mode beyond good SEO practices (Google Search Central, AI features). That is: in the Google ecosystem, AEO is primarily about not breaking anything; in the OpenAI/Anthropic/Perplexity ecosystem, there are specific directives that you do need to manage.

Classic SEOAEO / response engines
GoalPosition in the SERPBe a cited source in the response
Important botGooglebot, BingbotOAI-SearchBot, Claude-SearchBot, PerplexityBot, Bingbot, Googlebot
Content unitThe pageThe self-contained passage
Critical technical signalIndexabilityAccessibility to the bot + content in HTML without JS
MetricPosition, clicks, CTRPercentage of appearance in responses, citations
Snippet control—nosnippet, max-snippet, data-nosnippet

AI bots are divided into three families and confusing them is costly

There are three types of crawlers and each is controlled with its own token in robots.txt: training, search/citation, and user-triggered fetch. Blocking the training one is a perfectly reasonable business decision —you don’t want to feed models for free—; blocking the search one means that the assistant cannot read to you when someone asks about what you sell. Almost all the "anti-AI" robots.txt files copied from forums block both things at once unintentionally.

This is the table to keep in mind, with the official documentation from each provider:

User-agentProviderPurposeDoes blocking it remove citations?
GPTBotOpenAIModel trainingNo
OAI-SearchBotOpenAIIndexing for ChatGPT searchYes
ChatGPT-UserOpenAIUser-triggered visitYes (OpenAI warns that robots.txt rules may not apply)
OAI-AdsBotOpenAIVerification of ad landings in ChatGPTNo
ClaudeBotAnthropicTrainingNo
Claude-SearchBotAnthropicIndexing for search resultsYes
Claude-UserAnthropicVisit when a user asks ClaudeYes
PerplexityBotPerplexityIndexing and linking in resultsYes
Perplexity-UserPerplexityUser-triggered visit (ignores robots.txt, according to their doc)Yes
Google-ExtendedGoogleTraining and grounding of GeminiDoes not affect Google Search
BingbotMicrosoftBing index (and products that use it)Yes

Sources: OpenAI, bot documentation; Anthropic, web crawling; Perplexity, bots; Google, common crawlers. Consulted on October 26, 2026.

Access control scheme: some AI crawlers pass through open doors while others are stopped by closed barriers
Access control scheme: some AI crawlers pass through open doors while others are stopped by closed barriers

Three nuances that almost no one applies correctly:

  1. Tokens are independent. Disallow for GPTBot does not affect OAI-SearchBot, nor does ClaudeBot affect Claude-SearchBot. Each agent needs its own group.
  2. User fetchers play by different rules. OpenAI documents that, because they are initiated by a person, robots.txt rules may not apply to ChatGPT-User; Perplexity directly states that Perplexity-User often ignores them. Relying solely on robots.txt to "close the door" does not work uniformly.
  3. Google-Extended is not the switch for AI Overviews. Google documents that this token manages the training and grounding of Gemini, and that "it does not affect site inclusion in Google Search nor is it used as a ranking signal". To limit what Google shows in its AI features, the controls are nosnippet, data-nosnippet, max-snippet, and noindex.

If your text only exists after executing JavaScript, for an AI your page may appear empty

The second most common cause of invisibility is client-side rendering. A bot that does not execute JavaScript sees the HTML returned by the server; if your website is a SPA (React, Vue, Angular) without server-side rendering, that HTML may be an empty <div id="root"> and three script tags. Googlebot renders; not all AI crawlers behave the same or document that they do.

The test is done in two steps and does not require paid tools:

# 1) What a bot without JavaScript sees: count the words in the raw HTML
curl -sL -A "OAI-SearchBot/1.4" https://tudominio.com/una-pagina \
  | sed -e 's/<[^>]*>/ /g' | tr -s ' \n' ' ' | wc -w

# 2) Does it return 403 or 200 when you change the user-agent?
curl -s -o /dev/null -w "%{http_code}\n" -A "PerplexityBot" https://tudominio.com/
curl -s -o /dev/null -w "%{http_code}\n" -A "Mozilla/5.0" https://tudominio.com/

If the first number is two digits and the actual page has a thousand words, there’s your problem. If the second command returns 403 with the bot user-agent and 200 with a browser one, your CDN or WAF is making the block you never configured: it’s the most common silent failure, as it does not appear in any SEO tool.

Comparison between the raw HTML, almost empty, and the rendered DOM full of content from the same page
Comparison between the raw HTML, almost empty, and the rendered DOM full of content from the same page

What to do if the raw HTML is empty, in order of cost:

  • Server or static rendering for the pages you want to be cited (Next.js, Nuxt, Astro, or prerender at the edge). It’s the real solution.
  • Selective prerendering of critical content: the first response block, the H1, contact data, and JSON-LD in the initial HTML, even if the rest hydrates later.
  • Never serve bots a different version of the content that the user sees: that is cloaking and is a violation of Google’s spam guidelines.

If your page only shows content after accepting a consent banner, what the bot sees is the banner. Crawlers do not click "Accept": they capture the initial state. The same goes for registration walls, geolocation interstitials, and aggressive anti-bot protections.

This does not mean removing consent —you have GDPR obligations to fulfill—, but that indexable content must be in the HTML behind the banner, not conditional on its acceptance. A banner that blocks the <body> with an overlay is compatible with AEO; a banner that prevents the server from sending the content is not.

llms.txt: what Google says and what you can really expect

llms.txt is a community proposal for a file at the root of the domain with a Markdown index of the site for model consumption. It is not a standard (it does not come from IETF or W3C) and Google has explicitly stated that it does not use it: its AI features guide states that "you do not need to create new machine-readable files, AI text files, or markup" to appear in AI Overviews or AI Mode, and John Mueller denied in January 2026 that the presence of the file in some Google domains was an endorsement (Search Engine Roundtable, Jan 20, 2026).

Practical conclusion: putting it up takes ten minutes and doesn’t break anything, but it is not a visibility lever and certainly does not replace the above. If you decide to have it, make sure it is correct (a real index of your important URLs, maintained), and do not count it as AEO work done.

There is also no "magic schema" to appear in AI responses

Google is explicit: there are no special structured data that need to be added to appear in AI Overviews or AI Mode (Google, AI features). JSON-LD is still worth it for another reason: it disambiguates. An Organization with sameAs, an Article with author and datePublished, a FAQPage with the questions exactly as people write them, a Product with offers and aggregateRating tell any system what entity you are and what you claim, without having to deduce it from the markup.

So treat the markup as data hygiene, not as a visibility trick:

Page TypeJSON-LD that adds valueFields that make the difference
Corporate homeOrganizationname, url, logo, sameAs, contactPoint
Blog articleArticle / BlogPostingheadline, author, datePublished, dateModified
Product sheetProductoffers.price, availability, brand, gtin
Local service pageLocalBusinessaddress, openingHours, areaServed
Frequently asked questionsFAQPageliteral question + complete answer

How to write citable content: answer first and self-contained passages

A response engine does not cite pages: it cites passages. The pattern that works is to put the complete answer in the first two or three sentences of each section, so that fragment is understood in isolation, without depending on the previous paragraph or a "as we said before". Then, below, the development, data, and nuances.

Specific rules, not generic ones:

  • One header = one question or a verifiable statement. "Prices" cannot be cited; "How much does an AEO audit cost in Spain" can.
  • Self-containment. Each section must answer its own question without pronouns pointing outside. If copying the first paragraph and reading it alone does not make sense, rewrite it.
  • Data with source and date. Models and human reviewers discard orphan figures. "68%" without who measures it and when is worth nothing.
  • Tables for everything comparable. They are the format that best survives extraction.
  • Explicit entities. Name your brand, your city, your product, and your competitors with the full name at least once; do not assume it is deduced from context.
  • Visible dates. Real dateModified and a date in the text. Freshness matters.

How much traffic can you expect? Little, but different

The available figures point to the same thing: response engines reduce clicks and return much less traffic per crawled page than a classic search engine. Pew Research Center measured the actual behavior of 900 American adults on 68,879 Google searches (data from March 2025): when an AI summary appeared, users clicked on a result in 8% of visits, compared to 15% when it did not appear; links within the summary were clicked in 1% of visits (Pew Research Center, Jul 22, 2025).

DataFigureSource and date
Click on a result with AI summary8% of visitsPew Research Center, Jul 22, 2025 (data from March 2025)
Click on a result without AI summary15% of visitsPew Research Center, Jul 22, 2025
Click on links within the summary1% of visitsPew Research Center, Jul 22, 2025
Session ended after the results page26% with summary vs. 16% without itPew Research Center, Jul 22, 2025
Difficulty in getting a visit vs. classic GoogleƗ750 with OpenAI, Ɨ30,000 with AnthropicCloudflare, Jul 1, 2025
Expected drop in traditional search volume for 2026āˆ’25%Gartner, press release Feb 19, 2024

Two honest readings of that table. The first: if your business plan depends on a stream of clicks from ChatGPT, the numbers do not support it today. The second: the citation itself has brand and preference value — appearing as the recommended source when someone asks "which agency is good for me in Valencia" matters even if there is no click — and the cost of not being there is asymmetrical, because your competitors are. Gartner's prediction, by the way, was published in February 2024 and should be treated as what it is: a forecast, not a measured fact.

How to Audit Your Website for AI Engines in 7 Steps

This is the complete sequence, in the order that matters. Steps 1 to 3 are eliminatory: if they fail, the rest is irrelevant.

  1. robots.txt, agent by agent. Open https://yourdomain.com/robots.txt and evaluate the specific URL against each token in the previous table, respecting precedence (the specific group wins over *; between Allow and Disallow, the longest pattern rule wins).
  2. Actual response with bot user-agent. Check the status code with curl -A. Look for 403, 429, unusual redirects, and JavaScript challenges from your WAF.
  3. Trimming directives. Review X-Robots-Tag in headers and <meta name="robots">: a noindex, nosnippet or max-snippet:0 inherited from a migration self-limits you without anyone noticing.
  4. Raw vs rendered. Count words from the HTML without JavaScript and compare them with the rendered page. Note which blocks are missing.
  5. Extractable structure. A single H1, header hierarchy without jumps, each section answering its question in the first paragraph, tables instead of comparative paragraphs.
  6. Identity and structured data. Valid JSON-LD, real dates, identifiable author, Organization with sameAs.
  7. Measurement and repetition. Separate AI traffic in analytics (the referrers from assistants are identifiable) and repeat the audit after each deployment to see the before and after.
Process in three steps: paste the URL, read the report with the score, and paste the code patch on the website
Process in three steps: paste the URL, read the report with the score, and paste the code patch on the website

What CANNOT be Known for Sure (and distrust anyone who promises you otherwise)

No one can guarantee you a citation in ChatGPT, nor measure "your position" in a response engine like you measure a position in Google. Generative responses are not deterministic: the same question, asked twice, may cite different sources, and providers do not publish a consultable index or a ranking API. Any "ranked 3 in ChatGPT" is, at best, a sampling.

What can be measured rigorously:

  • Accessibility: whether a specific bot can or cannot download a specific URL. This is binary and verifiable.
  • Extractability: how much of your content exists without JavaScript and how it is structured.
  • Sampled appearance: ask the same question N times and report what percentage of responses your brand appears in, always with the date and number of repetitions alongside. It is an indicator, not a ranking.

If a tool or an agency promises you "to be in ChatGPT" with a guaranteed number, the problem is not the tool: it is the promise.

Aeopro: the same checks, on a URL and in a report

Doing the 7 steps manually for a page takes a couple of hours; for an entire site, days. Aeopro, the AEO auditing app from AIsToHire, automates exactly that sequence: you paste a URL and it returns 99 deterministic checks with two scores —AEO, which weighs 70% of the overall result, and classic SEO, 30%— and the weights of each category published within the app so you can track where each point comes from.

What fits with what you just read:

What to checkWhat Aeopro does
robots.txt agent by agentCrosses your robots.txt with 11 AI bots, separating training bots from response bots, and shows the exact line causing each block
Actual response to a botSurveys your website with bot user-agent to catch 403s, WAF walls, and noindex/nosnippet directives
Raw vs renderedDownloads the page twice (raw HTML and rendered DOM), tells you what percentage of content is available without hydration, and lists the missing blocks
Fixing what fails44 of the 99 rules bring a generable patch: the complete JSON-LD, the TL;DR and the FAQ drafted for your content, the section rewritten in answer-first format, the llms.txt and metas. The robots.txt patch is composed without AI, with a switch for each bot
Appearance in responsesVisibility test: three editable questions from your sector, launched three times each, with the percentage of your brand's appearance, tone, and which competitors appear in your place — always presented as sampling with its date, never as an absolute yes or no
Repeat after deploymentRe-analysis with the before and after, and a versioned log of the rules catalog so the score is explainable months later

The deterministic report and re-analysis do not consume tokens: you can audit and re-audit. They only consume actions where the AI writes, and always one by one, when you press them. You can see it on the Aeopro page, the AEO auditing and AI visibility tool from AIsToHire.

Frequently Asked Questions

Why doesn't ChatGPT cite my website?

In order of frequency: because your robots.txt blocks OAI-SearchBot (often unintentionally, by blocking "the AI" as a whole), because your server or WAF returns 403 to bot user-agents, because your content only exists after executing JavaScript, because there is no passage that answers the question in a self-contained manner, or simply because other sources are more cited for that topic. The first three causes are technical and can be checked in minutes.

If I block GPTBot, will I stop appearing in ChatGPT?

Not necessarily. GPTBot is the training crawler; the one that indexes for ChatGPT search is OAI-SearchBot. They are independent tokens in robots.txt, so you can refuse to feed training and still be citable. What does remove you from the responses is blocking OAI-SearchBot (OpenAI documentation).

Do I need a llms.txt file?

It's not essential, and as of today, there is no public evidence that major providers use it in production. Google claims that text files for AI or special markup are not needed to appear in their AI features, and in January 2026, John Mueller denied that Google supports it. Including it does not harm; believing that with that you have already done AEO does.

How do I know if my website is viewable without JavaScript?

Download the raw HTML with curl -sL https://yourdomain.com | sed -e 's/<[^>]*>/ /g' | wc -w and compare the word count with what you see in the browser. If the difference is an order of magnitude, your content depends on client-side rendering. In Aeopro, that comparison is done, with the two texts side by side and the list of missing blocks.

Does AEO replace SEO?

No. SEO remains the foundation: if you are not indexed and are neither fast nor useful, there is no possible AEO. Google states it in their words: good SEO practices still apply to their AI features, and there are no additional requirements. AEO adds a layer: accessibility to bots that are not Googlebot, extractable content without JavaScript, and self-contained passages.

How much traffic will I gain if I am cited in ChatGPT?

Less than you would like, and it's advisable to accept that from the beginning. Pew measured that links within a Google AI summary are clicked on 1% of visits, and Cloudflare estimated in July 2025 that getting a visit via OpenAI is about 750 times harder than with the classic search engine. The value of the citation lies more in branding and preference than in direct clicks.

Can it be guaranteed to appear in AI responses?

No, and no honest tool will guarantee it. Generative responses vary between executions, and providers do not expose a consultable ranking. What can be guaranteed is to eliminate the technical blocks that currently exclude you and measure appearance by sampling, with date and number of repetitions.

How often should I repeat the audit?

After each deployment that affects templates, robots.txt, CDN, or WAF, and as a thorough review every one or two months: the names of bots and directives change several times a year. In 2025 and 2026, new tokens have been added —OAI-SearchBot, Claude-SearchBot, OAI-AdsBot— that did not exist when many people wrote their robots.txt.

Conclusion: first remove the blocks, then write to be cited

AEO today is 20% writing and 80% not obstructing. Before rewriting a single paragraph, ensure that the citing bots can download your page, that your server does not respond with 403, and that your content exists without JavaScript; these three things explain most of the invisibilities seen in an audit. Then come the self-contained passages, tables, clean JSON-LD, and real dates.

If you want the complete picture of a URL without dedicating the afternoon to it, Aeopro —the AEO audit tool from AIsToHire— performs the 99 checks, shows you the exact fragment of your HTML that causes each problem, and for 44 of them, generates the patch ready to paste: audit a website with Aeopro.

Paste your website's URL into Aeopro and check in under a minute if the bots that cite sources can read you, what is preventing them, and what the exact fix is.

Paste your website's URL into Aeopro and check in under a minute if the bots that cite sources can read you, what is preventing them, and what the exact fix is.

↑ Back to top