Every page you publish now speaks to two readers at once, and it can tell them two different things.
Give one reader the raw HTML the server sends and give the other reader the page as a browser paints it after the JavaScript has run, and you can make those two versions disagree.
Put one value in the markup, have a script quietly swap it for another, and the page holds two truths simultaneously: one for the reader who only reads text, one for the reader who runs the code.
Which truth an AI assistant walks away with depends entirely on which kind of reader it turns out to be.

That is not a rhetorical flourish.
It is the most testable question in AI Search SEO right now, and for most of the last eighteen months we have been answering it with assumptions instead of evidence.
This piece is an attempt to replace the assumptions. I have spent the past weeks reading the current vendor documentation, the log-level crawler studies, the grounding reverse-engineering work, and, decisively, a controlled experiment that finally put the central claim to the test.
Much of what the industry repeats holds up well. One thing many of us have been saying, me included, turned out to be wrong.
The AI Bot Trinity: 3 Crawlers, 3 Jobs
Start with the settled ground, because there is more of it than the discourse suggests.
Every major vendor has converged on the same three-part crawling architecture, and the parts are worth naming precisely because the SEO consequence turns on the distinction:
There is a training crawler, whose job is to acquire content that may feed future model weights.
There is a search-index crawler, which builds and refreshes the corpus that grounds answers.
And there is a user-triggered fetcher, which visits a specific URL live, in-session, because a person asked a question that needed it.
Vendor | Training crawler | Search-index crawler | Live user-triggered fetcher | robots.txt on the live fetch |
|---|---|---|---|---|
Anthropic | ClaudeBot | Claude-SearchBot | Claude-User | Honored |
OpenAI | GPTBot | OAI-SearchBot | ChatGPT-User | "May not apply" (observed complying) |
Perplexity | — (no foundation models) | PerplexityBot | Perplexity-User | Generally ignored |
Google-Extended (control token) | GoogleBot | Googlebot for grounding; Google-Agent for agentic tasks | It is a token, not a fetcher |
Google is the structural outlier. It fields no distinct AI crawler; it runs Googlebot and exposes Google-Extended, a robots.txt control token rather than a user agent, which governs whether the content Googlebot already fetches may train and ground Gemini.
The consequence is the most important sentence in any AI-crawler audit: opting out of training is not opting out of citation. Blocking GPTBot does nothing to your visibility in ChatGPT's search surface, because that surface is fed by OAI-SearchBot and ChatGPT-User. Blocking ClaudeBot leaves Claude-SearchBot and Claude-User untouched.
Govern all these bots as a single undifferentiated category called "AI" and you will either train models you meant only to feed answers or vanish from answers you meant only to withhold from training. Govern by purpose, per token, or do not bother.

One file, two intentions: Disallow the training bots, Allow the search bots. This is what "opt out of training without opting out of citation" looks like in practice.
The JavaScript Execution Gap in Live AI Web Fetching
Now the finding most of the industry already half-knows: three of the four major assistants do not execute JavaScript on a live fetch.
The evidence base almost everyone cites is the Vercel and MERJ log analysis, and it is genuinely good work.
Across their network they measured GPTBot at 569 million requests and Anthropic's crawler at 370 million in a single month, together roughly a fifth of Googlebot's volume.
Their conclusion was blunt: none of the major AI crawlers rendered JavaScript. The bots would sometimes fetch script files — ChatGPT's crawlers leaning toward HTML, Anthropic's spending a striking share of requests on JavaScript files it never ran — but fetching a script as text is not executing it.
They also documented the operational immaturity of these crawlers, wasting roughly a third of their fetches on 404s where Googlebot wastes under a tenth.
One caveat the source many gloss over, and which I will not: that data is from late 2024. In a field where pipelines ship monthly, presenting a fifteen-month-old crawl snapshot as current behavior is exactly the kind of quiet staleness that turns a good finding into folklore.
Treat it as directional and structural — the shape of how these systems behave — not as a live readout of last week.
The mechanic beneath it is simple and brutal.
A client-side-rendered page ships an almost empty body: a root div and a script reference. A browser turns that into a full page. A non-rendering crawler sees precisely what is in the file — the empty div — and moves on. The parser captures nothing citable. You can rank first in Google, whose Web Rendering Service does execute the script, and be a blank page to an assistant that does not.
This is real, common, and for ChatGPT, Claude, and Perplexity not seriously in dispute.
Which left one comfortable exception standing. Google.
Access, retrieval, and visibility are different stages of the same pipeline. Once you know what AI search crawlers can reach, the next question becomes: do you make it through to the answer?
Advanced Web Ranking tracks that outcome across AI search, so you can see when your presence actually turns into impact.
Does Gemini Render JavaScript? Testing Google’s AI Search Pipeline
The received wisdom — and I held it too — was that Gemini renders JavaScript because it is Google, and for once the folklore had authoritative backing.
In an interview with Kenichi Suzuki of Faber Company, Google's Martin Splitt confirmed it on the record: the crawler behind Gemini renders too, because the Web Rendering Service is a shared service that, in his words, "Googlebot uses… and Gemini uses… as well."
The same WRS that paints pages for Search feeds Gemini.
There was empirical support to match: when searchVIU ran a controlled schema-and-rendering study in October 2025, Gemini was the only system that extracted a price injected purely by JavaScript at live fetch. The capability is real, Google has stated it plainly, and I had no reason to doubt it.
Then Andre Alpar and his team did the thing the rest of us had not: they built a trap and pasted links into twelve assistants by hand. The design is worth understanding, because its cleanliness is the point:
Each assistant received its own secret, never-linked URL, so every server hit could be attributed to exactly one system.
In the raw HTML, the visible "internal reference number" was a decoy — a fake value sitting in the source for any reader that only reads text.
A single line of external JavaScript replaced it with the real number, and — this is the elegant part — the real value existed nowhere in the HTML and nowhere in the script's own source.
To obtain it, a reader had to execute the code, which called a second endpoint that returned the real value.
A hit on that endpoint in the log is therefore hard proof of execution; there is no way to reach that value by reading files as text.
Every page also carried a unique canary string returned only on a genuine fetch, so a real read could be told apart from a confident hallucination.
The prompt was identical every time: summarize the page and report the internal reference number.
Assistant | Origin | Ran the JavaScript? | Reported |
|---|---|---|---|
ChatGPT | US | No | Decoy (raw HTML) |
Claude | US | No | Decoy (raw HTML) |
Gemini | US | No | Decoy (raw HTML) |
Perplexity | US | No | Decoy; also claimed it "couldn't access" the page |
Meta AI | US | No | Decoy (raw HTML) |
Microsoft Copilot | US | No — fetched the script via Diffbot, never ran it | Decoy (raw HTML) |
Grok | US | Executed on one node, ignored the result | Decoy (raw HTML) |
DeepSeek | China | Yes | Real number |
ERNIE | China | Yes | Real number |
Qwen | China | Yes | Real number |
Kimi | China | Yes | Real number |
Mistral | Europe | Yes | Real number |
The entire US top tier reported the decoy. ChatGPT, Claude, Gemini, Perplexity, Meta AI, Microsoft Copilot, and Grok... every one of them answered from raw HTML. Gemini included.
The only systems that reported the real, JavaScript-only number were the four Chinese assistants and Mistral. A clean geographic split that nobody on the team predicted, and that appears in none of the vendor documentation.
So how can Splitt be right that Gemini renders, and Alpar be right that it did not?
Both are true, because they describe two different pipelines, and the contradiction dissolves the moment you stop conflating them.
Read Splitt's words precisely and the tell is in the timing: the render he describes happens after Googlebot crawls a page, on the way into an index, and completes within minutes. That is the corpus-building lane.
Gemini then grounds its answers on content that was already rendered, days earlier, by someone else's crawl.
Alpar tested something else entirely: a cold, live fetch of a brand-new URL that no crawler had ever seen, with the real value reachable only by executing the script in that moment. He deliberately removed the pre-rendered corpus as a shortcut, and on that path, Gemini read the shell.
The two events sit in different columns:
Splitt's pipeline | Alpar's test | |
|---|---|---|
What triggers it | Googlebot crawls a page for the index | A user pastes a fresh URL and asks about it |
When rendering happens | At crawl/index time, into a corpus | At live fetch time, in-session — or not at all |
Does WRS run? | Yes — shared service, within minutes | No, per the decoy test |
What Gemini reads at answer time | The already-rendered corpus copy | Whatever the cold fetch returned — raw HTML |
Which lane | The index lane | The live-answer lane |
The real variable, then, is not "does Gemini render JavaScript?"
It is "can this content reach Gemini through the pre-rendered index, or only through a cold live fetch?"
If Googlebot has already crawled and rendered your page, Gemini can read that version and appear to render; when in fact it inherited a render it never performed.
If the content lives only behind a live fetch — a fresh URL, a page Google deprioritized and never rendered, anything that surfaces only after interaction — it falls back to raw HTML.
That also resolves the searchVIU result: its test page was discoverable, so Gemini could have answered from the corpus render rather than a live one.
Alpar's endpoint-gated canary is simply the stricter instrument, because it removes the index as a confound.
One correction while we are here, because the headline version of Splitt's quote travels badly. "Google-Extended renders JavaScript" is loose: Google-Extended is a robots.txt control token, not a crawler, and it renders nothing.
Googlebot and WRS do the rendering; Google-Extended is only the switch that decides whether that rendered content may be reused for Gemini, which keeps the rendering firmly in the index lane, exactly where the reconciliation puts it.
The industry, me included, collapsed those two lanes into one confident sentence — "Gemini renders because it's Google" — and the sentence was half-right in precisely the way that costs you visibility.
Two secondary lessons from that test are worth more than the headline.
Self-reports are worthless as evidence. Perplexity told the chat window it could not access the page while its own crawler had already fetched that exact URL and received a 200. Believe the assistant's account of its own behavior and you draw the opposite conclusion. The chatbot's description of what it did is a sign; the server log is the referent. When those two disagree — and they do — you trust the log.
The pipeline and the answer are different questions. Grok actually executed the JavaScript on one node of its proxy swarm; the follow-up endpoint fired in the log. And Grok still answered from the decoy. The machinery ran the script and the model ignored what it found. "Did a crawler touch my page" and "did the answer use what the crawler found" are not the same interrogation and conflating them will mislead you in both directions.
This is precisely why we must include a Breach Diagnostic analysis in any serious AI-crawlability audit: you do not ask a system what it can see, you construct a page whose two versions disagree and watch which one it reports back.
Alpar's decoy is a Breach Diagnostic run in public, at scale, on twelve subjects. Anyone auditing a client site can run a smaller version on their own domain tomorrow and should.
Access, retrieval, and visibility are different stages of the same pipeline. Once you know what AI search crawlers can reach, the next question becomes: do you make it through to the answer?
Advanced Web Ranking tracks that outcome across AI search, so you can see when your presence actually turns into impact.
Retrieval, Not Indexing: Understanding the AI Grounding Budget
Step back from rendering to the deeper architectural truth these systems share.
They are not indexers in the Googlebot sense, aka patient machines that store, re-render, and rank the whole web over years.
They are one-shot retrieval engines. They fetch a handful of URLs on demand, strip the markup, chunk the text into atomic passages, embed those passages, score them against the query by similarity, feed the winners into a fixed grounding budget, and discard the rest.

The index lane (top) vs the live-answer lane (bottom): the same content reaches an assistant by two different paths, and only one of them runs your JavaScript.
The most rigorous look inside that budget comes from Dan Petrovic's grounding work at DEJAN, published in December 2025.
Grounding parameter | What DEJAN measured |
|---|---|
Budget per query | ~2,000 words total (median ~1,929), stable regardless of source count or page length |
Average extracted chunk | ~15.5 words |
Allocation by rank | #1 source ≈ 2× the share of #5 |
Extraction method | Extractive — verbatim sentences, not paraphrase |
Snippet-to-page similarity | ~0.916 cosine |
Analyzing 7,060 queries across 2,275 tokenized pages and some 883,262 snippets, he found the budget stays remarkably consistent regardless of how many sources are used or how long the pages are, and that allocation follows relevance rank.
The extraction is extractive, not abstractive: the system lifts actual sentences rather than paraphrasing, a behavior Petrovic replicated by fine-tuning an open model to match it closely.
I flag, as he does, that the confounds here — authority, freshness, page structure — were not controlled, and the budget figure is a median rather than a hard ceiling. But the strategic implication survives every caveat, and it is the strongest argument I know for density over length.
You are not competing to have your page read. You are competing for a slice of a fixed pie, sentence by sentence, against every other source the system retrieved for that query. Fifteen words at a time.
This is where the Linguistic Moat does its work. If the sentences that answer a sub-question are yours — self-contained, entity-named, quotable in isolation — the extractive selector keeps reaching for you, and the moat is the accumulated defensibility of owning the language around your topic at the sentence level.
Read a candidate sentence aloud, out of context. If it still resolves — if it names its entity instead of leaning on an unresolved "it," if it states its claim without needing the paragraph around it — you have laid a course of stone. Petrovic's own next move was to test content built as modular "Lego blocks," assembled into different units without narrative fragmentation. That is the correct instinct. Write for the chunk, because the chunk is the unit of competition.
Schema Markup in AI Search: Why Live Fetchers Ignore Structured Data
Here is a finding that looks, at first, like a reason to tear out your structured data, and is in fact the opposite.
In the same October 2025 study, searchVIU tested whether AI systems read JSON-LD during a direct fetch. They do not. A price of €8.99 present only in the JSON-LD — nowhere in the visible HTML — was found by none of the five systems tested. Not one direct-fetch pipeline parsed the structured data.
At the fetch layer, these systems are visible-content extractors, and the schema is invisible to them.
The mistake would be to conclude that schema is dead.
The phase distinction is everything. Schema is very plausibly read upstream, in the index, in other words the index that feeds ChatGPT through Bing and Gemini through Google, both of which maintain dedicated structured-data parsers feeding their knowledge graphs.
What the searchVIU test isolated is the direct-fetch phase, the moment an assistant pulls a single live URL. Schema is ignored there and consumed elsewhere.
This is the distinction the Schema Sacrifice principle is meant to enforce: you sacrifice the expectation that structured data will be read at the point of live fetch and stop treating JSON-LD as a fetch-time signal, while keeping it fully intact for the index and knowledge-graph layer, where it still earns its place.
Sacrifice the wrong dependency, not the markup. Anyone who rips out their JSON-LD because "AI doesn't read schema" has taken a real finding about one phase and applied it, expensively, to the wrong one.
The Invisible Search Layer: Prompt Transformation and Parametric Memory in LLMs

Two bot families, two forms of memory: batch training shapes the parametric prior that decides whether you're even considered; live retrieval supplies the grounded context that decides whether you're cited. You need to win both.
Between the question a user types and the pages a system retrieves sits a layer most publishers still cannot see.
The assistant rewrites the prompt; compresses it, adds a date or a location, turns a question into entity keywords, or fans it out into several narrower sub-queries.
The candidate pool is defined by the rewritten query, not the one the user typed. You are optimizing for a query you never see.
For ChatGPT specifically, that retrieval leans heavily on Bing.
Seer Interactive's analysis of 500-plus citations, published in early 2025 by Christina Blake and Alisa Scharf, found that 87% of SearchGPT citations matched Bing's top organic results, against only 56% for Google, whose median matched citation sat around rank 17.
The translation is uncomfortable and clarifying: if Bing has not indexed your page, ChatGPT search generally cannot find it, whatever your Google position.
This is the front line of what I call the Architecture of Authority — the index presence and entity footprint that determine whether you are even eligible to be retrieved, before any on-page optimization enters the picture.
And it connects to the layer beneath retrieval, the model's own prior.
Before a single page is fetched, the system carries a parametric memory formed in pretraining — a baseline set of brands, entities, and associations built from the training corpus. This is the terrain of what I call the parametric consideration set: brands that are thin or absent in the training data start the race invisible, excluded from the model's internal shortlist before retrieval even begins.
Grounded search can rescue you at query time, but it is doing remedial work against a prior that already leans elsewhere.
Presence in the index gets you retrieved; presence in the training corpus gets you considered. They are different battles, and you need both.
A word now about the most interesting recent work on ChatGPT's retrieval internals, because it deserves both credit and care. Suganthan Mohanadasan and Chris Green each went looking at ChatGPT's actual network traffic rather than its outputs, and surfaced real internal fields the answer never shows you: a turn_use_case classifier that files some queries as pure text and skips the live web entirely, and a result_source field naming the retrieval pipeline behind each cited result — labels including labrador and serp, plus two that map to the commercial scrapers Bright Data and Oxylabs.
That these fields exist, and that some queries never touch the web at all, are genuinely valuable findings; you only need to see a field once to know it is real, and this work opens lines of investigation the rest of us should follow.
Both are also admirably clear about the limits of their own method, and it is worth repeating their caution rather than sanding it off.
Mohanadasan's capture was a few days of a single logged-in account, and he says plainly that the proportions should be read as the shape of the thing, not as a measurement.
Green's larger sample produced a materially different distribution, which is a reminder that which pipeline appears to "dominate" depends heavily on who is querying, from where, and about what.
So take the structure to the bank: there is a routing layer, and it decides whether your page is ever eligible to be fetched. Treat the specific percentages as promising early signal, and the opening of a research programme, not the close of one.
Access, retrieval, and visibility are different stages of the same pipeline. Once you know what AI search crawlers can reach, the next question becomes: do you make it through to the answer?
Advanced Web Ranking tracks that outcome across AI search, so you can see when your presence actually turns into impact.
AI Crawlers and Robots.txt: What SEOs Can and Cannot Control
The compliance picture divides along the same seam as everything else: automated crawlers versus user-triggered fetchers.
Training and search crawlers respect robots.txt.
The user-triggered fetchers are where it gets interesting, because several vendors treat a live fetch as an extension of a human action and therefore exempt it:
OpenAI documents that robots.txt rules may not apply to ChatGPT-User, though it has been observed complying in testing.
Perplexity states that its user fetcher generally ignores robots.txt.
Anthropic is the strict outlier, and Claude-User honors robots.txt even during active browsing, which means you can actually block live retrieval at the directory level for Claude in a way you cannot reliably do for the others.
Perplexity remains the documented problem case. Cloudflare accused it in August 2025 of using undeclared stealth crawlers to evade no-crawl directives, de-listing it from its verified-bot program; Perplexity disputed the characterization, and the disagreement was never formally resolved. The pattern later escalated: a US federal court blocked Perplexity's Comet agent from Amazon over allegations it disguised itself as an ordinary Chrome session. Whatever your read on the specifics, the operational lesson is fixed.
That lesson forces a correction of a claim I have seen repeated: Google-Agent is not Gemini's live grounding fetcher. It is the agentic-browser identity added in March 2026 as a general user-triggered-fetcher identity — first used by Project Mariner, and since that project's shutdown in May 2026, carried by Gemini Agent and AI Mode — the thing that fills forms and clicks buttons on a user's behalf, the analogue of ChatGPT's agent and Claude in Chrome, not the retrieval path that grounds an ordinary Gemini answer. It publishes its ranges in a dedicated file, it is experimenting with the Web Bot Auth signing protocol, and, as a user-triggered fetcher, it ignores robots.txt by design.
It is worth watching. It is not the mechanism that reads your page when someone asks Gemini a question and treating it as such muddles the very distinction this whole piece turns on.
Underneath all of it sits the fact that robots.txt was never access control. Per RFC 9309 it is a crawl instruction — a request that well-behaved bots honor and that badly-behaved ones step over.
If you genuinely need to keep a fetcher out, enforcement lives at the CDN, the WAF, and the IP layer, validated against each vendor's published ranges.
A permissive robots.txt paired with a WAF that 403s every AI user agent is a contradiction your logs will happily expose.
How to Run a Breach Diagnostic
Strip away the mechanics and the practitioner's mandate is short, ordered, and unglamorous.
Do this | Because | How to verify |
|---|---|---|
Server-side render or pre-render every citation-critical element | Three of four assistants — and all four on the live path — read only raw HTML | View Source, a JS-disabled reload, and a crawler-UA fetch |
Write self-contained, entity-named answers after each H2 | Grounding is extractive and rank-weighted, ~15 words at a time | Read each sentence aloud out of context — does it still resolve? |
Keep JSON-LD, but for the index, not the fetch | Ignored at direct fetch; consumed upstream by Bing and Google | Confirm indexation and rich-result eligibility in both engines |
Govern robots.txt per token; enforce at the WAF | robots.txt is a crawl instruction, not access control | Validate AI traffic against published IP ranges, not the user-agent string |
Run a Breach Diagnostic; read the log, not the reply | Self-reports contradict server logs; pipeline ≠ answer | Decoy/real page plus a canary; monthly log analysis of AI bot fetches |
Two of these deserve emphasis because they are the ones most often skipped:
Keeping your schema is not a contradiction of the finding that schema goes unread at fetch; it serves a different phase, and removing it costs you index-layer signal for no fetch-layer gain.
Running a Breach Diagnostic is the only way to replace belief with evidence on your own domain: build a page whose rendered and raw versions disagree, hand its URL to each system, and watch which value comes back. When a chatbot then tells you what it did, believe the server before you believe the sentence.
Access, retrieval, and visibility are different stages of the same pipeline. Once you know what AI search crawlers can reach, the next question becomes: do you make it through to the answer?
Advanced Web Ranking tracks that outcome across AI search, so you can see when your presence actually turns into impact.
What the Evidence Shows
We started with a page that could tell two truths. The whole exercise has been a way of asking what kind of reader each of these systems really is; and the answer, for the American assistants that most of your audience actually uses, is that they are literal readers. They read the text as delivered, not the text as performed. They take the page at its source, before the code runs, before the schema is parsed, before the JavaScript paints the version a human would see.
That is not a limitation to lament but a specification to build against.
The gap between what a system claims and what its log records is now the central discipline of AI Search SEO: the difference between the sign the chatbot offers you and the referent the server holds.
Vendors will ship new pipelines, the geographic splits will shift, and some specifics in this piece will age. One law will not: the only content that survives every one of these changes is the content that is actually there, in the source, the moment the page is fetched.
Build for the reader that runs no code. Then check the log to be sure.
Article by
Gianluca Fiorelli
With almost 20 years of experience in web marketing, Gianluca Fiorelli is a Strategic and International SEO Consultant who helps businesses improve their visibility and performance on organic search. Gianluca collaborated with clients from various industries and regions, such as Glassdoor, Idealista, Rastreator.com, Outsystems, Chess.com, SIXT Ride, Vegetables by Bayer, Visit California, Gamepix, James Edition and many others.
A very active member of the SEO community, Gianluca daily shares his insights and best practices on SEO, content, Search marketing strategy and the evolution of Search on social media channels such as X, Bluesky and LinkedIn and through the blog on his website: IloveSEO.net.





