In almost every discovery call now, someone asks me the same question. And it is, I have to say it, the wrong question:
"How do we rank in AI search?"
I perfectly understand why they ask it: it is the question we have all been asking for twenty years, simply with the noun substituted.
But it carries inside itself an assumption that does not survive the impact of how these systems really function: that visibility is decided the moment the query is performed, through a process we can influence at that same moment. For the ten blue links, this was true. Here, it is true only in part.
Whoever has worked on a film set knows that, by the time the cameras begin to roll, the decisions that matter have already been taken: the casting closed months before, the shooting script was locked. Principal photography is where the work becomes visible; it is not where the work was decided.
This, precisely, is the structure of AI search. A considerable part of whether your brand appears inside an answer was decided before the user had typed anything at all: during pre-training, during post-training, in the slow accumulation of what the model came to believe about your category. Retrieval is the principal photography: loud, measurable, the place where the deliverables live. It is not the place where the casting happened.
What follows is a map of that structure and a diagnostic ladder for working inside it. It is constructed deliberately to serve two functions: as a taxonomy of where visibility originates, and as a decision tree that can be run against a real client account.
Why to map AI search visibility by mechanism, not by platform
The first instinct is to branch by platform: Gemini and AI Mode here, ChatGPT there, and then Claude, Perplexity, Copilot, Grok.
I attempted it. The result is an enormous map, repetitive and analytically useless, because underneath every branch the same four or five mechanisms repeat with different dial settings; one ends up documenting the surface differences between products while obscuring precisely the shared machinery that determines the outcomes.
It is the SEO-centric habit reasserting itself in new clothes. We organised by search engine for years, because effectively there was one (I am sorry, Bing), and its interface was the model. The moment the interfaces multiply, that habit converts itself into a liability.
Organise by mechanism instead, and the platforms become attributes rather than branches. The operational differences are genuine and deserve recording; but they belong in your notes as tags upon the relevant leaf, not as five parallel structures obliging you to write "server-side rendering matters" five times.
The mechanisms sort themselves into three layers.
Layer 1: Parametric knowledge (what the AI model already knows about your brand)
This is what the model knows before it consults anything. Weights, not documents. When you ask an assistant to name the leading vendors in a category, and it responds without triggering a search, you are reading the parametric layer directly.
Corpus presence
Where you were read:
Common Crawl and its derivatives.
Curated snapshots.
Licensed corpora.
Books.
Papers.
Transcripts.
Wikipedia and Wikidata, which weigh far beyond their volume, and not for their dimension, but for their structural role as disambiguation anchors towards which other sources point.
Entity resolution
Whether the model has consolidated your dispersed mentions into a single coherent thing, which is the difference between a brand it knows and a brand it has merely encountered.
It depends upon consistency of naming, upon disambiguators where your name collides with something else, and upon the density of co-occurrence between your name and the terms of your category.
Post-training shaping
Instruction tuning, preference data, and the policy that constrains how willing a system is to formulate an unhedged recommendation at all.
Our industry discusses this rarely, and it decides whether the user receives named brands or a diplomatic list of "factors to consider."
Structural limits
The uncomfortable component. There is no write access to the weights, and the horizon is measured in model generations, not quarters.
My working assumption — and I signal it as an assumption, since no laboratory publishes its corpus sampling methodology — is that consistency and breadth of mention count matter more here than recency, which inverts a reflex most of us constructed over two decades.
We cannot optimise for this layer. We can eventually contribute to a substrate that may or may not be sampled and discover the result considerably later.
I have written previously about what I call the parametric consideration set: the shortlist a model has effectively already drafted before retrieval even begins. The formulation was sharpened in an exchange of comments beneath Duane Forrester's "When the Training Data Cutoff Becomes a Ranking Factor", where he describes the identical boundary from the opposite direction: parametric memory and retrieval memory not as two moments in time, but as two distinct systems coexisting inside one model, with different confidence signatures and different attribution behaviour. Brands absent from the model's prior are not competing on a level field at query time; they are competing to be added to a list that has already been written.
Layer 2: Retrieval (what AI search fetches in real time)
This is what gets fetched while the user waits, and it is the layer where the leverage is real within this quarter rather than within the model generation.
AI crawler access
It constitutes the entry condition, and it is far more granular than most enterprise policies still presume.
I will return to it in the forensic section, because there reside the largest and most easily remediable failures.
Query fan-out: how AI search splits one query into many
It is the mechanism that renders AI search structurally different from ranking. Google's own documentation on AI features describes AI Overviews and AI Mode as issuing multiple related searches across subtopics and data sources to construct a response.
Aleyda Solís has produced the clearest operational breakdown: a single query becomes a set of sub-queries spanning the live web, the Knowledge Graph and specialised sources, selected according to the inferred facets of the original intent. Mike King frames the matter as sub-intents, for instance, "best sneakers for walking" fragmenting into questions about seasons, terrain and form factor that the user never typed. And Marie Haynes reported from Google I/O that a dedicated Gemini variant generates the fan-out itself.
The forensic detail worth retaining is this: Google does not call the procedure "query fan-out" in its patent literature. The Search Engine Land guide traces the mechanism to US11663201B2, filed in 2018 and granted in 2023, where the formal term is query variant generation, aka a trained generative model producing related variants of a single query, each submitted separately, the responses subsequently synthesised. The name is ours; the mechanism is theirs, and it precedes our branding of it by years. As so often occurs, the thing we treat as unprecedented possesses a documentary trail.
Which AI crawlers render JavaScript (and which don't)
Most AI fetchers are not browsers, and here the evidence is unusually solid for our industry.
The joint analysis conducted by Vercel and MERJ across more than five hundred million GPTBot fetches found no evidence whatsoever of JavaScript execution. GPTBot downloaded JavaScript files in roughly 11.5% of its requests and ClaudeBot in roughly 24%; neither executed them. The same holds for PerplexityBot, for CCBot, for Bytespider, and for Meta's agent.
They fetch the raw HTML, extract what is present, and depart. No second pass, no render queue.
The exception matters as much as the rule: Google does not belong to this category. Gemini and the AI surfaces within Search employ Googlebot's Web Rendering Service and do execute JavaScript (as declared by Martin Splitt); AppleBot renders as well.
The failure mode is therefore specific and counter-intuitive: a client-rendered page can occupy position one in Google and be, simultaneously, an empty shell for every other answer engine. Two systems, two divergent verdicts upon the identical URL.
The simplest diagnostic frame I employ in every audit descends from this (and it is very effective with clients): compare the server-rendered HTML against the rendered screenshot. Everything falling into the gap is content you believe you have published and which most of these systems cannot see. And the engineering rule descends from the weakest reader, not from the strongest.
Grounding sources: which pages AI answers actually cite
The sources that receive the citation, and here we arrive at the part clients find hardest to accept: most of them are not ours. Third-party listicles, comparison pages, review platforms, Reddit threads, YouTube transcripts. Our domain is one candidate among many, and frequently not the strongest for the sub-query that matters.
Layer 3: Synthesis (what the AI answer says about your brand)
What the user finally sees, and where the two preceding layers converge.
I employ a four-state visibility framework here, since "did we appear" is not a question of sufficient precision:
Own-page citation — named, and your page is the source.
Third-party citation — named, but someone else's page is the source.
Unlinked mention — named, with no source attached.
Full absence — the answer proceeds without you.
These four states possess different causes and different remedies and collapsing them into a single "AI visibility" score is precisely how reporting degenerates into theatre. An unlinked mention constitutes a parametric success and a retrieval failure. A third-party citation constitutes a grounding success which nevertheless leaves you dependent upon a page you do not control. They are not degrees of the same phenomenon.
Attribute binding: whether AI describes your brand correctly
This is chronically under-audited.
Being named is not the outcome; being named for the right thing is. A brand can be surfaced reliably within its category and framed, with equal reliability, as the budget option or the legacy one.
Outdated claims are remarkably persistent, and the world of miniature paints offers us a wonderful example. Citadel renamed its entire range in 2012: Chaos Black became Abaddon Black, Skull White became White Scar, Devlan Mud became Agrax Earthshade. More than a decade afterwards, the retired names continue to circulate in old forum posts, in conversion charts that carry both nomenclatures, in the vocabulary of painters who learned the first version.
A brand can therefore find itself described by a system through product names it deleted before some of its current customers had begun to paint at all. Nobody issues a correction. The claim simply continues to be true inside the model until sufficient newer material outweighs it.
Context modifiers: why AI answers differ from user to user
Context modifiers explain why two people posing the identical question receive divergent answers:
Persistent memory.
Locale routing.
Prompt phrasing.
Session history.
This renders single-observation reporting worthless. If your tracking consists of one person running one prompt from one account, what you are measuring is that account.
How to measure brand visibility in AI search
Which brings us to measurement, where I prefer to be boring and defensible rather than impressive. A usable prompt set is designed, not collected: it covers the decision journey rather than the head term; it employs the sub-query phrasings fan-out actually generates rather than those a marketer would write; and it is executed repeatedly, from clean sessions, across models, with the variance recorded rather than averaged away.
A practitioner reports share of voice against a named competitive set, citation share held separate from mention share, and attribute sentiment across time.
Referral traffic belongs nowhere in that list: a downstream consequence of visibility, occasionally correlated with it and a poor proxy for it, particularly within a system whose entire design intent consists in answering without a click.
How Google, ChatGPT, Perplexity and Claude differ in AI search
Having argued that platforms should not constitute branches, I owe the concrete version of what they are.
Google (AI Overviews and AI Mode)
The most aggressive query expansion, drawing upon the Knowledge Graph alongside the live web, rendering JavaScript, and governed for AI-surface purposes by Googlebot rather than by the token most people presume.
ChatGPT
Persistent user memory layered over retrieval, which converts individual personalisation into a genuine variable, together with the clearest published separation between training access and search access.
Perplexity
Its own index, and the most citation-forward interface in the category; useful therefore as an early-warning system for grounding problems, and simultaneously the platform where crawler policy is most contested.
Claude
It merits a longer note, since its retrieval backend is more ambiguous than most write-ups admit.
Brave Search figures upon Anthropic's published subprocessor list and Claude's web search function carries a Brave-named parameter, which constitutes strong evidence; yet Anthropic has never officially named a search provider.
Then, a second web-search subprocessor was added during 2026, and Anthropic meanwhile operates its own Claude-SearchBot for indexing.
The honest description is a hybrid whose exact composition is not publicly established, and I should treat with real caution any playbook constructed upon a single day-one citation-overlap study.
The forensics: what each provider's documentation says
Worth real time, this section, because here reside the most expensive and most avoidable failures, and here the industry consensus is measurably mistaken.
There exists no such thing as "the AI crawler." Every major provider now operates a split fleet with independent controls, and blocking one accomplishes nothing whatsoever against the others.
GPTBot for model training.
OAI-SearchBot for appearing within ChatGPT's search features.
ChatGPT-User for user-triggered fetches.
Each setting operates independently; one may permit search inclusion while refusing training.
OpenAI's documentation is explicit that sites opted out of OAI-SearchBot will not appear in ChatGPT search answers, from which it follows that the common enterprise policy of "block GPTBot, we do not wish to train their models" is not, in itself, the visibility decision those enterprises believe they are taking.
Anthropic operates the identical three-tier structure. Its documentation update of February 2026 formalised ClaudeBot for training, Claude-User for user-driven fetches, and Claude-SearchBot for search indexing, accompanied by an explicit statement that blocking the search bot may reduce a site's visibility within Claude's answers. That same update observed that IP-based blocking is unreliable, since the bots operate upon public cloud ranges. One further item merits an audit: Claude-Web and anthropic-ai are deprecated; if your robots.txt still carries rules addressed to them, those rules accomplish nothing.
Perplexity operates a two-tier version, PerplexityBot for indexing and Perplexity-User for live retrieval, and here the picture becomes genuinely contested. Perplexity has argued that its user-initiated fetcher constitutes an agent rather than a bot and is therefore not obliged to honour robots.txt.
Cloudflare has publicly reported observing undeclared crawling which rotated identifiers to circumvent no-crawl directives; Perplexity disputed that characterisation. Its own help centre states that a blocked page may nevertheless be indexed at the level of domain, headline and a brief factual summary.
I do not intend to adjudicate the dispute and would encourage reading both sides before repeating either. But a diagnostic model is obliged to record the situation honestly: the behaviour of this token is contested, and if your access policy matters, enforce it at more than one layer rather than confiding in the file alone.
Myth: Blocking Google-Extended removes you from AI Overviews
Now, the point I wish to fix definitively, because I still hear it repeated by people who ought to know better.
Blocking Google-Extended does not remove you from AI Overviews or from AI Mode.
Google's documentation is unambiguous that the token neither affects Search inclusion nor operates as a ranking signal. But the corollary runs in the other direction as well, and it is precisely the half everybody discards: AI Overviews and AI Mode are constructed upon Googlebot's index. Google-Extended governs training and grounding for Gemini Apps and Vertex AI, which constitute a different set of products. Disallowing it opts you out of those. It does not touch the AI surfaces inside Search. The only way to not have our content included in Google AI Search surfaces like AIO and AI Mode is explicitly opting out from them via Google Search Console.
A further complication defeats log analysis: Google-Extended is not a crawler. It possesses no user-agent string and fetches nothing whatsoever. It is a control token evaluated against content Googlebot has already retrieved. You will never encounter it in your logs, and searching for it constitutes a diagnostic dead end.
This matters because I continue to encounter publishers convinced they have adopted a principled stance against AI by disallowing Google-Extended, while remaining entirely present within AI Overviews; and other brands that avoided the token from fear of a ranking penalty that does not exist. Both errors proceed from the same origin: treating a robot’s token as a policy statement rather than reading what it controls.
The diagnostic ladder: 4 gates for auditing AI search visibility
Taxonomy indicates where things reside. It does not indicate what to do on Tuesday. The same map therefore runs as a decision tree, with four gates in a determined order.
That order is not arbitrary: it proceeds from the cheapest to verify and the hardest blocking towards the slowest and the least directly actionable.
Gate 1: Can AI crawlers access and read your pages?
Pure plumbing:
Per-agent directives written separately for training bots and for search bots.
WAF and CDN rules audited, since a correct robots.txt undone by a rate limit returning 429s constitutes one of the commonest causes of absent citations notwithstanding an open crawler policy.
Server-rendered HTML for anything you need to be cited. Nothing concealed behind consent walls, tabs, or accordions.
As you probably know, I am a mini painter hobbyist, and the commonest version I encounter among paint retailers and manufacturers is a JavaScript storefront where the colour ranges, the price, and the description all arrive after hydration.
For Googlebot, it renders perfectly; for ClaudeBot, GPTBot, and PerplexityBot, the product page is an empty shell, so when somebody asks which range offers the widest selection of metallics, the brand that does offer it is simply not in the running.
If no: remedy this before anything else and expect results within days. There exists essentially no off-site work here, and the only external lever is your CDN vendor's defaults, which is exactly why leaving this gate closed is inexcusable.
Gate 2: Does the AI model know your brand without search?
Here I must be precise about something concerning which our industry is habitually careless. A training set cannot be queried. There exists no inspection API for weights, and whoever is selling you a "training data audit" is selling you an inference.
What can be done is to probe. Grounding disabled. One asks, cold: name the leading providers in this category. Then, separately: what do you know about [brand]?
Recall without retrieval constitutes a proxy, and it is a good one, tested across several models and several phrasings, but a proxy, nonetheless. Label it as such in your reporting, or you will find yourself defending a number you cannot substantiate.
Consider, again, the miniature paints. Ask any assistant, cold, to name the leading brands, and you will receive Citadel and Vallejo, almost certainly The Army Painter, probably Scale75 and AK Interactive.
What you will frequently not receive unprompted is Kimera Kolors or Monument's Pro Acryl, notwithstanding that serious painters hold both in extremely high regard.
This is not a judgement of quality: it is a corpus artefact. Citadel possesses decades of forum posts, retailer pages, wikis and army-building content attached to its name; a single-pigment boutique range does not, however excellent the paint may be.
If no, that’s parametric absence.
What can you do?
On-site, the work consists in entity self-description: complete Organization schema with a full sameAs set, founder, foundingDate, parentOrganization, knowsAbout, a single canonical entity home, and naming which never varies between footer, legal entity and social profiles.
Off-site: a Wikidata item with sourced statements, consistency across authoritative directories, presence in press and datasets which the corpora genuinely sample. Wikipedia only where notability is genuine, and whoever must ask whether it is, is not.
Gate 3: Does AI include you in category recommendations?
A different failure altogether. The model knows the entity but does not connect it to the category: it can tell you what you are, yet it does not think of you when asked who does this.
If no — weak category binding.
What could you do?
On-site: comparison and alternatives pages, use-case and vertical pages, definitional content binding you to the attribute vocabulary of your market.
Off-site: being named within third-party "best [category]" pages, review platforms, analyst roundups, and the community threads where the category is genuinely discussed.
This is the branch many misdiagnose. They read "we are not showing up" as a problem of volume and publish forty further articles. Volume does not move category binding: co-occurrence does, aka your name appearing alongside the category and its attributes, upon sources which are not yours. The link is secondary; the adjacency is the asset.
The paint category (yes! I love this hobby) contains here a beautiful and uncomfortable example of the mechanism operating against you.
Nearly every paint conversion chart on the web is keyed to Citadel: one looks up a Citadel colour and finds its Vallejo, Scale75, AK, Pro Acryl and Kimera equivalents. Every one of those charts is a competitor's content, and every one of them reinforces Citadel as the anchor entity of the category, since Citadel is the axis around which the table has been organised. Vallejo appears constantly within these documents, but structurally in the position of the alternative to the reference. Decades of well-intentioned community documentation have been quietly instructing every corpus that one brand is the category, and the remainder are conversions from it.
If you are the reference brand, that constitutes an asset you did not have to purchase. If you are not, the strategic question is not how to enter more charts, since you are already inside them, but how to become the axis of some other table entirely: to be organised around, rather than converted to.
Gate 4 — Are you cited in live AI answers?
Across the fan-out set, not across the head query. Whoever tests only the prompt a client would type is testing perhaps a tenth of the retrieval surface.
Run the fan-out for "best paint for miniatures," and you will encounter sub-queries concerning coverage, metallics, airbrush thinning, matte finish, beginner sets, and price per millilitre. The pages answering them are DakkaDakka's compatibility chart, threads on r/minipainting, YouTube reviewers conducting side-by-side coverage tests, and retailer comparison blogs. Almost none are brand-owned.
A paint manufacturer optimising its own product pages exclusively is competing for a small fraction of the surface that decides the answer.
If no — grounding gap.
What could you do? Divide it first: are you being fetched and losing, or absent from those sub-queries altogether?
The first is a problem of chunkability, which means self-contained passages, headings corresponding to the phrasing fan-out genuinely produces, answer-first paragraphs, specifications in HTML rather than inside not optimized images.
The second is a problem of placement: identify which pages are genuinely cited for your target prompts, then work towards presence upon those specific pages. That is digital PR and partnership work, not on-site SEO, and it is the discipline for which most in-house teams possess no process whatsoever.
How the 8-task content hub workflow maps to the 4 gates
In the past, I proposed frequently an 8-task content hub workflow: the Architecture of Authority. How does it relate to this map? The answer is more specific than "it helps with AI search."
It constitutes a Gate 3 and Gate 4 instrument, and it is almost entirely on-site.
Task 1, the 360-degree audit, feeds both gates:
Extracting PAA and fan-out queries.
Benchmarking against AI-generated answers.
Running cosine similarity against the synthetic answer to identify the semantic gaps. That is Gate 4 reconnaissance.
Tasks 2 and 3 constitute Gate 3 work:
Constructing the category.
Attribute coverage, which binds a brand to its topic.
Task 4, the internal linking, serves both, consolidating topical authority while furnishing retrieval systems with a legible structure.
Task 5, the selection of format, is Gate 4 chunkability under another name: choosing the comparison table over the essay, because the table is what gets extracted.
What the workflow does not do is remedy Gates 1 and 2, and I should prefer precision to overselling my own framework. No content hub opens a crawler blocked at the WAF, and none introduces you into parametric memory, which is constructed from corroboration across sources you do not own.
The hub can feed that process by furnishing third parties with something worth citing; but the causal chain passes through them, not through your sitemap.
The workflow is therefore the engine for two gates of the four. Running it while Gate 1 is failing constitutes expensive theatre. Running it in the expectation that Gate 2 will move constitutes a category error concerning what on-site work can reach.
Myth: Organization schema gets you into the Knowledge Graph
"Organization schema introduces you into the Knowledge Graph."
It does not. Google's own documentation describes organisation structured data as assisting Google to comprehend and disambiguate your organisation, with several properties operating behind the scenes for precisely that purpose — and states plainly that the features consuming structured data are not guaranteed to appear. Schema constitutes a corroboration signal: it indicates to a parser which existing entity your page refers to.
Entity recognition is resolved, not submitted. What moves it is independent sources agreeing about you; the markup renders that agreement machine-legible. To recommend better Organization markup is correct. To present it as the mechanism by which one enters the Knowledge Graph is a category error, and it establishes a client expectation you will be unable to satisfy.
Limitations of the framework: a loop, not a tree
Two matters a tree cannot express, and which I should prefer to state rather than permit the diagram to imply a false tidiness.
It is a loop, not a tree, and this part is my hypothesis, not documented fact.
I wish to be explicit about the epistemic status here, since this is at once the most interesting claim in the piece and the least verifiable. My argument runs thus: the parametric layer conditions retrieval, because fan-out drafts its sub-queries from what the model already believes about the category, which would signify that the shortlist is shaped before any fetch occurs, with retrieval largely furnishing fresher sources for brands already inside the set.
What is documented is that the fan-out sub-queries are model-generated, and that the model carries priors into that generation. What is not documented by anyone is the degree to which brand-level parametric knowledge biases which sub-queries get produced. It remains an inference: reasonable, consistent with what practitioners observe, unconfirmed. Treat it as a working model which predicts well, and whoever constructs client strategy upon it should say so in those terms.
It is a ladder, not a funnel. Two gates can fail simultaneously. A "no" at Gate 2 does not preclude a "yes" at Gate 4, because that is precisely the third-party-citation state, and it is fragile, since it survives exactly as long as the source page which carries it.
And the expectation-setting that must accompany any deliverable constructed upon this:
Gate 1 resolves within days;
Gate 4 within weeks or months;
Gates 2 and 3 operate upon model generations.
Whoever presents all four inside a single quarterly report against a single timeline will find the two that matter most judged as failures and cut before they could have worked.
The map is a working document
I constructed this because I had grown tired of holding it in fragments — a crawler audit here, an entity note there, a prompt-testing spreadsheet somewhere else — and of never being completely able to show a client the whole shape of the problem at once.
What surprised me while assembling it was how little of the structure is genuinely new, and how much of the confusion proceeds from our habit of organising knowledge around interfaces rather than mechanisms. Cease to branch by platform, and the machinery underneath proves small enough to hold in the head: three layers, four gates, two levers per gate.
The limitation is that the parametric layer, which matters most, is simultaneously the one we can least directly touch and least reliably measure. I do not consider that a reason to omit it from the map. I consider it a reason to be candid with clients concerning which portions of the work constitute engineering and which constitute cultivation, and to cease pretending that the second kind reports like the first.
As I wrote at the beginning, the casting happens before the shooting. The best one can do is to ensure that, when the next round of casting arrives, the material concerning you is abundant, consistent and unambiguous; and that meanwhile, while the cameras are rolling, nothing about your site prevents you from being seen.
Article by
Gianluca Fiorelli
With almost 20 years of experience in web marketing, Gianluca Fiorelli is a Strategic and International SEO Consultant who helps businesses improve their visibility and performance on organic search. Gianluca collaborated with clients from various industries and regions, such as Glassdoor, Idealista, Rastreator.com, Outsystems, Chess.com, SIXT Ride, Vegetables by Bayer, Visit California, Gamepix, James Edition and many others.
A very active member of the SEO community, Gianluca daily shares his insights and best practices on SEO, content, Search marketing strategy and the evolution of Search on social media channels such as X, Bluesky and LinkedIn and through the blog on his website: IloveSEO.net.
