Welcome back to The Search Session. I’m Gianluca Fiorelli. Joining me is Crystal Carter, an AI and SEO strategist, based in the UK.
Our conversation explores the information AI systems learn about brands and the importance of keeping it accurate and clear. Crystal shares practical ways businesses can start testing agentic tools through existing partners, along with the roles of llms.txt and MCP. We also discuss personalized search and the evidence brands can provide to show they meet their customers’ needs.
What you’ll learn in this session
Why SEOs need to pay more attention to agentic search: as adoption grows, understanding how agents access websites and interpret businesses will become increasingly important.
What the new rules of AI search cover: Crystal’s book explores the broader shift in AI search, including agentic systems, and why this new era calls for a new approach to SEO.
How Crystal began investigating brand visibility in AI search: agencies’ questions about appearing in ChatGPT and experiments with her own name and a new website prompted her to explore how LLMs recognize brands.
Why accurate brand information matters in training data: outdated facts about products or locations can persist in models’ learned knowledge, making it important to correct sources before future training.
Why clear entity associations matter for AI visibility: clearly connecting a brand to its core products, services, and topics helps AI systems understand what it does and distinguish it from unrelated businesses.
How small businesses can start testing agentic opportunities now: use tools available through existing platforms and payment partners to experiment with minimal effort and prepare for wider adoption.
How llms.txt can help AI agents use a website: it provides guidance and links to relevant content and tools, making information easier to access and process, without guaranteeing visibility in ChatGPT.
How MCP expands access to brand content for AI agents: it can let users access restricted information through authentication, creating opportunities for publishers and B2C brands.
How brands can support visibility in personalized AI search: understand target customers and provide clear evidence that products or services meet their specific needs, through detailed content and contextual reviews.
Tune in for a conversation grounded in Crystal’s work at Wix and her own experiments with AI search.
Topics covered: agentic search · agentic commerce · AI training data · entity recognition · llms.txt and Markdown · MCP · personalized AI search · customer-focused content · SEO fundamentals · contextual backlinks and brand mentions
About the Guest
Crystal Carter
Head of AI Search & SEO Communications at Wix
Crystal Carter has around 20 years of experience in SEO and digital marketing. She helps brands understand changes in search and improve their visibility across search engines, AI assistants, and emerging agentic platforms.
In 2022, she joined Wix, a platform for building websites and managing online businesses. Crystal leads launch strategies for AI search and SEO features, works with product teams, and manages key partnerships. She also creates research and educational content and supports Wix’s SEO and AI visibility efforts.
She is the author of The New Rules of AI Search, published in 2026, a practical guide to brand visibility in ChatGPT, Google Gemini, and generative search. She regularly speaks at industry events, sharing insights on SEO, AI search, and AI agents.
Video Chapters
Transcript
Full conversation between Gianluca Fiorelli and Crystal Carter.
Gianluca Fiorelli: Hi, I'm Gianluca Fiorelli, and welcome back to The Search Session. Today we are going to have an Amazon bestseller writer. Our guest wrote The New Rules of AI Search, and we are surely going to talk about her book. But you probably know her because you have read a lot of posts she wrote and listened to her talking at conferences like Google Search Central, BrightonSEO, SMX, MozCon, and many other talks.
BrightonSEO, surely I already said it, but I remember having seen her there, also at MozCon. And she's a wonderful speaker. If I tell you her name, you probably already know her. So our guest today is Crystal Carter. Hey, Crystal, how are you doing?
Crystal Carter: Hello. Ciao, Gianluca. Thank you so much for having me.
Gianluca Fiorelli: It's been a long time since I was looking to have you as a guest. Finally, we had the opportunity to do it. How are things going? Are you having a good time? I said that we are used to meeting up sometimes, especially at BrightonSEO. But are you going to speak in other places during this fall and winter?
Crystal Carter: I just wrapped up a session at BrightonSEO where I was talking about agentic optimization for websites. And over the summer, I spoke at MozCon. This autumn, I'll be speaking at Semrush Spotlight. And in the same week, basically, I've got a crazy week in October where I'm speaking at Semrush Spotlight on Tuesday, and at the NEZ conference run by John Shehata, all about news, also on Tuesday. Then I'm speaking in Cyprus on Thursday at an AI event there with the lovely team at Boussias Event. And then on Friday, I'm speaking in Rome at SERP Conf.
Gianluca Fiorelli: We will meet because I will be speaking there too. During that period, you're going to substantially live between airplanes and hotel rooms.
Crystal Carter: This is the thing. And it's okay; I've got lounge passes. I get all my air miles. I'll be fine.
Gianluca Fiorelli: Anyway, this is great news. That means that surely everything you are sharing is resonating so much that people want you to share it even more at conferences. But in general, how is SEO treating you lately?
Crystal Carter: I think, like most people, my head's spinning. It certainly keeps you on your toes. Every day there's something new. Every day, Google is changing how it does things, both on the SERP and in the mechanisms it uses.
I've been talking about agentic since the beginning of the year, and one of the reasons was that, at Wix, we work with a lot of partners. Google is one of our partners. Stripe is one of our partners; Etsy and eBay—we work with lots and lots of different partners.
And as a marketer, I'm seeing a lot of our partnership discussions talking about agentic. Particularly, we've got agentic partnerships with Stripe, with OpenAI, and with PayPal, for instance.
I started seeing lots and lots of those discussions happening within our team. But the other thing that I wasn't seeing was lots of SEOs talking about it. So I started talking about agentic more in that space, and I think that's something that's really important for folks to be thinking about.
It's slowly gearing up. We've been saying that it's going to be the age of agentic every year for the last couple of years. And I genuinely think that the time is coming soon. I think it's going to be slow until it's fast.
Basically, we are doing lots of stuff about optimizing for LLM responses and things like that. But with agents, those responses happen at scale, and they happen in a programmatic kind of way. And if you're not jacked into the program, then you're going to miss the boat, basically.
So I think it's really important for people to start getting their heads around how agents will access their websites and how agents will think about their brands and their businesses.
What Are the New Rules of AI Search?
Gianluca Fiorelli: Am I right in thinking that, because if you look superficially at the title of your book, the title is The New Rules of AI Search, substantially. To have a title about AI search for a book about AI search with the new rules at first can look weird because it was only four years ago that OpenAI came out with ChatGPT.
So am I right to think, also as a hint for the potential new readers of your book, that the new rules are substantially the rules of agentic search, which are different from the rules of classic, let's call it already classic, AI search that we have talked about until now?
Crystal Carter: I think that The New Rules of AI Search is partially talking about the fact that we have new rules for AI search as a whole. And there are definitely considerations around agentic in the book. So I talk about how agentic is something that we need to be playing into.
In terms of how the title positions itself across the whole of the SEO ecosystem, it's very tricky because, as you know, as somebody who's been studying semantic SEO for years, AI has been a part of search for years. Featured snippets have been generated by AI and have had a generative element to them for years, I think since 2016 or something. And it's always been the case that if you had a website that had bullets on it, sometimes the featured snippet would show with numbers. Or if the website had numbers, sometimes the featured snippet would show with bullets, and they would mix and match some of the content in there. So there's absolutely been AI and semantic search, and semantic search has been powered by AI, but not necessarily generative AI, for years and years.
When we're thinking about new rules, we're thinking about this is a new era, and this needs new ideas and it needs a new approach. So it's a question of making sure that people are aware of that. With the agentic element, that's absolutely something that I touch on.
People have asked me, “How are you able to write something, and how will it last, a book, as soon as it's been printed?” And I've been talking about this since 2024. One of the things that I found really surprising was that in autumn of 2024, there was hardly anyone. I don't remember seeing anyone talking about ChatGPT or being visible in ChatGPT or gearing up to be visible in ChatGPT at BrightonSEO, for instance.
And I looked at that, and I thought, people are asking me this all the time. They're asking me this question all the time. So I'm going to be the first one. I'm going to just throw my hat in the ring, and here we go.” This is what I think around that.
How Crystal Began Investigating Brand Visibility in AI Search
Gianluca Fiorelli: I think that one of the reasons that maybe made you so early and aware of everything like this is also your position at Wix. Just out of curiosity, how many websites exist based on Wix?
Crystal Carter: I don't know the number. Websites are built every day, but we have over 300 million users worldwide, 22 different languages, and 190 countries that are using Wix.
Gianluca Fiorelli: So let's say that just 1% of all the owners of these websites are actively talking with Wix, asking for features, asking questions, and wanting doubts to be solved. Probably, from the same customers you have as a company, you started to see this sort of interest. And so maybe this is also why.
Crystal Carter: I think yes and no. I don't remember a lot of our Wix users asking us that direct question. However, we launched, and we're launching a new version soon of Wix Studio. Wix Studio is a platform that's built for professionals, for agencies, for B2B, and for people who are building websites for others. Wix is a really great platform to build a website for yourself. Wix Studio is a really good platform to build a website for someone else.
I remember talking to lots of agencies and lots of professionals about this, and lots of the agency owners kept asking me this question. When I was going to conferences, and I spoke at loads of conferences all across the year, people kept asking me this question in person: “How can I be visible in ChatGPT?” I had one agency who said to me, “How come I go on ChatGPT, and I say, ‘What's the best agency in Paris?’ And we don't show up?”
I looked them up, and I went to their website, and it said something like, “We are an award-winning agency for creatives and brand innovation,” or something like that. And then I Googled that same phraseology, and like 20 other websites showed up with the same thing. This sort of got me thinking, got my gears going around that.
Gianluca, we've chatted about a lot of these conversations. We often find ourselves thinking about similar things, but this is how I got into the visual search conversation years ago. I started looking at visual search as well, and I wonder if you went down this track as well. Google kept talking about Google Lens, and I was like, “How do you optimize for Google Lens? They're really into it. How do you optimize for it?”
I kept finding that everyone was just saying the same thing. They were like, “You just update your alt text, and you just update the description.” And I was like, “That's not it. That's not it. This is a machine learning process. It's not just about your alt text. Sure, it'll help, I guess. But it's not just that. You can't just end there.”
So if I find the conversations are happening a lot, and if it just sticks in my brain, that tends to be the kind of thing that I'll go down a rabbit hole on. AI search was one of them.
The other point was when ChatGPT first broke big, I did the thing that you always do, and I asked ChatGPT, “Who's Crystal Carter?”
Gianluca Fiorelli: I did the same.
Crystal Carter: They said, “Huh?” I said, “Crystal Carter, that lady?” And they were like, “We don't know her.”
Gianluca Fiorelli: You started to influence them.
Crystal Carter: Well, a few months later, I asked the same question, and they said, “Crystal Carter is that lady from Wix,” and blah, blah, blah. And I was like, “Okay, this is interesting. How does that work?”
Then I got involved with another project that we were working on, a brand-new website project that we were launching. It was called Site of Sites. I was starting from fresh. How do you start from fresh when there's nothing online about this idea? And I started influencing there as well. There's lots of information that I found really interesting about the inner workings of some of these tools as well, which I think is really valuable.
Why Training Data Shouldn’t Be Overlooked
Gianluca Fiorelli: I think, and I must confess that this is also partly what happened to me, that SEO, and I'm talking just of SEO—because other people outside SEO were immediately super interested in generative AI—maybe because our mental muscle was so used to thinking just in a Google-centric way, until we saw Google really pushing hard on AI search, we thought, “Okay, ChatGPT is that fun thing. It's interesting. We're having fun.” We were using it as a tool but not really thinking about it as a search surface that was going to be a true competitor to Google.
Then Google started to do it—before, with the awful Bard experiment, substantially, but then it improved, and Gemini came out, AI Overviews came out, and then AI Mode, et cetera. I think that was the moment, so 2025. That's why in 2025 everybody started to really focus on this, and there was this big explosion.
But you were saying something interesting about this classic experiment we always do, searching for ourselves. We did it before on Google, and now we are doing it repeatedly on LLMs. I think there's always this kind of—I don't know if it's superficiality or overlooking or not paying too much attention. Normally, in our industry, when we talk about LLMs, we're usually talking only about the grounded experience of the LLMs and not so much about the experience when we ask a question and ChatGPT, for instance, or Claude answers us with a parametric answer, so with the training data.
I don't know if you agree with me, but this can also lead to the conception that the most important thing is not really being in the training data, because it's updating so slowly every time, but being in the grounded data. I think it's a myopic way of thinking.
Crystal Carter: Yes. I absolutely agree. In my book, there's a whole section that talks about optimizing for training data, and I do not think that this is something that should be overlooked at all. Gianluca, I know that you're very interested in entity optimization and semantic optimization. Optimizing for your entity is essentially very, very similar to optimizing for training data.
I think particularly with the agentic movement that's happening, all the agents have a base model that they work from. Your Gemini agents will have Gemini as their base model. All your Claude skills are going to work with a base model from Claude, et cetera. Depending on which agents you're building individually, they will also have a base model that they're connected to. There's always a base model at the end of it, and they will use their training data as much as possible because it's cheaper. It's cheaper than running a crawl every single time.
The thing about the training data that's really important is that it's accurate and it's complete. What you don't want is your training data to go around saying that your HQ is in Boston when your HQ is actually in Rome. If the training data says, “Actually, they're not based in Italy at all; they're based somewhere else,” that's not good for you.
The fact that it updates more slowly is a challenge, so that means it's something that you have to fix. It's easier and quicker to fix the grounded queries. It's easier to fix the RAG data, so the retrieval-augmented generation. It's easier to fix the ones that are connected to the web.
If you see an inaccuracy as you're publishing in real time, let's say you launch a new product and somebody reports it incorrectly. Let's say you launch a new ice cream brand, and they say you have Rocky Road, but you don't have Rocky Road. You have something else, Chunky Monkey, whatever it is. You should go and get that corrected before it ends up in the training data.
Let's say you used to sell Rocky Road and you don't sell it anymore, but it was really popular before, and maybe you did a big campaign launch about your Rocky Road. Then when people go to look for Rocky Road, they'll say, “That brand sells Rocky Road ice cream. It's great. You should go and check it out.” And people will come to your website, and they'll be disappointed. So you need to find out where you have those old comments and where they're getting that information from. Normally, it is easy to find. You can find where it is. Tracking down someone to get it fixed can be challenging. But where possible, you want to correct those statements and potentially create new content to address that.
Case in point, Wix has an AI website builder. We have our classic drag-and-drop. We have our AI website builder, Wix Harmony, and the AI website builder is not our first AI website builder. When ChatGPT broke big, people were like, “But what about websites?” And our CEO, Avishai Abrahami, was like, “We've been doing AI websites since 2016.” We have something called ADI, and we've been doing that for ages. When we launched ADI, it was called Artificial Design Intelligence. There was a big PR push and all of that sort of stuff, and it was huge.
So for ages, there was lots of stuff in the LLMs that was like, “Wix has ADI, Wix has ADI, Wix has ADI.” We spent a lot of effort and a lot of energy making sure that it's very clear that we have a new AI website builder. We've got blogs specifically on our website that say the evolution from ADI to the Wix AI website builder.
So you can do this, and this is a challenge, particularly for legacy brands. If you're an old brand and you've been around, Wix has been around since 2006 or whatever, and in internet days, that's a long time. We're very well-established, and that means there's a lot of stuff on the web about us. Sometimes you need to make sure that those are accurate conversations about who you are today.
How Entities and Query Fan-Out Organize AI Knowledge
Gianluca Fiorelli: I totally agree about this particular nature and the kind of action that must be done for the training data. But I think it's also important to do this work and be present in the training data—with some kind of obvious exception that can usually be in the publisher arena—but, for all the other types of websites, I think it's important to be in the training data in the way we want to be in the training data. At least from experience and from a few tests that I did, if a brand is present in the training data, it's also going to be present in the fan-out when the live search is triggered. So if you are not, you have less chance of being in the answer than a brand that is in the training data.
Crystal Carter: I think it's important to think. The way I've always talked about it is that training data is like an encyclopedia. The encyclopedia will have definitions, information, and a base knowledge of what a flower is or what a bee is or what the Acropolis is. They'll have that base knowledge. So that's your training data.
And then RAG is like somebody who has read the encyclopedia, so they've got a base knowledge of information, and then they also read the newspaper every day. That's information that's coming through every day.
For instance, with the Acropolis, they'll know the Acropolis now has an elevator. The elevator was closed last week because there was a storm, that sort of thing. So they'll be able to update that kind of information. But the training data is what will create connections. If you think about the Acropolis, for instance, you'll know that the Acropolis is an entity that is related to Greece, ancient Greece, and Greek mythology. It's related to lots of specific things, like architecture. So it will be related to those things. And because it is related to those things, it will become parceled into a knowledge base around that.
In the book, I also talk about mixture of experts and optimizing for a mixture of experts architecture, which is essentially what query fan-out is. Google calls it query fan-out; machine learning people call it mixture of experts architecture. Essentially, what it means is that back in the day, they had dense transformers, and they would just put all the information into a big bucket. That's really expensive when you need to find something because you have to sort through everything to find it all. Like trying to find your socks when you haven't paired them. But if you put all your socks into different pairs, into different sections, or you put them into drawers, then when you need specific information about a specific thing, you can go to that section.
For instance, if somebody asks a question about ancient architecture, then they would find the Acropolis in that group of information about ancient architecture because it's related in that entity. Again, part of the training data, part of all of the current information, all of that sort of stuff. If someone was looking for information about the Buffalo Bills football, American football team, the Acropolis is not going to be anywhere near that because there are no entities related to that, unless they had some random PR something. But there won't be any entities related to the Acropolis that have to do with that.
So when you think about the query fan-out, this is the context in which your website and your brand are mentioned regularly. Wix is regularly mentioned as a website builder, so if somebody's looking for website builder information, we will show up as part of that base of knowledge. If someone was looking for information about movies, then IMDb would be one of the websites that the query fan-out would check. They're not going to necessarily have an answer for every single thing, but they're definitely a reliable source for this. They are an entity that is related to that.
Keeping Your Brand Salient and Avoiding Entity Confusion
Crystal Carter: Now, when you think about your brand, it's really, really important that your brand has clear salience and clear visibility around what your entity is and who you are. If you don't, then they don't know which bucket to put you in. And when they need to find you, they don't know how to find you.
I was talking about the sock drawer. This is the first time I've used this metaphor, but I have three main drawers. I have my socks and stuff. I've got my shirts, and then I have my trousers. These are my main drawers, and then everything else goes in the wardrobe on a hanger. But sometimes I have a hoodie. Do I put the hoodie in with the shirts, or do I put it on the hanger? I'm not sure. So you have to make sure that it's really clear that they know exactly which drawer to put you in and that they're not unclear about it. What about a shirt dress? Where do you put that? You want to make sure that your brand is really, really clear.
If you're an ice cream place and you sell ice cream, make sure that they know that you're an ice cream place and that's what you sell. If you also sell candy or milkshakes or things like that, for instance, I don't know why I always talk about ice cream, but if you were also selling those sorts of things, then you want to tie them in and ideally relate ice cream to it where possible.
If you want to pivot, which is something that lots of brands do, then maybe think about not having that on your domain. Maybe think about having a dedicated domain specifically for that new pivot so that you can build up the entity relationships around that and keep your core brand salient with that particular entity relationship. I say this because I've seen this happen. There was a brand I was working with, and they sold spices and things like that. They had this issue where the LLMs kept saying that they did denim refurbishment and event planning and lots of other random things. And I was like, “Why is this? What?”
So I started digging around, trying to find where the LLMs were getting this from. Why are they hallucinating this? Because hallucinations very rarely come from thin air. They normally come from something that they've gotten confused about. It turns out that this business was a small business, and during COVID, they set up a marketplace to support other small businesses. They put on their website lots of profiles of these other businesses in this marketplace. They had gotten press coverage, and it did really, really well. So the LLMs took all of that and thought all of that was one business, rather than that being a showcase for other businesses.
If you're getting that sort of thing, then it's important to make sure that, if you want to keep that going, maybe spin that off into a separate website so that you've got this new project, et cetera. Or think about maybe noindexing it so that people who want to access it can still access it, but you're telling LLMs that this is not part of the thing. You could also go into your robots.txt and say, “Don't pay attention to this page,” for instance, that sort of thing.
Have a look at that kind of information and make sure that your brand is really salient with the entities you want to be associated with.
Gianluca Fiorelli: There's a variant to what you just said, and it's happened to me with a client I have. The headline would be “Whenever you are buying a domain name, always be sure to know the story behind that domain name." Once it was for the link profile, whatsoever, but now especially because of the entity story that domain had.
For instance, a client of mine created an informational microsite all about health. They bought a wonderful new domain name. The problem was they didn't check what that domain was before. This new domain was about headaches. One of the terms used with headaches is aura, migraine aura.
Crystal Carter: Okay.
Gianluca Fiorelli: This website before was the website of a new-age therapist able to read your personal…
Crystal Carter: Aura.
Gianluca Fiorelli: …aura. So in the training data, not in the grounded data, obviously, this website is still seen as a new-age therapy website talking about aura in terms of what the color of your aura is. It's quite dramatic because when you are talking about headaches, a lot of the answers that are coming out are parametric, not grounded. So that website is never coming up because it hasn't been able to be included in the training data. Whenever we test it, it's always related to aura.
Crystal Carter: If I were to get creative with that problem, you could potentially make some content that addresses it directly. Like, “Auras don't fix your headaches,” that sort of thing.
Gianluca Fiorelli: Or you can create an article about an unconventional way to resolve your headache.
Crystal Carter: This is true.
Gianluca Fiorelli: —but that we do not suggest to you. That could be an idea.
How Small Businesses Can Start Testing Agentic Opportunities
Gianluca Fiorelli: Anyway, we have talked about this situation, but let's slowly move to agentic. Agentic — at least WebMCP, UCP, ACP, OKF, all these acronyms that, as an Italian, I dislike a lot because I always get confused. But they are still very young. For instance, WebMCP is not even standardized yet, from what I've known.
What can we foresee? What can we eventually test? Recovering one phrase you used before: when going fast is going to be faster, when do you think businesses should start really trying to test these kinds of opportunities?
Sometimes, obviously, it seems to be very focused on everything e-commerce from the beginning. It seems to be the thing that is training this agentic search into something more practical. But when do you think a classic business, an average business, should start trying to say, “Okay, let's test it to see if we can get prepared for this when it's really going to be a rush”?
Crystal Carter: I think there are some great opportunities for smaller businesses to get involved right now. The easiest way for a small business to get involved with the agentic movement is through partners. With partners, a lot of the agentic stuff, the UCP, the ACP, the AP2, the A2A, all of these different things. We've got an article. At the beginning of the year, I spoke to my team and I was like, “We need an article on this,” because my head is spinning with all of these acronyms. We've got a great article on that, written by one of our team that's specifically working in agentic.
Essentially, a lot of these are happening at scale. The agentic payments protocol, for instance, is an arrangement between folks like Stripe, Mastercard, Google, Visa, PayPal, and all of these different payment teams to make sure that agentic payments can happen smoothly and safely.
One of the things that I recommend when I talk about agent readiness—and I've posted blogs, I did a course with Semrush, I've got a blog on our website that talks all about how you can do it as well. Essentially, one of the things I recommend is looking at your partners.
Wix users have lots of different agentic tools available to them. We have the NLWeb app, which uses the Ask protocol, developed by the team at Microsoft. It allows LLMs to ask natural language questions to your website using your schema markup, your structured data, and by that I mean your H1 tags, your H2 tags, your bullets, and all that sort of stuff, and using your RSS feed and your product feed. So using all of that information as grounding for their conversation.
If I say, “I want a size 12 shoe with blue laces and patent leather that's good for dancing,” or something like that, I can ask that really long-winded question, and it'll give me back an answer that makes sense. That's something that you can enable with a button on Wix, and that's something that we've built in the back end. We've done all that legwork.
If you, as an individual business, were to try to do that, it probably wouldn't be worth the investment because of the technical configuration and all of the things it would take you to do. But if you're a Wix user, you just click a button. That's definitely worth the investment. Clicking a button is worth the investment.
The other thing is, and I know people have opinions about llms.txt, but if you want to enable llms.txt, you can even edit the files. If you want to test it out and see if it affects how agents access your website, then that's something that you can also enable on Wix with the click of a button. Again, because the opt-in is such a low amount of input, the ROI can be high. If you get one bit of value out of it, then it's useful for you.
When you get into the commerce space, we are also connected with Stripe for agentic payments, and we're connected with PayPal for agentic payments. This means that you have access to the ACP. We're one of the only CMSs that are a signatory for that. We were the first CMS that was a signatory for the Agentic Commerce Protocol. Then there are lots of other elements.
If you're an even smaller business, for instance, if you're working with Etsy, Etsy is connected with a lot of these programs. eBay is connected with a lot of these programs. This means that you can connect your business to that without having to do loads of the legwork.
For instance, if you think about ChatGPT, I know they've paused their instant shopping things for the moment, but if you wanted to get involved with that, you'd have to sign up and be eligible and all of that sort of stuff. But if you do it through eBay, then it's easier for you to do that.
Again, I think small businesses should just look around at their partners and see where the opportunities are, and if there are opportunities for them to get in without it being much of a lift. If this is not a big lift for you, you can click a button, and you can register to say, “Yes, I would like to make my website available for agentic shopping," and then it's something that will allow you to do it.
I think Klarna also has a tool that lets you do this as well. It can make things a lot easier so that you can get involved and see early on whether or not this is useful for you. If it is, then it means that when it does start to move fast, you're already there.
What SEOs Misunderstood About llms.txt
Gianluca Fiorelli: Now, let's go to the topic of llms.txt, because you wrote and talked a lot about it.
Crystal Carter: I know.
Gianluca Fiorelli: Two questions. What have SEOs misunderstood about llms.txt? Second, I don't want to talk about the polemic, nothing at all. What can be one real use case for llms.txt? To clarify the space, because I think there is a lot of confusion. Confusion creates misunderstanding, and misunderstanding creates heated discussions and stupid discussions also.
Crystal Carter: I think that llmstxt.org, put together by Jeremy Howard, when he originally put it together, he said that it was a tool to make it easier for LLMs to read your website. I think it was early days of talking about Markdown being particularly useful for LLMs and agents, and I talk about this in my book as well.
Basically, what SEOs did is what SEOs always do. Somebody gives us an inch, and we make it a mile. We say, “They said it makes it easier. Fantastic. That means we must rank in ChatGPT with llms.txt.” There were lots of people doing experiments about this to see if this was the case.
Essentially, what has emerged is that they just changed it, so on August 10th, I'm looking at the page right now, llmstxt.org changed their page to say that it was for agents. This comes off the back of lots and lots of developers talking about llms.txt as being a useful tool for agents. While SEOs were going back and forth about whether or not it helps you rank on ChatGPT and whether or not ranking on ChatGPT is a thing at all, developers were quietly, consistently using it for agentic building.
Google's Agent Development Kit—I haven't checked it in the last couple of days, so it may have changed, but for a long time, it had a section where essentially their MCP had a line in it for their llms.txt so that the agents could go and use that to ground their information.
I did lots and lots of research looking at lots of individual llms.txt files and how they all work. One of the things that I found most interesting was the agentic use case. The agentic use case is essentially where you can use it as a really efficient-to-crawl piece of text where you can give the agent specific instructions. You can tell it where your MCP is. You can tell it how to use your content. You can tell it where your other Markdown files are. The reason why this is useful for agents is that it's so cheap. Agents cost money. LLMs cost money. I've been experimenting with an agentic tool recently, and I keep running out of credits because they just run through all the credits.
Essentially, if you give it Markdown, it takes fewer credits. If you compare a Markdown page and look at the source, the Markdown page looks exactly the same in the source as it does when you don't look at the source. Every single character is a token. If you look at an HTML page, they've got to crawl, they've got to scrape, they've got to move all of those things to get to the meat and potatoes of the content.
So when you're working with agents, llms.txt is super useful, or can be super useful, if you're using it to give agents good instructions. This is backed up by the fact that Chrome for Developers added it into their agentic Chrome browser assessment tool. It can be useful. If you expect it to help you rise through the ranks of ChatGPT when people are searching individually, it probably won't. However, it is something that can be used for agents. If you're working with agents a lot, or if you're having people use your website with agents a lot, it can be useful.
Gianluca Fiorelli: It's substantially a sort of classic README file for agents.
Crystal Carter: Yes.
Markdown Files for Gated and Restricted Content
Gianluca Fiorelli: Regarding the .MD file, I'm probably going to start an experiment because, for instance, there are many websites that, for legal reasons, cannot show the content. Or let's think of cases like Statista, which has wonderful content behind sign-in and subscription, et cetera. With Google, it's relatively easy to say, “Google, nosnippet, data-nosnippet,” so you can control what Google is going to use or not in classic and AI search. But obviously, LLMs don't respect something like data-nosnippet. They eat everything, all the content they can encounter.
The experiment that I'm going to roll out with a client is somehow reminding me of the AMP era. Why don't we create a sort of Reader's Digest version of all the things that you can say, and LLMs can say publicly to the general public, of all the things that you are hiding behind a subscription because of legal or commercial reasons?
The problem is that if people are searching for something on ChatGPT, for instance, right now you don't show it because you are blocking them. It's citing websites that are legally allowed to talk about the things you're talking about behind the login. So to show the LLMs that you are the source, why don't you create an LLM folder with just .MD files, open just to LLMs, blocking everything that's related to Google, so Google is not going to have access to them? You are feeding the LLMs with this information, the information that you can publicly make available.
People tend to think, “Why are you going to recreate an entire .MD version?” Obviously, nobody wants to create a Markdown version of a website. Also, for Google, that would be utterly stupid. But there are nuances, and sometimes people forget that there are nuances in everything we are doing now.
Crystal Carter: I think the other thing is, when people were having lots of discussions about llms.txt and Markdown files, there are a couple of providers who are doing that. You go to the page, and it says, “Read this in Markdown,” and literally they've got the whole website in Markdown. At one point, I need to double-check if they're still doing this, but Google had loads of their pages like this. Loads of their pages were available in Markdown.
People say, “They can't crawl them,” or “it was duplicate content," that sort of thing. But this has always been the case. It's always been the case that there have been Markdown pages on Google, and there have been different versions of pages on Google.
If you go into the deep depths of the internet, I looked at the website where they verify all of the protocols, where robots.txt got officially verified. That website, where it talks about them verifying robots.txt, which actually didn't happen that long ago, they put all of their pages in Markdown and HTML. You can see all of them, and Google only indexes the HTML. They don't index the Markdown. I think that's important to remember. Google has the tools. They can manage. They're good.
The other thing I would say is that I think that's a really interesting experiment for these kinds of sites; llms.txt might actually be a good place for that. People say, “Why does it have to be called this and not that?” One of the things is that there are naming conventions. We could call it robotics.txt, or we could call it robots.txt. If we call it robots.txt, then people know where it is. So I think that's fine. An llms.txt might be a good place to put that. Otherwise, if you're planning to feed it to LLMs in another way, then you'd have to tell them directly, specifically for that.
MCP as an Opportunity for Publishers and B2C Brands
Crystal Carter: Additionally, I think it's also worth thinking about MCP. MCP is a great tool for agentic optimization. I always include it in my list of things for agents because, with an MCP, you can make specific content available for users. You can also make it so that people have to log in to get access to that content.
For instance, there are some productivity tools that I use, and I have to log in before I can use them. I have to log into the tool with my Google login or whatever so that I can use those in my agentic stuff. There's no reason why you couldn't do the same thing with a public-facing publication. If I had a Financial Times subscription and I wanted to fire up my Financial Times MCP so I could find out what the latest news on one stock or another, then it would make sense to be able to do that. Maybe if I didn't have a paid subscription, then I could only get a certain amount of information. I think that's something to consider as well.
One of the things that I recommend in the book, and generally, is that where there are new features in LLMs, if they're opening up ChatGPT apps, custom GPTs, skills in Claude, or connectors with this, that, and the other, get your brand involved. Get your brand involved. I see a lot of B2B brands in the MCP space, but I don't see that many B2C brands. I think this is a real opportunity for new brands. I recently saw Tilly's, which is a retailer in the US, and they've got a B2C MCP. I think that's great. I think that's a really good opportunity, and I hope to see more in the future.
Gianluca Fiorelli: This is something that I also was preaching somehow. That's why at vBase, it's always been really working on creating your own business website knowledge graph in order to feed and inform the MCP.
Not only the MCP, but also, usually, the knowledge graph at the enterprise level is used just internally in order to understand what we have in terms of documentation, private documentation, website, customer care, subdomain, et cetera. It's never really used with external input in order to understand how to improve and update, for instance, the architecture of our website, and from there inform everything else, like creating an MCP for a personalized type of assistance or buying experience. All the things that usually are made with cookies, you can eventually do, and even better, I think, with an MCP installed on the front side.
Crystal Carter: Absolutely. I think people need to get really creative with the tools that are available to us, and I think AI unlocks a lot of that.
Personalized AI Search and Knowing Your Customer
Gianluca Fiorelli: This is obviously going to lead me to the next question, which is about personalization. You wrote an article about personalization and the collapse of universal SERPs, which sounds quite dramatic, but it's actually what we are seeing daily.
If search on every surface—I'm not just talking about Google—is going to be so firmly and strongly personalized, what kind of strategy or what kind of approach can be taken by a business? Obviously, we cannot think of being able to be visible at the micro level of each potential customer we have because it's impossible. But how can you create a strategy that makes you possibly visible to cohorts or potential buyer personas and customers?
Crystal Carter: Again, I cover this topic in the book as well. These are great questions, Gianluca, I must say. One of the things I say in the book, one of the rules, if you will, is to really, really know your customer. I think it's really important to take into account that the AI your potential customer uses knows them really well. My AI knows me really well. I wish it didn't, because I've asked it about my personality, and it came back with some things that I was like, “Me?” And they were like, “Yes.”
Gianluca Fiorelli: That is a kind of question that I always avoid.
Crystal Carter: I think it's really important to realize that they really know them. A really easy way to check this is to ask your AI, “What are the things you know about me based on your memory?”
If you really want an AI-forward one, go to one of the competitors of whichever AI tool you use. If you use Perplexity, go to ChatGPT. If you use ChatGPT, go to Gemini, whichever. There'll be an option to import memory, and they will give you a prompt that you can put into your AI. Then the AI will spit out all of the things that it knows about you.
This is fascinating. The thing that I find most fascinating about this for brands is that your brands will be in there. If you're an Apple user, Apple will be in there. Why? Because I'm going to ask for tech support. For whatever tools I use really regularly, I'm going to say, “How do I do that thing in Screaming Frog? Because I've looked for 15 minutes and I can't figure it out.” Or, “How do I do that thing on my Android phone?” Or, “How do I do this on my smart refrigerator?” Whatever it is.
So brands are always in memory. That means that you can personalize it for those kinds of folks. With personalization, you don't necessarily have to personalize it for every single person, although there are tactics for that. But you should have signals on your website that make it clear that you satisfy the needs of your target customers, and I mean explicitly clear.
A really good example of people who've done this really well for years is recipe bloggers. Somebody makes a chocolate cake recipe, and then underneath it they say, “Substitutions. If you're gluten-free, you can do this. If you don't eat eggs, you can do this. If you don't eat coconut, you can do this. If you don't have caffeine, you can do that.”
They'll put all of those different things there based on the questions they get from people. People are going, “But can I have it sugar-free? Can I have it with this thing?” So they'll have the substitutions in there. When you have that there, it means that when somebody's looking for something very specific for their specific needs, there's evidence on that website that you can satisfy that need.
Another good example, and I hope it's still on their website because people update their sites all the time, is Gymshark. They were really good for this. Gymshark has a review page, but the review page is really detailed on who the reviewer is. They'll say, “These shoes are great, five stars from Crystal Carter, who runs 20 miles a week and is five foot eight and is—we'll not get into how old—X years old.” I say five stars. Then somebody else will say two stars, and this person is a bodybuilder; they exercise six days a week, and they're this, this, and this. So there's plenty of evidence and plenty of context on the website that says that these things are for that.
If I go into an LLM or an agent and I say, “I'm a bodybuilder, and I want you to find me some good shoes,” they're not going to recommend that one because a bodybuilder said it wasn't good. But if I say, “I'm a runner, I run 20 miles a week, and I'm this year's age," they'll say, “Actually, that might be good for you based on this information.”
This is one of the reasons why they use Reddit a lot. Reddit has loads of context for all of their replies. They're all in a subreddit around that topic, and if it's not the right topic, they get booted out of the subreddit. Everybody has to say who they are, and everybody's replying with the right context. So give them signals. Give them the information that tells them that you are for who they are supporting.
Gianluca Fiorelli: I want to underline the fact that you were saying before, be very clear. Clear means that you have to clearly say the things. This is something that worked well also in travel for a client because we were saying, “We are targeting families, we are targeting young couples, and we are targeting senior couples with no children, no kids, finally able to travel again alone.” But obviously, we cannot create three different landing pages for every destination. So we started to create one landing page but clearly separated it, using even semantic blocks.
Crystal Carter: Yes.
Gianluca Fiorelli: In order to make it understood, if you are a family, our brand manager suggests you go here, here, here. If you are young, you surely want to experiment with something more adventurous. So maybe this specific destination in the Dominican Republic is more for you than for a family, and so on.
Doing so, we started to see that the same page was able to… It is clearly good also for classic search because this same landing page started to be visible for more queries. The query count was exploding, and also for a wider range of LLM chats.
Crystal Carter: Yes. That's how people search. People say, “My family and I want to go to Puerto Rico. Where's a good place to go?” They're going to say all of those things, and you have to have the evidence on your site to verify that.
Why Clarity and Simple SEO Fundamentals Still Win
Gianluca Fiorelli: At the end of the day, this is also making me think, when we were thinking of it, we were already overcomplicating. Is overcomplicating our work not a classic SEO problem?
Crystal Carter: Yes.
Gianluca Fiorelli: Because if we think, at the end of the day, what works usually is the simplest. I'm not saying the simplest solution in the cheap way, but the not-so-overcomplicated solution. Usually, that's the one that works.
Crystal Carter: Sometimes when I'm doing SEO, I feel like I'm… I don't know if you've ever heard the phrase Captain Obvious.
Gianluca Fiorelli: Yes.
Crystal Carter: Sometimes I feel like Captain Obvious, but normally it's the thing that you need. You're just like, “You don't have an H1.” And they're like, “What?” I'm like, “You don't have an H1 on your page. You don't have any H1 on your page. I recommend that you put an H1 on the page.” And then they're like, “It improved our SEO by 20%.” It's like, “Yes, because you told them what the page is. You're welcome.” I know that was not super complicated. That's SEO 101, but also nobody had done it.
Gianluca Fiorelli: Sometimes we have this feeling that if we are saying, “You don't have any H1 on this page,” it's making you feel dumb because it's a dumb mistake.
Crystal Carter: Right, but if nobody noticed it…
Gianluca Fiorelli: It's going on and going on and going on.
Crystal Carter: It's very simple, but I always like to think of these things as very smart, but also not that smart at the same time. I feel like Google's very smart, but sometimes you ask it a question, and it doesn't understand the basic question. I think LLMs are the same. So you have to give them the information and make sure it's super clear. Those context blocks that you're talking about are really, really important. Also, the context of where you're talking about it. Backlinks are really important, but where the backlinks are is really important.
Gianluca Fiorelli: The content surrounding the backlinks or the mention.
Crystal Carter: Exactly. One of the things I was talking about with Gymshark, Wix does this as well. If you look up what's a good website for a student, Wix will always show up because we've run a Wix for Students program for years. We have partnerships with Yale University and loads of other universities, and we've been doing it for years. We show up on Student Beans for discounts and stuff.
For your travel company, for instance, maybe they have one package that's like, “Get the family package,” and you put that on a family website, on Mumsnet or something like that. Maybe there's a senior one, and you put that on retirees.com or something like that. You're giving it context. You're putting it with context.
Think about that when you're thinking about the influencers, the affiliates, the blogs, the backlinks, and the publications where you're talking about your product as well. As you said, you can do one page, but just make sure that it's very clear who it's for on the page.
Explore the agentic web further with two more episodes of The Search Session:
Bastian Grimm discusses preparing businesses for AI agents through reliable data, machine-readable content, and practical experiments, as AI moves from answering questions to taking action.
Alex Moss explores agentic commerce, WebMCP, and how structured data and content governance help websites support AI agents.
Gianluca Fiorelli: One hour already, I think.
Crystal Carter: All righty.
Gianluca Fiorelli: It's maybe time to let you rest and let you come back to your work, or maybe you have a coffee break.
Crystal Carter: Maybe I'll go and read a book.
Gianluca Fiorelli: Or click on the link that we are going to put along with the description of the episode. I hope Amazon is replenishing the books in their stores because I've read on LinkedIn that there were only 15 copies left.
Crystal Carter: They sold out on Amazon. They're restocked now. I've been floored by the reception. I've had a lot of great feedback, and it's been really nice.
Gianluca Fiorelli: It's great to know. I'm always happy when a friend of mine is able to publish a book, also because I generally envy them because they found the time to write a book. It's something that I always have in the back of my mind. Really, like in the movie, it's a real back room, but the back room of a back room, the idea of myself eventually writing something like a book, but I never have.
I really envy you because you were able to do it, to pursue it, and to find the time to do it. So congratulations, Crystal. It was also a great pleasure to have you here at The Search Session.
Crystal Carter: It's been lovely. Thank you so much.
Gianluca Fiorelli: And thanks to you all too. Let me do the YouTuber. Remember to subscribe to the channel and turn on notifications so you know when a new episode comes out. Thank you and bye-bye.
Podcast Host
Gianluca Fiorelli
With almost 20 years of experience in web marketing, Gianluca Fiorelli is a Strategic and International SEO Consultant who helps businesses improve their visibility and performance on organic search. Gianluca collaborated with clients from various industries and regions, such as Glassdoor, Idealista, Rastreator.com, Outsystems, Chess.com, SIXT Ride, Vegetables by Bayer, Visit California, Gamepix, James Edition and many others.
A very active member of the SEO community, Gianluca daily shares his insights and best practices on SEO, content, Search marketing strategy and the evolution of Search on social media channels such as X, Bluesky and LinkedIn and through the blog on his website: IloveSEO.net.
