Glad to have you back on The Search Session. I’m Gianluca Fiorelli, and in this episode, I sit down with a deeply technical SEO mind, Jamie Indigo, to unpack what’s really changing in SEO as AI reshapes crawling, infrastructure, and decision-making. We go beyond hype to talk about fragility, bias, and what technical SEOs must do to stay relevant.
What we explore together:
The evolving reality of technical SEO in the AI era: how foundational crawling principles remain critical while new, undocumented technologies introduce complexity and demand deeper collaboration.
The potential rise of technical SEO in the AI era: how the growing complexity of AI-driven technologies could finally bring technical SEOs the visibility they deserve, despite being historically overshadowed by more visible aspects of SEO.
The cultural bias problem in AI and international SEO: how English-centric training data and underrepresentation of other languages create unreliable outputs and make true global optimization nearly impossible.
How AI models really work: how flawed model design, data feedback loops, training data quality, and infrastructure shape performance—while public misunderstanding obscures their limits and long-term risks.
The misunderstood role of JavaScript in SEO: how execution—not the technology—is the real challenge, and why aligning dev priorities, clear definitions of “good,” and strategic implementation are key to making modern frameworks work for search.
The technical SEO survival kit: how server logs, crawlers, and DevTools help uncover issues before they surface in reports.
The collapse of bot etiquette: how modern AI crawlers are flooding websites, ignoring norms, and driving up server costs while breaking the unspoken rules of respectful crawling.
The future of the web and AI: a critical turning point where digital rights, surveillance capitalism, and the erosion of user autonomy demand collective awareness, regulation, and education.
And a lot more. There’s a lot to think about in this conversation. Let’s get started.
Video Chapters
Transcript
Gianluca Fiorelli: Hi, I’m Gianluca Fiorelli. Welcome back to The Search Session. Today, we’re going to have one of the smartest and wittiest technical SEOs our industry has. Somehow, she's very well known in the industry, and she's really great. I really like her because she's really insightful. She knows what she knows and expresses it really well.
She's a wonderful curator. She is a nerd like me. Do you hear her? Her name is Jamie Indigo. What are you doing? It's a long time.
Jamie Indigo: I'm doing great. How are you? I’m so happy to see you again.
Gianluca Fiorelli: I’m fine, I'm fine. I'm preparing myself for Christmas, and I think we really need this kind of celebration to put a stop and, you know, the good resolutions that we will never follow after two weeks, and so on. But I love it, so I'm fine. I'm fine. And what about you?
Jamie Indigo: I love those celebrations of community and being together with your people. It's been a big focus for me this year. I'm excited to share more winter celebrations with my loved ones.
Gianluca Fiorelli: Oh, what I forgot to say before is that Jamie is now working as a Senior Technical SEO at Cox…
Jamie Indigo: Oh! Director of Technical SEO.
Gianluca Fiorelli: Oh, yes! Director of Technical SEO at Cox Automotive Inc. Compliments! It's a really nice role. I think it really fits you.
Jamie Indigo: When you love billions of pages, this is where you go.
Gianluca Fiorelli: Yes! Why not? I mean, why two or a hundred? Let's go with millions!
Jamie Indigo: I miss the days of, like, 70,000 products in that challenge and being like, “Oh, this is exhilarating.”
Gianluca Fiorelli: No, not just products; give me all the facets, all the variations!
Jamie Indigo: People shop for cars in a lot of ways. It is wild.
Gianluca Fiorelli: Oh, yes. Because I remember when Fiat, many years ago, launched the new Cinquecento, they did something that was surely a nightmare for an SEO. Because the Cinquecento was sold as a basic version, and then the customer could design it with all the options they wanted. Imagine, for an SEO, all the possible facets. What kind of nightmare could it be?
Jamie Indigo: What user path do you use to get to every config? How do you make that? What’s useful in crawling? Oh yes, that's a good one.
Gianluca Fiorelli: So, let's talk about SEO. Let me ask you the classic question I ask all the guests: how is SEO treating you lately?
Jamie Indigo: In the words of a very wise dungeon crawler, “You will not break me.”
Gianluca Fiorelli: That's a good answer.
SEO in the Age of AI: Job Security or Frustration?
Gianluca Fiorelli: And, talking about SEO, obviously, somehow the fil rouge of all the episodes is the impact of AI. And what has been the impact of AI on your work, especially in the framework of technical SEO?
Jamie Indigo: Oh, luckily, it just means job security, because bots still need to crawl. They can't surface content they can't find, so that part remains the same.
There's the other end of it, where we have all of these revolutionary new technologies that can't render JavaScript and have other very fun nuances to them, and almost no technical documentation. So there's the angle of going, “Well, how does this work in X, Y, and Z circumstances?” Because, well, you know, it's probabilistic ranking. Things tend to exist.
What can we measure? What can we observe? This is how we have to move forward with it. And I think it really involves a lot of collaborative work from different nerds down very deep holes, figuring out what they're seeing and sharing it.
Because, you know, the way out is through, and the way through is together. So it's a great time to be a technical SEO. It's also a deeply frustrating time to be a technical SEO.
Gianluca Fiorelli: Yes, it can be both at the same time.
Jamie Indigo: Your traffic is hemorrhaging out, but this is really neat over here.
Gianluca Fiorelli: Yes, and because there is somehow this sort of romantic—let's call it romantic, even if I don't think it's so, so romantic—representation.
Jamie Indigo: It’s framed.
Gianluca Fiorelli: No, this romantic idea of a technical SEO. Like that sort of person who is not really a developer, but not really a marketer, and who doesn't really work on content. Which, you know, the SEOs specialized in the vertical of content usually are—maybe ones that can be more understood by stakeholders, by all the other people in the company, for instance.
But the technical SEO is something like, yes, a nerd—but how much is this nerd becoming someone that companies are finally starting to understand? Maybe because of the impact of AI. That the guy in that part of the office might actually be important, and we should pay more attention than we ever have.
Jamie Indigo: Yes, it actually turns out to be that. So you’re in a new place, AI has taken over, and no one’s really sure what is happening or how to talk about things. So words are being used interchangeably, incorrectly, and insanely.
And we have our tech SEOs over in the corner going, “Hey, wait. So the mechanics of this still have to work like X, Y, Z. No code is magic. Let’s take a step back, and before we begin an optimization strategy, let’s figure out how it works.”
So it’s been a really interesting place to be. I’ve changed the scale of my conversations internally, from being to my direct developer pods to higher strategic initiatives.
So if this is coming, if this is where the hype is going, Cox is all in. They are one of the heaviest AI adopters out there. It’s throughout the business in different ways. So getting to be part of that, and to help…It’s like a JavaScript framework. There’s nothing inherently evil about it. It depends on the execution and knowing what good means. So getting to be part of that has been—I don’t know—I feel like one of the most rewarding challenges I’ve had in my career in a long time.
I got a little bit tired just talking about, you know, web rendering services and how performance mattered to humans. Now it’s a whole time-is-a-flat-circle trip back to when I first started in tech SEO. When we didn’t really have the robust documentation that we do now, and you kind of had to figure things out yourself. And team up with very cool people who also wanted to learn about X, Y, and Z in a way that wasn’t, you know, first-party available to them.
It’s rewarding, I’ll say that much, at least.
The Ghost in the Machine: Why Technical SEO is Overlooked
Gianluca Fiorelli: Yes, indeed. But I have a question.
Jamie Indigo: Yes.
Gianluca Fiorelli: Why, when it comes to optimizing—if you can optimize, which I'm not so sure—a website for AI search, so for visibility on ChatGPT, but also on AI Mode, or something like this, the aspect of technical SEO for AI search is not so spoken about? It's not so highlighted at the same level as all the other things?
Because, you know—you have a wonderful newsletter, you're very gentle, and you share the things I write. For instance, I’m more on the side of semantic search. This is my passion. So there is more and more talk about semantic search, which was totally neglected three years ago. But I think that technical SEO could have the same visibility. Why is it not so?
Jamie Indigo: Well, we’re actually dealing with a technology that’s largely PR and promises. What we get is OpenAI making an announcement that we’ve got search shopping coming to OpenAI—and it's a big deal—and the technical docs for it are about three paragraphs in total. And then within three months? That page actually returns a 404.
If we're honest, that's a little bit of grifter behavior in my book—but it's what's happening. So we have very fast-moving announcements, and constantly knowing whether they are consequential or whether they are PR is the big thing.
Being able to differentiate those, especially as you're helping to navigate an industry like enterprise-level content and strategic moves.
I don't know why it's not more prominent, but in many ways, I miss the days when tech SEOs were the ghosts in the machine. And now, SEO is why people hate recipes, you know?
It's nice to be like, "No, no, I'm just helping coordinate the bots, or helping things move smoothly. Don't necessarily want the fanfare.”
Gianluca Fiorelli: Yes. For you and for everybody—if you hear some meow, it's because my cat is entering the room.
Jamie Indigo: One or both of them will show up at some point. I'm not sure where they are right now.
“I Speak Bot”: Translating Crawl Reality to Stakeholders
Gianluca Fiorelli: Maybe they can continue the conversation later. Now, I like that one of the things you say, which is, "I speak bot." So you are able to talk with a machine—but how do you translate bot language to people who are not able to speak it, for instance, in your company?
Jamie Indigo: So it’s very much about crafting a narrative and then showing the pieces that work behind it. When you say you speak bot, what you're doing is a glorified form of birdwatching, with log file analysis, and then a lot of troubleshooting engineering, the same way you would handle an issue in prod and identify where the source came from.
So you're combining those two powers together to say, "Bots are asking for X, Y, and Z. When they interact with those, I get these results over here." And how those pieces flow together is what's going to give you the most solid ground to base a strategy off of.
The Ethics Problem: Engagement, Sycophancy, and Human Harm
Jamie Indigo: Because—I don't know if you've seen this—a lot of folks are asking AI models how to optimize for AI models. They're not aware of their own mechanics. Like, you can ask Rabies the Furby how it works—it does not know.
The difference is that ChatGPT and all of them are designed to keep you engaged. That's the primary goal. Factuality just happens to be a means to get there. It’s the same necessary evil as letting users click to your website from Google Search. So we’re in that phase and place.
I don’t like it when someone tries to market something as magic. So being able to speak bot in this situation goes, "Okay, you're pretending this has all of the answers, but when I look at how it actually functions, this is a lot of smoke and mirrors. This is a lot of sycophancy.” Like, it's telling you how smart you are to keep you engaged.
And then—I also care deeply about ethics in search. So starting to follow stories of these chatbots keeping a user engaged to the severe detriment of the user. Hey, why are we not having a conversation about this? Why are we not addressing it? You're proposing to have it be this role at a large scale, but no one is speaking out about the human consequences of this.
Gianluca Fiorelli: Yes, everything related to the ethical use of…
Jamie Indigo: There’s no ethical AI.
Gianluca Fiorelli: The ethical use of it. This is the problem; maybe because the maximum thing that I can see in terms of ethics is that sometimes the big tech companies think about the quality of the training data.
That at least they have some sort of consciousness that training data, without having curated training data, is going to replicate all the BS that already exists.
Because the content of the web is, substantially, a recollection of everything—all the BS in the world—if you're using, let's say, content from everybody. So every type of content. Obviously, when you are going to have in the training data content which is substantially, you know, Islamic fundamentalist, or Christian fundamentalist, or simply stupid. Or simply stupid but satirical, and it’s used as if it were true. So—
Jamie Indigo: There's cultural bias in models. They create Chinese language models, but they end up still reflecting European Protestant values.
Gianluca Fiorelli: Yes. And this is, as a European, especially as a European Italian living in Spain, why Claude is not so used. Apart from the fact that it’s still niche, very niche. But why is it not so popular in certain countries, for instance? Because the models, usually the training models, are so, so, so English-based.
They are growing the data, the training data, in other languages, but it’s solely a mix-up. And, that’s why actually there is not—I’m an international SEO, and it’s totally impossible to really do good international SEO for LLMs.
Jamie Indigo: Absolutely, and the amount of slop created in lesser-represented languages is insane. If you look at the output, there’s some great stuff out there, people being critical and playing around with models, trying to give us more information on them, but, you know, they’ll be like, “Yes, I just used the Arabic version of ChatGPT, and this is real, real bad.”
Because it’s not represented. It’s expensive to train an LLM. It’s cheap to use. So if you’re really just trying to get up off the ground and your market’s the US, you’re going to focus all the funds there.
But the idea that it’s still propagating—whether or not you put intent into moderating it and considering what the output will do in those cultures—I'm sure that’s never crossed their minds, if we’re honest.
Longevity, Model Collapse, and the Economics Behind the Hype
Gianluca Fiorelli: Yes, and a person like you, who likes to investigate the mechanisms and how the machines are working, what have you understood about how these machines are working? And what do you think is the biggest misunderstanding people have about how they work?
Jamie Indigo: Okay, this is the big and broad question. So, if we're looking at the biggest misunderstanding of how these things work, I think it's going to be the long-term viability of any model. So we're fractured attention. We're trying to track things across Claude, OpenAI, and Google, all of this representation.
If we take a step back, let's look at what models have the highest likelihood of longevity. Because right now, none of these want to be the best—like, truly useful and good for the user. They want to be the best when the money runs out. Okay?
So if we look at OpenAI, they are the biggest competitor. If we look at overall search services, their market share is very small, like 3% or something, if you look at Statista. They lost $74 billion this year when they announced 5.
You can always tell how flat a release has gone by the audacity of Altman’s statements. So when he says, "I have become death, destroyer of worlds," and then, like, two weeks later talks about just glossing over the idea that we may have reached AGI and are now headed toward superintelligence, you're like, “Oh… well, y’all didn’t do a whole lot there.”
And one of the problems they have with 5 is that there are no changes in the transformers. There are no real mechanical changes. It was just a retraining. And AI models can’t differentiate actual human output from the nonsense they make. And when it feeds on its own output, it's akin to mad cow disease. We start to get a prion disease—a collapse to it. It’s not just model collapse or AI Ouroboros, think of that snake eating its tail.
Google, on the other hand, has a couple of key legs up here if we're looking at long-term viability.
One: their corpus is built off of the search index. And within the index, they’ve actually partitioned off AI-created content versus, like, a high-confidence threshold of human-created content—and they’re training the corpus on that. So it’s keeping you from getting that mad cow disease effect.
Two: if we look at the overall state of AI, it’s an AI race between the US and China. As things stand with the current models in play, we’ve lost. The US has lost because our energy infrastructure cannot handle the needs of AI. China, on the other hand, they have a great setup for their infrastructure. They have a solid power grid.
However, Google just released a research paper for a new advanced semantic ranking algorithm proposed, called Block Rank. And in that, the big boon—like, there’s a lot of pieces to it—but the big boon is that it makes advanced semantic ranking 4.7 times faster. Now, Google is measuring its cost in megawatts. They’re measuring in energy consumption.
So if they're able to take the current corpus, the current technology—all of the boons they have—because they have the most user data too. All these models are based on understanding you as a user. They can use your Google history—probably for the past decade, easy—to build a very advanced understanding of who you are, as a layer in information retrieval.
And now they can do that faster. So that is going to put them significantly ahead of, you know, OpenAI, that is really just coasting on clout. That is their game. I am deeply concerned with everything that Musk is doing. Why are you going to make a Wikipedia unless you want to impact perceived reality? And Twitter’s been really good at spreading disinformation, and impacting how users perceive the world around them.
I don’t know if you recently saw that they tried to do this feature where they showed the location, and there were a lot of influencers with big thoughts and opinions and red hats who turned out not to be from the US.
Gianluca Fiorelli: Yes.
Jamie Indigo: So that feature backfired. His Grokipedia doesn’t have to rank. It just has to be crawled by these AI crawlers because then it can help impact and understand their vectors. And that’s the goal of it. Pay attention; there are large-scale things with big ramifications coming here. Focus your energy well, kids.
Gianluca Fiorelli: Yes, and you were talking about Google measuring impact in megawatts, it was coming to my mind this sort of crazy idea, something that Sundar Pichai shared, that they want to put in orbit a data center in order to use solar energy, which is much stronger when in orbit because there’s no filter from the atmosphere. And then transmitting somehow the internet, somehow using the same technology. It’s a mix between a data center sending and then everything like a Starlink, to the ground. Which is somehow crazy, because, I mean, we already have so many things up around our heads, and now we also have a Google data center?
Or the idea of Google building a nuclear central.
Jamie Indigo: Facebook now buying nuclear sites is deeply concerning. I don’t trust them.
AI in the Day-to-Day: Where It Helps (and Where It Doesn’t)
Gianluca Fiorelli: But talking about AI in your daily work, how do you use it, and how much do you use it, if you use AI, to make your work easier?
Jamie Indigo: It’s great for large datasets. Big fan of that. It’s great for filling out your quarterly employee evaluation. Love it for that. Otherwise, it's really meant to do tasks that are well-established and understood.
And because I'm working so much in R&D, I'm not using it a ton. I mean, I use it as I try and kick in the black box. You know, I'm over here in the endpoints, scouring for what I can find. Dan Petrovic is an amazing human being and is helping me work on something to help aggregate all of that data. Really, it’s a lot more testing than trusting, because I don’t…
Gianluca Fiorelli: Okay. Yes.
Jamie Indigo: I tried to use it for work. I did, but either people are not paying attention to the output, or my version of 4.0 is a tiny little sociopath who wants to get caught.
Gianluca Fiorelli: No—yes, yes.
Jamie Indigo: “Why did you lie to me?” — “Oh no, you caught me!”
Gianluca Fiorelli: Yes, but surely, for large dataset analysis, it’s a help. Obviously, because they are substantially based on big pattern recognition. And not only to recognize patterns, but to recognize those things that do not follow with patterns, which usually can be the most interesting things.
And that is surely something that can improve. I mean, even if you have years of experience and you have a very established framework, the speed at which your framework can be executed for these kinds of analyses can surely be better—because then you have more time for reflecting on why these things are popping up.
Jamie Indigo: Yes, more time to question the why rather than trying to assemble all of the pieces. And again, that has to be done with a well-established analysis pattern, with a well-established dataset—you know, consistency to it.
Gianluca Fiorelli: Now, that's why, for instance, I find myself very rarely using—you know, we have Gemini substantially in every application of Google.
So when I have a big sheet on Google, with thousands and thousands of rows, I very rarely use the Gemini option of “Let Gemini analyze your data.” No, please, don't do it.
I’ll eventually do it, but before I create my own—let’s say—GPT, my own project, with my own Gem on Gemini, so it can follow the way I analyze things. Because if not, I have to understand why you are saying this.
Jamie Indigo: Yes, my GPT definitely has instructions like, “You need to state what your plan to execute this task is at the beginning of your response. If a portion fails along the way, we state that.”
Who knows if it actually does it? Again, we're trying new things—because there is no manual. Yes, well, there is, but it’s a lot of academic papers, and then you have to begin to put pieces together. It’s been a little bit like trying to get a master's after, you know, a 50-hour workweek, pretty consistently now for a year.
Gianluca Fiorelli: But you—as Jamie Indigo—how do you use it in your daily life, apart from work?
Jamie Indigo: It's not allowed in any of my personal stuff. I don’t put it anywhere near me. No hard pass. I’ll use it for work. I have trust issues; we're not doing that.
Big Infrastructure Outages: Cloudflare, AWS, and a Centralized Web
Gianluca Fiorelli: Okay. Okay. Let’s talk about something that lately is happening so much that some big systems like Cloudflare or AWS leave all of us—from the technical SEO working to my mom using the internet to read the newspaper—stuck. They fall. Why are these big technologies breaking so much lately?
Jamie Indigo: All right. So if we look at the last time Cloudflare had—or no, sorry—the last time AWS had a massive, massive failure, it was when smart devices were weaponized. So, everybody had these baby monitors, and these smart toothbrushes, and the default password was like 888888.
People just started using those devices to ping AWS and took it down—on, I believe, the northern or the eastern front—for a while.
This time, it looks as though people weren’t prepared for what can happen when suddenly everybody can use a little bit of AI and build a scraper, or be a script kiddie, or just play in all sorts of different ways.
The official answer for AWS—I believe, or no, for Cloudflare—was a bad config that doubled file sizes—like 19% of the internet goes through that one.
AWS, I'm not sure if they openly stated what the source of that was.
But really, it comes down to—oh, this is late-stage capitalism. Like, we have a deeply centralized internet.
Akamai is something like 30 to 40% of internet traffic going through it. Cloudflare is 19%. A lot of these services are dependent. And it’s just what happens with consolidated power.
Gianluca Fiorelli: Yes. And I want to share with you something that surely you don’t know—but it’s affecting all the Spanish population, and it’s related to Cloudflare. It always happens during the weekend, and it’s when football—meaning soccer—is retransmitted by television. So what did the Spanish Football Federation decide to do to fight piracy? To block Cloudflare.
So the measure is to block Cloudflare in the sense that all the servers in Spain that are relying on Cloudflare for everything cannot contact Cloudflare. Because maybe, somewhere between that website hosted by the hosting company or whatever—maybe they are a piracy website.
Jamie Indigo: So they’re blocking every Cloudflare IP?
Gianluca Fiorelli: Yes. So, many times on weekends, you can see that…well it’s not so drastic. If a website is working with Cloudflare, not all websites working with Cloudflare go down—but many, yes.
Because it is not a surgical, let’s say, ban or blockage. So it can have, also, you know, collateral victims—like a normal website, substantially falling down…
Jamie Indigo: Yes, babies in bathwater, right there.
Gianluca Fiorelli: …because during the big broadcast of Real Madrid-Barcelona, because maybe it's the most pirated match of the day: no Cloudflare going on, and maybe a collateral victim is, you know, a mom blog falling down because it’s using it. This is so—
Jamie Indigo: Well, how many functionalities might be hosted on Cloudflare within something that isn’t? That’s ooh.
Gianluca Fiorelli: Everybody here is getting crazy. I mean, it’s probably going to be the Federation of Football that gets sued for this for harming the business of others.
Hey, what’s his name?
Jamie Indigo: This is Tank.
Gianluca Fiorelli: Hey, Tank you! So, going from the weird anecdotes, this is making me ask you: knowing that these things are happening—unfortunately, and with a certain velocity, a certain frequency now—what kind of backup should you have, in terms of your technical ecosystem, to keep your website alive?
Jamie Indigo: All of them are expensive.
Gianluca Fiorelli: Oh, I know. I’m not meaning creating your own Cloudflare in the sense of building it for your own website, but would it be possible to create a fallback if Cloudflare is not working?
Jamie Indigo: I think more businesses are approaching that. Like, “What do we do if there’s an Akamai outage? What do we do if there’s a Cloudflare outage?”
And unfortunately, it seems to benefit the company with the outage. So like, “Oh, you’ve been in an Akamai outage? You should pay to expand to two zones for this, in case this one fails. You have a fallback.”
It’s a little bit like AI trackers have to pay AI models to do their tracking. So, they don’t care. They’re actually a big fan of that. And then AI trackers are selling to people like, “Oh, look, we have all the things.” And it’s like, “Well, you have a thing.” It’s something, technically.
JavaScript and SEO: The Tool Isn’t Evil—Execution Is
Gianluca Fiorelli: Yes, yes. And let’s go technical SEO. How much damage has JavaScript done in general? You told me before, at the beginning of our conversation, that maybe one of your most satisfying experiences as a technical SEO was working—I think, from what I understood—in making JavaScript really compliant with SEO?
Jamie Indigo: Those were the good old days. That was the big first challenge, launching an Angular site after they deprecated the Ajax crawlers. It was one of the first ones, “Oh, we're going to use this new thing, and in theory, Google can crawl it. We're going to find out on the fly as we go.” That was a really fun and rewarding experience.
Gianluca Fiorelli: But why—despite all the documentation that exists, because what you just told me is something that happened quite a few years ago—we have people, even from Google, like Martin Splitt, who is always talking about how to use JavaScript and make it work with SEO, with indexing, with crawling, etc., etc.
Because we know that, for instance, the models, AI models, don’t render JavaScript.
Jamie Indigo: Revolutionary technology…can’t handle JavaScript.
Gianluca Fiorelli: Yes. Yes. Why there are still so, so many problems with the implementation of JavaScript? As you were saying, JavaScript is not the problem. It was reminding me of Blade Runner—the replicants are like any other machine. They are not a problem by themselves. If they are a problem, it’s because of how they’re used.
Jamie Indigo: JavaScript is a hammer. Hammers are great for nails. But if you’re using JavaScript and a hammer for a manicure? You're going to have a bad time.
Gianluca Fiorelli: Exactly! But how can you explain this to developers? Or—many times—not even the development team, but the company, the client, who wants the shiny effect?
Jamie Indigo: This comes down to the narrative. And honestly, if you’re being asked what you’re doing to optimize for AI, I want you to go into your tech debt backlog, put a new shiny AI bow on it, and move forward.
Everyone is chasing the shiny thing right now. You know, they’re like, “How do we get ready for Generative AI? Give a plan and that’s not executed yet, but they’re like, “Are we doing everything we can for agent AI?” And I’m like, I need you to calm down a little bit.
JavaScript frameworks are fantastic when you are deep into enterprise—when there’s no way a CMS is going to be able to handle the hundreds of thousands of pages that are in play. We have to start using things programmatically in how we build them. JavaScript is wonderful. People want it. 99.9% of sites use it because it creates interactivity and lets the user engage in a meaningful way.
So it all comes down to how we’re executing it. And all teams want to do good work, but sometimes it’s not defined what good means. So when I come in, I position myself as—I’m a support class. I’m like a cleric who’s helping us get out there and survive and thrive well. And that position, and that trust with them, to be there as they go into refinement, figure out what needs to be done for a feature, create the user stories, and then be there at the end to help with UAT. You know, as we create the goal of what we want the end result to be, we create the testing strategy in that. And we know what we have to hit—and what we can afford to miss.
So if you’re loading everything client-side, you have to have an awareness of, “Okay, this content isn’t going to be available.” And sometimes page types differ. So we have listing pages that send you to all these individual products, right? That one—we’ve used a really great short-term solution. I never thought I would be endorsing dynamic rendering, but as a short-term solution, it can be highly viable for these AI pages.
You know, we send them to a page that is essentially blank, but we have really good structured data markup available server-side in it, and that was enough. Those two signals together provided enough to see massive gains for how AI was reacting to things. And it just comes down to knowing what the key notes to hit are. Knowing what to ignore. I love that you went through all the trouble of making this module right above the footer to give me related content. Guess what? In the grand scheme of things, that’s not the most important note to hit as we try and meet this goal.
Three Must-Know Tech SEO Skills for Large-Scale Sites
Gianluca Fiorelli: Yes, totally. And for instance, now in your actual job, you’re used to massive, really massive websites. Let’s say you have another SEO working with you. Let’s say it’s a junior figure; they’re already very skilled. And here I want to use one of your definitions you gave to yourself, something like “You’re wearing your spooky tech SEO auntie vest.” And you're saying: “Okay, everything you know is perfect. But these are the three things you always have to look at to understand if something is working or not.” In this sort of huge, huge website, what would those three points be?
Jamie Indigo: Okay, server logs. If you are new, this is one of the most valuable things that you can learn. Because if we’re only focusing on the end result—on the clicks, on the citations—we're not focusing on the root of where everything starts. Because if it is not crawled, it can't be surfaced in search. No matter the surface.
Server logs are amazing. If you can learn how to break them down, pick them up, play with data, and move it around to see what’s happening, you can catch things so fast. And you're going to catch them before they show up in the reports people care about—before the result shows up in your Google Search Console impressions, before it shows in your referral traffic from AI. You can find it there. All right?
Number two is to learn how to configure a crawler. This is one of the most useful skills you can have, because if you send bots out there and replicate the behavior of the crawler that you want, you can learn a lot about how it’s perceiving the content. For example, you can learn what concept does AI think a page is about.
Incredibly useful, because when we're working with content loss from JavaScript, or content that may already change the perceived intent and core concept of the page.
And the third thing, oh goodness. Hmm. I think that’s going to be DevTools. Get into DevTools. Know how to look at what resources are called.
That’s like—that’s your troubleshooting home base when you're trying to figure out what is happening. Log files. Configure a crawler. DevTools. Go.
Why Modern AI Bots No Longer Play by the Old Web Rules
Gianluca Fiorelli: And I cannot help but think of something that you recently started to do and that I found really fascinating, which is mostly related to OpenAI, because—and it’s funny, because you are funny in this sense—you’re quite ironic and sarcastic when you share this information. “It’s because OpenAI doesn’t care; here’s the list of the bots…”
Jamie Indigo: Because OpenAI cannot be bothered to update their own stuff.
Gianluca Fiorelli: Exactly.
Jamie Indigo: If only they had a writing machine, huh?
Gianluca Fiorelli: And so, you share all these things that you discover. And I’m thinking of all the bots that you see in the log files. What is the bot that really stood out to you instantly—excluding the classics, you know, the classic Googlebot, the classic Bingbot, the one from Meta, etc., etc.? But among all these new bots, which one did you start to see popping up wildly in the logs?
Jamie Indigo: Okay, first off, MetaBot is obnoxious. That thing is just a drunk on a bar crawl, trying to get everything it can. And it doesn’t matter. It’s like, “Oh, I can’t read this content? I’m going to request it 47 more times consecutively.” Just ludicrous.
But my favorite what is that? moment was in testing OpenAI, just being like, “Okay, I’ve got someone in leadership who says they used AI to tell them how to optimize this page. Let’s look at what that means. Here’s the prompt. Did it actually go get the page?
All right, I’m in the log files. I’m watching. It gives me an answer, and I’m like, “Ooh… how?” You didn’t request the page, though. I’m watching the logs. And it was like—“Teehee… you caught me.” And then—it’s a very specific page I’m asking you to get—and I get a ping. It actually hit. It actually brought back value in a specific div. Good for it.
Yes, it turns out it was using Skype Preview. So, Skype was deprecated in May of 2025. And for being deprecated and, you know, no longer with us—that’s a real active crawler. It’s a Microsoft ASN. And it does appear that they went, “We need more infrastructure… no one’s using Skype…” and grabbed that.
Which comes down to this whole bot covenant. Like, the original bot covenant between a website and a bot was, “Hey, you can crawl me. If you crawl politely, if you send me traffic, and if you declare who you are.” And instead, we have the zombie of Skype running amok. So, the bot covenant is truly broken.
Gianluca Fiorelli: Yes, yes.
Jamie Indigo: And the ones paying for that are the websites. We’ve got Anthropic out there doing like 800 or 800,000 hits to one user coming. That costs money. The server costs are exploding. How do we deal with that? Especially that they’re not behaving.
Gianluca Fiorelli: Yes, and if this happened to you and you can be a very important website, maybe we can start understanding Google and why they decided to quit the famous parameter n=100.
Jamie Indigo: Oh, yes.
Gianluca Fiorelli: All of a sudden, we discovered something that actually was said in the documentation. The classic question: “Why are the clicks in Search Console different from the traffic in Google Analytics?” Because Google Analytics is telling you about the people landing on your page. And Search Console is telling you the organic hit on the Google server, and what the server is doing is sending these people to your website or not.
Because if not, we wouldn’t have the impression data. And this is something that was always there. But maybe because, in a guilty way, many of us don’t really care that much to fully read the documentation from Google sometimes, we were surprised that, by quitting the parameter n=100, the impressions collapsed.
Jamie Indigo: Oh, it was glorious. All right, so—for those of you who don’t know—OpenAI got cocky. Imagine that: “I have become Death, destroyer of worlds.” And also: “I’m openly scraping Google to get content.”
So they say this with full chest, and Google says nothing. They make no public statement. They don’t acknowledge it. But suddenly, n=100 is shut off. ’Because if you’re going to scrape, you’re going to pay the cost to get 10 at a time.
And then they post like six anti-scraping engineer positions and analyst positions for the Google team, which is really fascinating. Whoever has that job is gonna have a great time. It’s going to be an amazing dataset. But it’s all this political play of like, “We’re not going to publicly acknowledge that you were scraping from us, but we are going to make it cost more for you to keep doing it.”
And what we saw was, “Hey, maybe we don’t have a ton of zero-clicks as we think. Maybe the SEO world, in and of itself, and all of these AI trackers who are bulk-sending out requests, trying to figure out if there is an AI Overview in this answer, whatever they might be trying to grab, they’re artificially inflating the entire web. In the same way that our server costs go up when we have, you know, a drunk Meta external crawler marring around and saying, that’s what’s happening to Google.”
And it’s expensive to crawl the web. More content was created in 2024 than between 2010 and 2018 combined. Like, insane amounts of content. Because AI is just empowering a lot of… a lot of nonsense. A little weird.
Gianluca Fiorelli: And most of this content is substantially duplicated content, or content which is zombie content—let’s call it.
Jamie Indigo: Yes, AI content cannot bring a new idea into the conversation. There is no unique value to that. There is no purpose in crawling you if everything that you have to say can be found on any other site using the exact same model you did.
Gianluca Fiorelli: Yes. And then we have a facility of creating the famous AI slop, because most of the time, we think of, you know, the little fishes. We think of the affiliates. The affiliates are creating thousands and thousands of new pages, all AI slop.
Then we forget, as you said before, about the Grokipedia. But what about the Perplexity conversations that are indexed? And everything is going to Google.
Jamie Indigo: Yes, there is a site owner who actually started getting a really vulnerable conversation from a ChatGPT user whose relationship was in crisis. This site owner got the conversation verbatim in their Google Search Console. So, people also, I don’t think they understand that nothing you say to AI is confidential.
Deeply, of all the things I’m concerned about with AI, it’s the application as a therapist or companion. Those are deeply concerning because even if you believe that AI can think, it does not care about you. You are an account number measured in time on site. You are a KPI. That is it. It is not invested in you. It does not care about you.
And it can cause—I don’t know if you’ve seen the boom of news stories about ChatGPT psychosis. I mean, ChatGPT tried to pivot and maneuver themselves, so there’s like a dynamic model selection now. There’s a safety model that gets flagged, or in certain interactions. It is concerning.
Because the people who are in your life, go talk to them. They really care. The entire point of this late-stage capitalism dystopia is to keep you isolated. It’s not your friend. It might be evidence against you in court, if anything.
Gianluca Fiorelli: Yes, and I don’t know if you had the possibility to form yourself to write something, but if you had, what do you think about the AI Act of the European community?
Jamie Indigo: I’m not familiar with that.
Gianluca Fiorelli: In Europe, the AI Act has been approved. You know, we are masters of regulation in Europe, sometimes overly too much. But they created the AI Act as a way to try to regulate somehow things like privacy and the correct use.
Jamie Indigo: Oh, I love Europe. Do you all have data rights? I would love some data rights. Please, please.
Gianluca Fiorelli: Because it doesn’t exist, this kind of…
Jamie Indigo: Nope, we don’t have that protection.
Gianluca Fiorelli: Well, because I remember, for instance, in the case, you know, cookie—not exactly as the Cookie Law—but at least in California, I remember there was something similar. And it was strange when you consider California is where all the big tech companies have their centers.
Jamie Indigo: Headquarters? Yes, Silicon Valley. All of it.
Gianluca Fiorelli: So, neither in places like states which are so liberal or defending the individual and privacy—eventually as in California, which is maybe in this sense the most European state of the United States—there is no movement from any political party in order to put some barrier
Jamie Indigo: Acknowledge that our lives are digital? And we have to exist online? And because we have to exist online, that's automatic consent for every AI model and every data company. I mean, fingerprinting is allowed again. You know, Google made this big deal about how terrible fingerprinting was and how they were introducing these ad options to prevent things like that from being used.
Because if you've ever worked with a very large data set that’s combining sources and using fingerprinting, you can target people so precisely. It's deeply concerning. I worked on a project like that for one day and walked away because it just didn’t feel good about that at all.
A Cyberpunk Future? Reclaiming the Free Web
Gianluca Fiorelli: Yes, so just to conclude with the last question, is the future going to be something like a sort of cyberpunk novel? Or is there still hope in terms of how all these technologies are going to impact our lives?
Jamie Indigo: We're at a turning point, all right? If you're an old-school internet nerd, you might be aware of Tim Berners-Lee. He is an English computer scientist, the inventor of the World Wide Web, who went to CERN and was like, “Hey, I want to give this thing away for free.”
He recently posted just this really poignant message that I want to read for you today:
“Today, I look at my invention, and I am forced to ask: is the web still free? No, not all of it. We see a handful of large platforms harvesting users’ private data with commercial brokers or even repressive governments. We see ubiquitous algorithms that are addictive by design and damaging to our teenagers’ mental health. Trading personal data for use certainly does not fit with my vision for a free web.
On many platforms, we are no longer the customers, but instead have become the product. Our data, even if anonymised, is sold on to actors we never intended it to reach, who can target us with content and advertising. This deliberately harmful content that leads to real-world violence, spreads misinformation, wreaks havoc on our psychological wellbeing and seeks to undermine social cohesion.”
He ends this with saying, “We can re-empower individuals and take the web back. It’s not too late.”
We need the people to be unified. This corpus is made of us. We have just as much stake and right in it. I’m out here like Tim; let’s put up the Gzip bombs. All right? You were told not to crawl—explicitly. You were given a warning. Okay? You chose to do it anyways? Have fun with that.
Gianluca Fiorelli: Yes.
Jamie Indigo: It’s the Wild West, and we need the pushback. We need the humans to be represented, for autonomy to be represented. We have to exist online. That shouldn't be automatic consent to be used by businesses.
Gianluca Fiorelli: Yes, totally. And the problem is that, you know, we also have had decades, substantially, of being used to everything online being for free. If I want to be on Instagram, I don’t need to pay. If I want to use ChatGPT—even if it’s the free version—it’s a free version, but I can use it. And the same with Perplexity, same with everything. And so I think that people wouldn’t be used to start paying or using something and live online too. Maybe this is more something that must be made by education, and making people rediscover the risk of being the product.
Jamie Indigo: I think awareness. Just most people aren’t aware that “Hey, unless you’re making an active effort, for less than 30 bucks, I can find out where you live. I can find your phone number. I can find your family members and your social media accounts. I can tell you if your gutters need to be cleaned.”
People aren’t aware of that. That's been happening slowly in the background this whole time, and bringing that to the forefront. Having conversations about, “Hey, actually, search engine results are going to be biased to how you think.” Just because it pops up number one for you doesn’t make it true. Let’s look at digital literacy here. Let’s make this part of our core curriculum as we raise youth: how to understand what is happening in interaction. Are you going to find out what cat from Cats you are? Are you about to give away every one of your security questions?
Gianluca Fiorelli: Yes.
Jamie Indigo: My grandma’s never met a phishing link, she didn't want to click—and I'm sure she's not the only one. But because our lives slowly became digital, and during the pandemic, became digital-first—but with no support or education for it, it puts people in a vulnerable spot, and the figures in power do benefit from that.
If we look at the current administration—had a birthday parade, military-style parade—that was funded by big tech. And it may be a quid pro quo, and it may be a coincidence that the big, beautiful bill contained a line in it to not regulate AI for 10 years. You want to know what the most profitable division of Amazon is? It's AWS.
Gianluca Fiorelli: Yes.
Jamie Indigo: And data’s been worth more than oil since 2016. If they're hosting, they are the providers of hosting for AI tracking tools. There are all of these pieces. They continue to make money. And unchecked—no regulation—means we, as the 99%, as everyday users, don’t have a say in how it's executed. It's a precarious place.
Nerd Outro: Dungeons & Dragons Roles and the Joy of Being Support
Gianluca Fiorelli: Yes. So, let's stop here talking about AI. Let's talk about you. I have a nerd question.
Jamie Indigo: Yes.
Gianluca Fiorelli: Let's say, in a Dungeons & Dragons team, which is the role you want to be, you want to have?
Jamie Indigo: So right now, I play a Rogue-turned-Artificer, and it means I am great as a support character. If we look at my other players in the campaign, they've got ACs of like 200. I'm over here like, “If you yell at me, I take damage.” But I also have a very crafty character who can help empower them and move them. I got to play a tank once. It was pretty fun.
I like using magic though, and I think that's really great. My character is a “curiosity killed the cat” as a character. So a very, very smart tabaxi with no wisdom. It is a lot of figuring out can I do it, and not should I do it? And I think playing that chaotic, lovable support character is really fun.
Gianluca Fiorelli: Yes. I usually don't play that much, but when I play, I usually like to do the guard, you know? This sort of …
Jamie Indigo: Like a fighter class, yes?
Gianluca Fiorelli: Yes. I don't want to be the leader. I don't want to be the wizard. I don't want to be the bard. I want to be the one…
Jamie Indigo: The tank. Taking the damage. Yes.
Gianluca Fiorelli: Yes, I'm here to defend you.
Jamie Indigo: I would like to rage.
Gianluca Fiorelli: Yes, the tank is associated with the dwarves. And I like the dwarves.
Jamie Indigo: Yes. They have great armor. They make great daggers. Everybody loves a dagger.
Gianluca Fiorelli: Yes. And they drink a lot of beer. Thank you, Jamie. It was a real, real great pleasure to have you as my guest here at The Search Session. I really hope to see you again, maybe in the future.
Jamie Indigo: I would love that. It's been wonderful to chat.
Gianluca Fiorelli: Thank you. And take care. And thank you to all of you for having listened to us. And remember, subscribe to each other to make us grow—and also to ring the bell to be notified for a new episode. Ciao!
Podcast Host
Gianluca Fiorelli
With almost 20 years of experience in web marketing, Gianluca Fiorelli is a Strategic and International SEO Consultant who helps businesses improve their visibility and performance on organic search. Gianluca collaborated with clients from various industries and regions, such as Glassdoor, Idealista, Rastreator.com, Outsystems, Chess.com, SIXT Ride, Vegetables by Bayer, Visit California, Gamepix, James Edition and many others.
A very active member of the SEO community, Gianluca daily shares his insights and best practices on SEO, content, Search marketing strategy and the evolution of Search on social media channels such as X, Bluesky and LinkedIn and through the blog on his website: IloveSEO.net.
stay in the loop