How machines see: the anatomy of a visual search
Everything in this guide follows from one distinction: what a system extracts from your pixels, and what it reads from the text layer you built around them. These are two separate pipelines. They fail independently, they are consumed by different surfaces, and optimizing one does nothing for the other.
Most image SEO advice collapses them. It treats alt text as though it were a description of the image to the machine, when for a growing share of visual searches the machine has already looked at the picture and formed its own view. Conversely, plenty of teams now assume that because models can "see," the text layer no longer matters, which is wrong in the specific case that matters most commercially: retrieval.
This section explains both pipelines, then explains when each one is in play.
Embeddings, object detection, and the salient object
When a visual search system ingests an image, the first thing it produces is not a word. It is a vector, aka a long list of numbers, typically several hundred or a few thousand of them, positioning that image in a high-dimensional space. This is an embedding.
The useful analogy is a map reference. On a map, two points close together are geographically near each other. In an embedding space, two images close together are semantically near each other: a rattan pendant lamp shot in a Barcelona showroom and the same style of lamp shot in a Copenhagen apartment land near one another, despite sharing no filename, no alt text, no domain, and no pixel-level similarity in the ordinary sense.
This is why visual search can return products that share zero text with the query. There is no keyword matching happening. The retrieval is a nearest-neighbor lookup: find the vectors closest to this one. Pinterest, which has published more openly on this architecture than Google ever has, describes moving to unified visual embeddings, a single representation serving multiple visual-search products rather than a separate model per feature.
The second thing the system produces is a set of detected objects: bounding boxes drawn around distinct things in the frame, each with a class label and a confidence score.
Note what those confidence figures tell you. The chandelier is the salient object or the thing the system is most sure about, because it is well-lit, unoccluded, and clearly separated from its background. The chair, half out of frame at the lower left, scores 52%. Same photograph, same model, radically different legibility.
This is the first place where photography becomes a technical discipline rather than an aesthetic one. Object confidence is not a vanity metric. A low-confidence object is a weakly-anchored vector, and a weakly-anchored vector retrieves poorly.
Two further points matter for ecommerce:
Detection is multi-object, and secondary objects count. Google’s own description of its visual search fan-out technique explicitly covers "recognizing subtle details and secondary objects in addition to the primary subjects." In a styled interior shot, your lamp may be the hero — but the rug, the cushion and the side table are all being detected, indexed, and made retrievable. If you sell all four, that photograph is doing four jobs.
Labels are a separate output from objects. Detection tells the system where things are. Labelling tells it what the image is about as a whole.
OCR: the text inside your images
There is a third extraction pipeline that image SEO discussions routinely omit, and for ecommerce it is often the most valuable one: optical character recognition.
Any text rendered in the pixels — on packaging, on a label, on a spec panel, on a screenshot — is read and made available as text.
Look at what has been recovered there: the brand lockup, the product name, the contents breakdown, the manufacturer’s URL. None of that was in an alt attribute. None of it was in a paragraph tag. It was extracted from a photograph.
The implication is direct. Your packaging is machine-readable content, and photographing it legibly is an indexing decision, not a merchandising one. A product shot that crops out the spec panel, or renders it at a resolution where OCR fails, has thrown away structured product data that would otherwise have been free.
Myriam Jessier, writing on image SEO for multimodal AI (Search Engine Land, December 2025), puts concrete thresholds on this: character height of at least 30px and grayscale contrast of at least 40 for reliable extraction. Those are useful numbers to hand to a photographer, and they are more actionable than "use high-quality images."
Three parallel retrievals
Put the pipelines together and you can describe what actually happens when someone searches with a picture. Three retrievals run in parallel, and their results are blended:
Visual retrieval — nearest neighbors in embedding space. Finds things that look like the query, including things with no shared vocabulary.
Object retrieval — each detected object is treated as its own query and matched at scale against an object index. This is what powers "that chair, specifically" rather than "that room."
Annotation retrieval — the conventional text index. Labels, alt text, captions, surrounding copy, structured data. This is where classic image SEO lives, and it has not gone away.
The reason this model matters more than any individual tactic: you can win or lose on any of the three legs independently. An image with immaculate alt text and a badly-lit, cluttered composition will do fine on annotation retrieval and poorly on the other two. A stunning studio shot with an empty alt attribute and no structured data inverts the failure. The systems blend all three, so partial optimization produces partial visibility.
What multimodal models actually read
Now the part that most needs stating plainly, because it contradicts an assumption baked into a decade of image SEO practice.
When a user uploads an image directly to a frontier model, that model reads the pixels and ignores your metadata entirely. This is not inference — it is in the vendor documentation:
OpenAI states explicitly that GPT-4o/GPT-5 vision does not process file names or image metadata.
Anthropic states that Claude does not process image metadata.
Google’s Gemini documentation describes native image understanding including object detection with normalized bounding boxes and segmentation, hence operating on the image itself.
So: a shopper photographs a lamp and asks ChatGPT what it is. Your descriptive filename does nothing. Your alt text does nothing. Your IPTC fields do nothing. What determines whether your product is identified correctly is whether the object is legible, and whether comparable images of your product exist in the systems doing the matching.
The second mode is different. When a model is browsing or retrieving — answering from indexed web content rather than an uploaded file — the economics change. Fetching every image on every candidate page and running vision inference over it is expensive and slow. Text is cheap. It is therefore very likely that textual signals dominate in retrieval, and that vision is applied selectively.
I want to be precise about the epistemic status of that second claim: it is a practitioner inference, not vendor-confirmed fact. No provider has published how their retrieval stack weights image content against surrounding text. The reasoning is sound and the cost asymmetry is real but treat it as a well-supported hypothesis rather than documentation.
What follows from the pair of them is not ambiguous, though:
Optimize both layers, because you don’t control which mode the user is in.
The same product, in the same week, will be encountered by a shopper uploading a photo (pixels only), a shopper on an Images SERP (annotations plus pixels), and an AI answer assembled from retrieval (text-dominant, probably). A strategy that serves only one of those is a strategy with a hole in it.
Where image recognition still fails
Recognition has improved enormously, but it fails in patterned, predictable ways, and every one of those patterns is something you control.
Compression artifacts degrade the signal. This is worth restating in the terms the machine experiences: an over-compressed image does not become "slightly blurrier," it becomes noisier at exactly the scale the model samples. Aggressive lossy compression introduces information that is not in the original scene, and the model has no way to know which details are real. Jessier’s framing is that this pushes models toward hallucination, with the system fills gaps with plausible guesses. This is the sharpest available answer to the old performance-versus-quality tension, and section 8 returns to it.
Clutter suppresses confidence. A frame with many overlapping objects produces lower confidence scores across all of them. Disorder leads to confusion, confusion leads to weak detection, and weak detection leads to poor retrieval.
Unusual angles and occlusion break matching. A product photographed obliquely, partially hidden, or in unusual lighting produces an embedding some distance from the cluster where that product’s other images sit. It becomes, from the system’s perspective, a slightly different thing.
Small in-image text fails OCR silently. No error is raised. The text simply is not extracted, and you have no way to know from the outside.
None of these are edge cases, and none of them require better algorithms to fix. They require better inputs, which is why later on I will turn the whole of this into a photography brief that a product-content team can actually execute against.
Continue the guide
Guide Introduction ← Previous · Next → Google Images: How Image Search Works
Article by
Gianluca Fiorelli
With almost 20 years of experience in web marketing, Gianluca Fiorelli is a Strategic and International SEO Consultant who helps businesses improve their visibility and performance on organic search. Gianluca collaborated with clients from various industries and regions, such as Glassdoor, Idealista, Rastreator.com, Outsystems, Chess.com, SIXT Ride, Vegetables by Bayer, Visit California, Gamepix, James Edition and many others.
A very active member of the SEO community, Gianluca daily shares his insights and best practices on SEO, content, Search marketing strategy and the evolution of Search on social media channels such as X, Bluesky and LinkedIn and through the blog on his website: IloveSEO.net.
