Advice and answers from the Advanced Web Ranking Team
Search for articles
Where AWR Data Comes From
How AWR collects ranking data: live queries for your keywords, location-matched IPs, what gets stored, what doesn't, and where the rest comes from.
Every position you see in AWR is read off a live results page. We count the results from the top down, as deep as your project's search depth goes, and your site's position is simply where it falls in that count. Nothing is estimated and there is no database behind the values in your reports: for each keyword, on each update, AWR runs an actual search on the search engine you selected and reads the page that comes back.
This article puts the whole method in one place. It covers where the queries are sent from, how your location and other search engine settings change what comes back, where each number in your reports actually comes from, and why a tracked position may differ from what you see in your own browser in 2026. If you have ever had to explain your ranking data to a client or a colleague, this is the page to send them.
The short version
AWR data comes from live queries only. Every position is read from a search engine results page collected at the moment the ranking update takes place, not pulled from a stored index of rankings.
Each query emulates a manual search, unpersonalized, the same way you would run it in an incognito window, with the search settings you chose for that search engine: location, device, language, and tracked results type.
Location is handled with IP ranges matched to your setting. A country-wide search engine uses a broad pool of addresses across that country. A custom location uses a narrow pool close to the place you track.
What AWR shows you is exactly what the search engine returned for that query, including SERP features and AI Overview citations, with the full page stored for 40 days as an HTML snapshot.
Since 2025, and much more since the May 2026 core update, Google can serve a rank tracking tool a different results page than it serves a person. That affects every rank tracker and every SERP data provider. We documented it rather than quietly working around it: the full analysis is here.
Since SERPs are retrieved through live queries, you can always check the Top Sites report to view the SERPs retrieved for your keywords during each ranking update, along with a breakdown of the URLs appearing in those results. Each recorded position links directly to the SERP where the result was found, with its position determined by counting from the top of the search results.
Live queries, not a database
This is the part most people get wrong about rank tracking in general, so it is worth stating plainly.
Some SEO data comes from databases. Most providers crawl or buy a large set of keywords, refresh it on a schedule, then serve you whatever was last stored for your domain. It is fast, it covers millions of keywords, and it is an estimate of where you stood at some point in a refresh window.
AWR works the other way around. Your keyword list is the only list that matters, and every keyword in it gets its own query, every time an update runs either on scheduler or on-demand. Nothing is interpolated, modelled, averaged across users, or carried over from a previous update. If a position appears in your report, a results page was collected for it.
In practice, that means three things:
Your data is as fresh as your last update, not as fresh as someone's database refresh cycle.
Keywords nobody searches still get tracked. There is no volume threshold to clear and no requirement that a keyword exists in a popular keyword database. Long tail, head and brand terms, niche B2B phrases, misspellings, get their own query.
Two updates can legitimately disagree, because a live measurement captures a moment rather than an average. When results are volatile, consecutive measurements show that volatility instead of smoothing it away, which is why trends read better than single updates.
Where AWR searches from
Where you appear to be searching from is one of the strongest signals a search engine uses to decide what to show you, and since country-code domains retired it is the only thing that still carries country-level intent. The same signal is what a city or region setting uses, just pointed at a smaller area. So the location you set on a search engine in your AWR project is not a label on the data, it is a clear instruction about how the queries should be made.
AWR sends each query through addresses that match the location configured for that search engine:
Your setting | What AWR uses | What you are measuring |
|---|---|---|
Country-wide search engine (Google United Kingdom) | A broad pool of addresses spread across that country | A broad market view, not a page a person is served |
Region-level search engine (Gogle United Kingdom, Scotland) | Addresses within that region | A view consistent with that part of the country |
Custom location (Google United Kingdom, Manchester) | A narrow pool close to the tracked place | The page people in that place are served |
The narrower the location, the narrower the address range behind the query, and the closer the collected page sits to what a customer standing there would see. This is the single most useful lever you have over your own data quality, and it costs nothing to test.
💡 One honest caveat about country-wide tracking. A query that says "this country, no particular place" is a question no real person's browser asks. Search engines answer it by composing a country-neutral page rather than anchoring to a stable in-country viewpoint. The result is usable for broad market trends, but it will drift further from any individual city than a precise-location engine does. If your revenue comes from three cities, track those three cities.
What a tracked search engine actually is
When you add a Google search engine to a project, you are describing a search: which tracked results type, where from, on what device, in what language. AWR reproduces that description on every update.
💡 Google is where this goes deepest, since it exposes the most settings to work with. The other engines are tracked the same way, but with fewer settings available, so what you can configure there is more limited.
Tracked results. On Google you choose what gets read off the page. Traditional results cover all results, which includes AI Overviews, as well as organic and Places, or either one on its own. AI results cover AI Overviews only and AI Mode. Each result type needs its own search engine.
Device. Desktop and mobile are collected as a real browser on each device type would see them, because the two pages genuinely differ in layout, feature mix and what sits above the fold. Each device needs its own search engine, so Google UK Desktop and Google UK Mobile are two engines, not one setting.
Language and country settings. On Google, the interface and result language parameters are set to match the engine you configured, so a French project is tracked on French results rather than on an English page served to a French address.
Location. Country, region or precise place, handled as described above.
Change any of these and you have changed the question being asked, which means you should expect a different answer. This is also why a keyword can sit at a different position on Google United Kingdom · Desktop and Google London · Mobile without either number being wrong.
When the data is collected
AWR collects on a schedule you control at project level, and on demand whenever you ask for an update. Both paths go through the same collection process, so an on-demand update is not a lighter or different kind of measurement than a scheduled one.
A few things follow from that:
A position carries the date and time of its own update. It is a measurement at a moment, not a daily average and not a rolling figure.
Updates for a large project are spread out rather than fired all at once. Once a ranking update is triggered, each keyword is processed individually over a period of time, ensuring that the entire update is completed by the time you need the data, rather than all keywords being updated simultaneously.
A gap between your update and your manual check is a real variable, as results move during the day. Comparing a position delivered in the morning with a browser check in the afternoon compares two different moments, and that alone explains the difference.
If you need two numbers to be comparable, make them comparable: run an on-demand update and do your manual check right after it.
Where each number comes from
Not every number in AWR comes from the same place. Some are read off the results page retrieved from the search engines, some come from outside sources, and some things we leave out on purpose.
Read off the results page
Ranking positions for your site and for every other site parsed from the results page, down to the search depth set on the project: 1, 2, 3, 5 or 10 pages
The ranking URL, resolved to its true destination, and the title Google shows for it, with changes flagged between updates
Pixel position and pixel height, so you know where a result sits on the page and how much room it takes
SERP features present on the page, including AI Overviews, and which of them your site holds
People Also Ask questions and the domains Google cites as answering them
In AI results, your brand mentions, citations and citation rank
A dated snapshot of the collected page, kept with the update for 40 days so any number can be traced back to the page it came from
That is the short list, and it is shorter than you would expect. Almost everything else in AWR is worked out from these few things, which is how a handful of collected values turn into the 130-plus metrics defined in the AWR Metrics Index.
Sourced from elsewhere, and labelled as such
Search volume, CPC and competition come from Google Ads Keyword Planner, for the search engine's country or for your custom location
The link metrics behind the difficulty scores, including Trust Flow, Citation Flow, referring domains and backlinks, come from Majestic
The CTR curves behind Click Share and the traffic estimates come from AWR's own Organic CTR study
Impressions, clicks, CTR and sessions come from your own Search Console and Analytics, once you connect them
Left out on purpose
Clickstream or panel data. AWR does not buy browsing behaviour from third parties and does not model your rankings from it.
Your actual traffic. AWR does not know how many visits a position brought you. Estimated Visits, Traffic Cost and the other traffic figures are models built from search volume and CTR curves, and they are labelled as estimates throughout the reports.
Personalized results. Every query is unpersonalized, close to what you get in a clean incognito window: no account signed in, no search history, no cookies carried over from earlier sessions. That is what keeps positions comparable to each other over time, and it is also one reason your own browser, which carries all of that, can show you something different.
Anything collected from a stored ranking index. There isn't one.
The unpersonalized query is the one trade-off here, and it is worth understanding. It gives you a clean baseline rather than one person's view, which is exactly what you want for trends, competitor comparison and reporting. It was never meant to be a simulation of a specific person's search.
How the AWR data collection keeps up with Google
Collection is not a fixed recipe. Google has changed how it answers automated queries over the past two years, country-code domains retired in 2025 and localization became fully dynamic, and since the May 2026 core update the page a tool is served can differ from the page a person is served. We wrote that up in full, mechanics and evidence included, in Why your tracked rankings no longer match what you see in Google. What matters here is what it means for the way your data is collected.
Three things define the method as it stands today.
Pages are rendered, not fetched. AWR collects results pages as a browser composes them, on desktop and on mobile. Pixel position, above-the-fold visibility, AI Overview citations and feature layout do not exist in the raw HTML of a results URL, so parsing raw code would measure a page no user ever sees.
Suspect pages are re-queried before anything is stored. A page that comes back structurally thinner, or inconsistent with the market being measured, has a recognisable signature. The pipeline detects those and runs the keyword again rather than recording the first response it gets, so a suspect page never becomes a number in your report.
Result URLs are resolved after collection, not during it. Where links are masked behind goto redirects, AWR works out the true destinations in its own pipeline afterwards. One query retrieves one page, which keeps collection as light as a real search session.
Alongside those, we keep expanding and diversifying the pool of addresses we collect through, and region-level tracking lets a country search engine stay anchored to a consistent region, which keeps pages comparable between updates and closer to what people in that market see.
How this compares with the rest of the industry
Most large SEO and digital intelligence platforms publish a data methodology page, and reading them side by side is useful, because they describe genuinely different things.
Semrush documents a blend of sources. For search engine rankings and keyword analytics it partners with third-party data providers, which collect Google results pages for hundreds of millions of the most popular keywords and record the sites listed in the top 100. Its traffic and market data comes from a different place again: a panel of over 200 million anonymized internet users across more than 190 countries, built through partnerships with clickstream providers and processed through a neural network into estimates.
Similarweb describes four input categories feeding one engine: direct measurement from sites that share their first-party analytics, a contributory network of consumer products, partnerships that collect digital signals across the internet, and public data extraction. Its support documentation is admirably direct about what that implies, saying plainly that it is an estimations tool and that its numbers are not expected to align exactly with a user's own direct measurement data.
Ahrefs is the closest comparison, since it also tracks rankings. Its help documentation explains that keyword positions in Site Explorer are refreshed on a cadence tied to search volume, with high-volume keywords typically updated every one to three days and low-volume ones at intervals of weeks or months, and that its rankings are based on snapshots of the results page taken at the time of sampling.
All of these approaches are legitimate, and each is good at the job it was built for. Panel and database methods cover enormous keyword universes, estimate traffic, and reconstruct competitor visibility for domains you have no access to. AWR was built for a different job: the keywords you chose, in the places your customers are, measured one query at a time, with the results page kept as evidence. Narrower on purpose, and more direct where it counts.
Five questions worth asking about any ranking number, and how AWR answers each one against a database or panel method:
AWR | Database and panel methods | |
|---|---|---|
Source of a position | A query run at that moment for that keyword | A refreshed index of rankings, modelled or blended |
Keyword universe | Your list, whatever is in it | Pre-existing database; your keyword has to be in it |
Freshness | As of your last update | As of the last refresh cycle |
Evidence behind a number | The dated page snapshot, attached to every position | Rarely exposed |
Nature of the figure | A measurement | Often an estimate |
The honest summary is that estimates are better for breadth and measurements are better for accountability. If you need to defend a specific number to a specific client on a specific date, you want the page it was read from.
Check our work on any number
None of this has to be taken on trust. Every update stores the results page it was read from for 40 days. Click a position in the Keyword Ranking report, then View SERP, and you are looking at the page AWR collected for that keyword.
The Top Sites report shows the same collected page from the other direction: every URL parsed from it, so you can check who else is there without opening the snapshot one competitor at a time. Is your site where the report says? Are the expected competitors there? Does it look like a normal, full results page for that market?
If what you find there is not what you expected, the full diagnostic, including the alias check and the location test, is in Why your tracked rankings no longer match what you see in Google. Send us the project, the keywords and the update date if it still does not add up. Those cases feed straight into our collection engineering.
Frequently Asked Questions
Does AWR buy ranking data from third-party databases?
Why does a country-wide search engine show different numbers than a city one?
Does AWR use my search history or personalize results?
Can I see the actual page a position was read from?
Our site appears in an AI Overview but our organic position dropped. Which one is right?
My tracked position does not match my browser. Is the data wrong?
stay in the loop