How to track 50 000 keywords using AWR

Track Large Projects

Track Large Projects

We often receive feedback from our clients about tracking large number of websites with lots of keywords faster. But the speed of Advanced Web Ranking depends entirely on search engines. They use advanced systems to protect themselves from automatic queries, because they affect their bandwidth and processing power. From this reason there is no way to pass those systems without having your IP address temporarily banned.

To deal with this issue, Advanced Web Ranking uses “human emulation technology”, which makes the queries indistinguishable for search engines. In the same time using “human emulation technology” Advanced Web Ranking will not affect their bandwidth and processing power. This is because it adds a random delay between queries, so that your IP address will not get banned by search engines.

A query represents the search for a specific keyword in a search engines on a page.

You can disable or decrease the delay between each query, although we strongly recommend you to not do this. If you decrease or disable the delay between queries, you will be most probably banned by search engines in a matter of seconds.

To find the time Advanced Web Ranking will take to complete your projects multiply the number of keywords with the search depth and with the minimum delay between queries.

The maximum number of queries Advanced Web Ranking can make in 24 hours is about 5 000 with a delay between queries of 15 to 20 seconds. So if you have to track an website in 2 search engines for 300 keywords with a search depth of 5 pages (50 results) then you will have 1 500 queries and the update will be finished in about 6 hours and a half (300 x 5 = 1 500 queries x 15 seconds = 22 500 / 3 600 = 6.25 hours).

If Advanced Web Ranking has 50 000 queries to make, things can become a little tricky because it can take up to 8 days for Advanced Web Ranking to update such a large project.

But there are some tricks you can use to update your projects faster. To speed up the update for large projects you have two possibilities:

  • Use multiple proxy servers
  • Update Advanced Web ranking projects from multiple computers with different IP addresses

1. Multiple Proxy Servers

Multiple proxies allow you to increase the number of queries AWR can make by multiplying the queries with the number of proxy servers you will add. For example, if you will use 10 proxy servers, you will be able to make a maximum number of about 50 000 queries in 24 hours.

2. Different IP addresses

Another possibility to update large projects in less time is to use multiple computers with different IP addresses for updates. Using a Server license you can then share the same database and have all the projects in one place.

If you are looking to read more details about this we also added a topic about this on Advanced Web Ranking Forums.

In case you have any further questions about this, do not hesitate to ask them in the comments section.

  • Nick

    Hey, thanks for this blog post, I’ve been looking for this kind of clarification for a long time now.

    However, the first solution you mention is kind of incomplete. Ok, AWR supports multiple proxies, but where the hell do I find these proxies ? I did not manage to find any “serious” proxy rental service so far.

    The main focus of proxy providers is directed towards user who are looking for a single IP dedicated to anonymous web surfing, which won’t be suitable for SEO needs.

    Any thoughts on this ?

    • http://blog.advancedwebranking.com/author/cosmino/ Eugen Oprea

      Nick, I got back to you regarding this via email. I hope everything is clear now.

      Michael, we can recommend a reliable proxy server provider. Here you can find more information:

      http://www.advancedwebranking.com/trustedproxies

  • Michael Wissekerke

    I hear nick on this one!

    i have exactly the some problem, i’ve tried everything to find proxies that worked, even went as far as harvesting a list, and none of them work!!!

    is there anyway that you could point us the right direction?!

  • http://www.timgriffin.com/ Timmy G

    I just bought a 10 pack of IPs from http://www.yourprivateproxy.com/ for under $30 USD a month which I thought was quite reasonable.

    Be open to any other services you guys find!

  • http://www.super-nails.org/ pirmasens nagelstudio

    on-line for articles that can assist me. Thank you

  • http://how-to-blog.tv Derick Schaefer

    First of all, great blog . . thanks for doing this. We just got started with AWR and I am so impressed with the well thought out features. In terms of the proxies, ProxyPal is a great iPhone app that gives proxies that users contribute on a daily basis. Another one is an off the shelf product that you can buy at Staples called WebGhost. It has a good proxy but also provides extended lists of public proxies daily.

    Best of luck with it!

  • http://www.bropass.com bropass

    Couldn’t possibly love you guys more than I already do!, thanks for making my life easier!

  • Jim WIlson

    I am having great problems with Yahoo on my queries.

    I am running 30 proxy servers in the software – do the queries rotate through the proxy list ?

    What are the best delay settings for Yahoo. I am using -

    20 Connections
    Search depth 50 (stop if found)
    15 – 20 between queries
    Sleep 1 minute after 50 queries
    Sleep 5 minutes after 3 failures

    • http://www.advancedwebranking.com/ Robert Dinu

      We have released a patch yesterday (build 7) which should solve the proxy issue. Please install it and tell us how it goes.

      What you can change is the last option, to have “Sleep 30 minutes when a search engine fails 3 times”. Also, right click on the Yahoo search engine on the left input panel, select Properties and click on the Restore Defaults button.

  • Josh

    Hmm I tried to set up proxy’s but I appear to be having some issues with the ports with regards to the proxy set up. It appears that Im only able to get successful test on a proxy when its set up on port 8080… Is anyone else having this issue?

    Please advise if someone has some insight

    • http://blog.advancedwebranking.com/author/cosmino/ Eugen Oprea

      Hi Josh,

      This is the first time when we hear about this problem and we suspect to be an issue with the proxy server you are using or that a firewall blocks connections on other ports than 8080. So please double check your firewall settings and try using a new proxy server, but if you still have problems, please send us an error report from the [Help] menu, [Report Error].

      Eugen

  • http://www.imagefilm24.com Heiner Etzler

    AWR takes time but the results are excellent. And it works in Germany and the local search engines of Google, Yahoo, Bing and all the others. Take a look to our website and the review of AWR.
    The review of AWR on the WebVideoBlog: http://www.imagefilm24.com/advanced-web-ranking/

  • http://www.freeads.in Naval Sharma

    I have a query i need a software or a tool who can find my more than 5000 keywords ranking on google.co.in on monthly basis because i am dealing with local india search engine and a free classfieds site, whether it is paid if you give me proper guidance in this respect that would be graet help for me.

  • Pingback: Advanced Web Ranking – software tool | PaulG.nl