Question

Let’s say I have a squid proxy server with 10 IPs assigned to it (locally on my gigE network).

And let’s also say I have a different server with a different 10 IPs assigned to it, running a c# application which is running 10 threads, each thread being used to crawl a website. Each thread using a different assigned IP to send out its HTTP request.

My question is, if I run 10 threads in that application where each thread uses a proxy IP from the proxy server to crawl the website. And I also try 10 threads in that app where each thread uses a locally assigned IP directly on the server to crawl the site. Which will go faster to crawl the site from page to page, a thread using a proxy IP, or a thread using a direct IP from its own server?

I would imagine a direct IP on the server because proxy servers have inherent latencies but its proving to show the proxy server is being slightly faster. So I’m confused. Maybe its because using a direct IP uses some processing power and by using a proxy server it is able to outsource that processing to the proxy server? Help is much appreciated.

Answer 1

You mentioned that 10 different threads “crawl the site”, which sounds like you have 10 threads scraping different pages of the same site.

In this case, using a proxy is often faster. Remember that most html pages look something like this:

foo1.html
  |
  +--- Link: somefile01.xml
  |
  +--- Link: somestlyesheet.css
  |
  +--- Image: somepicture01.jpg
  |
  +--- Image: somepicture02.jpg

foo2.html
  |
  +--- Link: somefile01.xml
  |
  +--- Link: somestlyesheet.css
  |
  +--- Image: somepicture01.jpg
  |
  +--- Image: somepicture03.jpg

After a real web brower pulls foo1.html, it performs http GETs for somefile01.xml, somestylesheet.css, somepicture01.jpg, and somepicture02.jpg (usually in parallel). Presumably, if you scrape pages from the same site, you’re scraping pages that have some common overlap in content (such as shown above between foo1.html and foo2.html).

If you have http object caching enabled on your proxy, your threads will be faster because you will get cache hits for some fraction of those files. If there is a cache hit, then the file is delivered to your code locally from the Squid cache on your LAN, instead of having to source it from the remote server every time.

If you have 10 threads which all crawl different sites, then you should expect slower performance unless the only file they included was something like jquery.js. Incidentally, google offers web developers a way to use their cache to load several commmon javascript files.