I have about a million urls pointing to HTML pages on a public web server that I want to save to my disk. Each of these is about the same size, ~30 kilobytes. My url lists are split about evenly in 20 folders on disk, so for simplicity I create one Task per folder, and in each task I download one URL after the other, sequentially. So that gives me about 20 parallel requests at any time. I’m on a relatively crappy DSL, 5mbps connection.
This represents several gigabytes of data so I’m expecting the process to take several hours, but I’m wondering if I could make the approach any more efficient. Is it likely I’m making the most out of my connection? How can I measure that? Is 20 parallel downloads a good number or should I dial up or down?
The language is F#, I’m using WebClient.DownloadFile for every url, one WebClient per task.
==================================
EDIT: One thing that made a huge difference was adding a certain header to the request:
let webClient = new WebClient()
webClient.Headers.Add(HttpRequestHeader.AcceptEncoding, "gzip,deflate")
This cut the size of downloads from about 32k to 9k, resulting in enormous speed gains and disk space savings. Thanks to TerryE for mentioning it!
If you are using a downloader API, then make sure that it is issuing a
request header so that the site that you are scraping knows to return compressed HTML. (Most webservers wil be configured to compress the HTML data streams if the client uses this request header to let the server know that it will accept compressed data streams.)
This will reduce the data transferred by roughly a factor of 4. (E.g. this page was 40K raw HTML, but only 10K was transferred to my browser (the HTML is zipped).