Question

I have a program which I’m trying to get to work as fast as possible. This program loads a number of different websites and performs some scraping on them.

I used to perform the scraping by using the Forms.HtmlDocument (I basically download it by using WebRequests then I push it into a document using a WebBrowser control) – however that is impossible to parallelise cleanly due to not being able to force the WebBrowser to update when its not the main thread.

So I decided to try out the HtmlAgilityPack, thinking that perhaps I could paralelise that. However then I read the following post:

How to get max performance using Parallel.For/ForEach? (performance timings included)

Which suggests that it doesn’t really paralelise very well.

Converting all the code will take some time (due to the quirks and the complexity of it all) – however I’d like to know whether its worth it or not. If I avoid using WebGet (and instead obtain a stream using WebRequest and push that into the AgilityPack) – will that give me a useful performance increase? Currently it takes about 19 seconds for each iteration, with the majority of time spent waiting for the page to download.

Any other ideas will be considered. Thanks.

EDIT: While we’re here, is there any speed increase with using either method (even in a single-threaded environment?)

Answer 1

For my case htmlagilitypack was working very well. One big issue is RFC compliance that should be overiden when you create srapping applications that run as fast as possible.

1)Add this at the beggining of your main method

System.Net.ServicePointManager.DefaultConnectionLimit = 100;

2)Parallelize with normal multithreading library and not tpl.