Question

I am trying to parallelize my web parsing tool but the speed gains seem very minimal. I have i7-2600K (8 cores hyper-threading).

Here is some code to show you the idea. I only show Parallel.ForEach but you get the idea:

List<string> AllLinks = this.GetAllLinks();
ConcurrentDictionary<string, Topic> AllTopics = new ConcurrentDictionary<string, Topic> ( );

int count = 0;
Stopwatch sw = new Stopwatch ( );
sw.Start ( );

Parallel.ForEach ( AllLinks, currentLink =>
{
    Topic topic = this.ExtractTopicData ( currentLink );
    this.AllTopics.TryAdd ( currentLink, topic );

    ++count;

    if ( count > 50 )
    {
        Console.WriteLine ( sw.ElapsedMilliseconds );
        count = 0;
    }
} );

I get these timings:

Standard foreach loop:
24582
59234
82800
117786
140315

2 links per second


Paralel.For:

21902
31649
41168
49817
59321


5 links per second

Paralel.ForEach:
10217
20401
39056
49220
58125

5 links per second

Firstly why is the “startup” timing is much slower in Parallel.For?

Other than that the parallel loops give me 2.5x speed over the standard foreach loop. Is this normal?

Is there a setting I can set so that the parallel loops can use all the cores?

EDIT:

Here is pretty much what ExtractTopicData does:

HtmlAgilityPack.HtmlWeb web = new HtmlWeb ( );
HtmlAgilityPack.HtmlDocument doc = web.Load ( url );
IEnumerable<HtmlNode> links = doc.DocumentNode.SelectNodes ( "//*[@id=\"topicDetails\"]" );

var topic = new Topic();

foreach ( var link in links )
{
    //parse the link data
}

Answer 1

A brief perusal of HtmlAgilityPack.HtmlWeb confirms that it is using the synchronous WebRequest API. You are therefore placing long running tasks into the ThreadPool (via Parallel). The ThreadPool is designed for short-lived operations that yield the thread back to the pool quickly. Blocking on IO is a big no-no. Given the ThreadPool’s reluctance to start new threads (because it is not designed for this kind of usage), you’re going to be constrained by this behaviour.

Fetch your web content asynchronously (see here and here for the correct API to use, you’ll have to investigate further yourself…) so that you are not tying up the ThreadPool with blocking tasks. You can then feed the decoded response to the HtmlAgilityPack for parsing.

If you really want to jazz up performance, you’ll also need to consider that WebRequest is incapable of performing asynchronous DNS lookup. IMO this is a terrible flaw in the design of WebRequest.

The BeginGetResponse method requires some synchronous setup tasks to complete (DNS resolution, proxy detection, and TCP socket connection, for example) before this method becomes asynchronous.

It makes high performance downloading a real PITA. It’s at about this time that you might consider writing your own HTTP library so that everything can execute without blocking (and therefore starving the ThreadPool).

As an aside, getting maximum throughput when chumming through web-pages is a tricky affair. In my experience, you get the code right and are then let down by the routing equipment it has to go through. Many domestic routers simply aren’t up to the job.