Question

I have a code with parallel class to download pages from web. Since I download about 3000 pages I want to know if it the best way.

Parallel.For(0, 3000, i =>
            {
                Console.WriteLine(i.ToString());
                //HttpDownloader is my class for downloading
                HttpDownloader ht = new HttpDownloader(s[i]);
                string a = ht.GetPage();
                Console.WriteLine(i.ToString());
            }); 

After that I run 2 func: pharsing(string html) and save()//Save into DB
How Can I do it by Parallel??
And, if I want It to run background I need to insert it to BackgroundWorker?

Answer 1

The Task Parallel Library (TPL, where Parallel comes from) modules are the way to go – like you’re already doing. But, you can make things clearer by using Parallel.ForEach over Parallel.For:

var urls = new List<string> { "http://google.com", "http://yahoo.com" };

Parallel.ForEach(urls, url => {
    using (var client = new WebClient())
    {
        var contents = client.DownloadString(url);
        // parse contents
        // add to database
    }
});

Further reading, especially to show how the asynchronous pattern can be used to save on threads (but not necessarily speed) is here: http://blogs.msdn.com/b/pfxteam/archive/2009/08/04/9857477.aspx

This is a useful read but I think the code is at this point the right balance of performance and readability.