Question

List<string> urls = this.populateRequestList();

this.Logger("Starting");

var reqs = urls.Select<string, WebRequest>(HttpWebRequest.Create).ToArray();
var iars = reqs.Select(req => req.BeginGetResponse(null, null)).ToArray();
var rsps = reqs.Select((req, i) => req.EndGetResponse(iars[i])).ToArray();

this.Logger("Done"); 

Things I noticed so far:
When I run this code, “Starting” shows up in my log, but “Done” never shows up. When I view the whole process in the debugger, it seems to skip over it like it’s not even there. No exceptions are being thrown either. When reqs.Select is looping through req.EndGetResponse(iars[i]), it’s like it freezes or skips over stuff. When I view it in the debugger, I don’t get past 10-15 loops before it just skips to the end.

Questions:
How do I stop this from “skipping” sometime during var rsps = reqs.Select((req, i) => req.EndGetResponse(iars[i])).ToArray();?

How to I get the html from rsps? I think this problem doing that stems from the “skipping”. I tried looping through each response and calling Repsponse.GetResponseStream() etc…, but nothing happens as soon as it skips.

Answer 1

The problem with your code is that BeginGetResponse(null, null) accepts a callback as the first argument which is invoked when the operation completes. This callback is where EndGetResponse should be called. When you call EndGetResponse, the operations are not yet completed.

Look at this article to see how aync web requests can be made in C# using iterators: http://tomasp.net/blog/csharp-async.aspx.

If using the task parallel library or .NET 4 you can also do this:

var urls = new List<string>();

            var tasks = urls.Select(url =>
                {
                    var request = WebRequest.Create(url);
                    var task = Task.Factory.FromAsync<WebResponse>(request.BeginGetResponse, request.EndGetResponse, null);
                    task.Start();
                    return task;
                }).ToArray();

            Task.WaitAll(tasks);

            foreach (var task in tasks)
            {
                using (var response = task.Result)
                using (var stream = response.GetResponseStream())
                using (var reader = new StreamReader(stream))
                {
                    var html = reader.ReadToEnd();
                }                                
            }