Question

If website pops out a 503 then webclient will just throw an exception.

For example, go to http://www.google.com/sorry/?continue=http%3A%2F%2Fwww.google.com%2Fsearch%3Fq%3Dkucing

If we open it in internet explorer it’ll open a page. If we use livehttpheader it returns a 503 rather than 200. However, it still show something.

Now try curl the same page.

http://www.google.com/sorry/?continue=http%3A%2F%2Fwww.google.com%2Fsearch%3Fq%3Dkucing

The curl will just stop failing. So how to make curl treat 503 like 200?

Samples are when we try to search something at google. Google sometimes require captcha. Well, just pop that out so I can fill the captcha. But webclient simply throw an exception without assigning the content to a file. That’s not good.

How to ensure that webclient do not throw things out.

Same goes for curl

Answer 1

For WebClient you need to process the WebException.Response. E.g. this LINQPad query dumps the HTML provided by my web server’s “Not Found” error web page:

Dim wc = New System.Net.WebClient
Try 
  Dim rd = wc.DownloadData(New Uri("http://localhost/test"))
  rd.Dump
Catch Ex As System.Net.WebException
 Dim rs = Ex.Response
 Call (New StreamReader(rs.GetResponseStream)).ReadToEnd.Dump
End Try

_{Now actually using WebClient.}