This is kind of a two part question. Is it possible to retrieve data from a page that loads content dynamically through javascript by using WebClient/HttpWebRequest?
I’d also like to how I would be able to use WebClient/HttpWebRequest to replicate a XMLHttpRequest like you would see executed via javascript.
Edit:
I captured the headers of the request I am trying to replicate which looks like this:
http://www.tagged.com/api/?application_id=user&format=json&session_token=6thk20fhv7d727emgdhfka6034
POST /api/?application_id=user&format=json&session_token=6thk20fhv7d727emgdhfka6034 HTTP/1.1
Host: http://www.tagged.com
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,/;q=0.8
Accept-Language: en-us,en;q=0.5
Accept-Encoding: gzip,deflate
Accept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.7
X-Requested-With: XMLHttpRequest
Content-Type: application/x-www-form-urlencoded; charset=UTF-8
Content-Length: 88
connect_status=-1; __utmb=50703532.0.10.1303366930
DNT: 1
Connection: keep-alive
Pragma: no-cache
Cache-Control: no-cache
method=tagged.usermgmt.addFriend&uid_to_add=5402501977&api_signature=&track=1mJ0lY7-W3
I understand everything up until the method. As far as I know you can only supply GET or POST to HttpWebRequest.Method. Could someone maybe fill in the blanks for me?
If you are trying to talk to the server in the same manner that javascript does however you might have some options.
First, you have to determine how those pages communicate with the server. For example, JSON or SOAP. You can do this with a network sniffer such as wireshark.
Once you have done this, you can send a JSON request of your own to that server using and parse the JSON response using one of the available C# JSON parsers such as JSON.NET.
You can use a similar method in the case of SOAP. Be mindful they might not like you talking to their API’s this way.