I’ve been trying (unsuccessfully, I might add) to scrape a website created with the Microsoft stack (ASP.NET, C#, IIS) using Python and urllib/urllib2. I’m also using cookielib to manage cookies. After spending a long time profiling the website in Chrome and examining the headers, I’ve been unable to come up with a working solution to log in. Currently, in an attempt to get it to work at the most basic level, I’ve hard-coded the encoded URL string with all of the appropriate form data (even View State, etc..). I’m also passing valid headers.
The response that I’m currently receiving reads:
29|pageRedirect||/?aspxerrorpath=/default.aspx|
I’m not sure how to interpret the above. Also, I’ve looked pretty extensively at the client-side code used in processing the login fields.
Here’s how it works: You enter your username/pass and hit a ‘Login’ button. Pressing the Enter key also simulates this button press. The input fields aren’t in a form. Instead, there’s a few onClick events on said Login button (most of which are just for aesthetics), but one in question handles validation. It does some rudimentary checks before sending it off to the server-side. Based on the web resources, it definitely appears to be using .NET AJAX.
When logging into this website normally, you request the domian as a POST with form-data of your username and password, among other things. Then, there is some sort of URL rewrite or redirect that takes you to a content page of url.com/twitter. When attempting to access url.com/twitter directly, it redirects you to the main page.
I should note that I’ve decided to leave the URL in question out. I’m not doing anything malicious, just automating a very monotonous check once every reasonable increment of time (I’m familiar with compassionate screen scraping). However, it would be trivial to associate my StackOverflow account with that account in the event that it didn’t make the domain owners happy.
My question is: I’ve been able to successfully log in and automate services in the past, none of which were .NET-based. Is there anything different that I should be doing, or maybe something I’m leaving out?
For anyone else that might be in a similar predicament in the future:
I’d just like to note that I’ve had a lot of success with a Greasemonkey user script in Chrome to do all of my scraping and automation. I found it to be a lot easier than Python + urllib2 (at least for this particular case). The user scripts are written in 100% Javascript.