Question

I am trying to scrape an ASP.NET website but am having trouble getting the results from a post. I have the following python code and am using httplib2 and BeautifulSoup:

conn = Http()
# do a get first to retrieve important values
page = conn.request(u"http://somepage.com/Search.aspx", "GET")

#event_validation and viewstate variables retrieved from GET here...

body = {"__EVENTARGUMENT" : "",
        "__EVENTTARGET" : "" ,
        "__EVENTVALIDATION": event_validation,
        "__VIEWSTATE" : viewstate,
        "ctl00_ContentPlaceHolder1_GovernmentCheckBox" : "On",
        "ctl00_ContentPlaceHolder1_NonGovernmentCheckBox" : "On",
        "ctl00_ContentPlaceHolder1_SchoolKeyValue" : "",
        "ctl00_ContentPlaceHolder1_SchoolNameTextBox" : "",
        "ctl00_ContentPlaceHolder1_ScriptManager1" : "ctl00_ContentPlaceHolder1_UpdatePanel1|cct100_ContentPlaceHolder1_SearchImageButton",
        "ct100_ContentPlaceHolder1_SearchImageButton.x" : "375",
        "ct100_ContentPlaceHolder1_SearchImageButton.y" : "11",
        "ctl00_ContentPlaceHolder1_SuburbTownTextBox" : "Adelaide,SA,5000",
        "hiddenInputToUpdateATBuffer_CommonToolkitScripts" : 1}

headers = {"Content-type": "application/x-www-form-urlencoded"}
resp, content = conn.request(url,"POST", headers=headers, body=urlencode(body))

When I print content I still seem to be getting the same results as the “GET” or is there a fundamental concept I’m missing to retrieve the result values of an ASP .NET post?

Answer 1

This isn’t technically an answer, but you could use Fiddler to examine the difference between what you are sending with your python code, versus what would be sent if you used a web browser to do the post.

I find that usually helps in these types of situations.