Question

I am trying to submit a simple POST form to a website to get past the form screen and download data from the site. It is not working because I do not know the exact URL of where to submit the form (no *.cgi script).

I tried to follow the tutorial at http://www.voidspace.org.uk/python/articles/urllib2.shtml#data

How would I determine where to submit such a form?
In this example, the form is at this location:
http://www.verizonwireless.com/b2c/store/controller?item=phoneFirst&action=viewPhoneOverviewByDevice&deviceCategoryId=1

import urllib.parse
import urllib.request

## URL is wrong...
url = "http://www.verizonwireless.com/b2c/store/controller?item=phoneFirst&action=viewPhoneOverviewByDevice&deviceCategoryId=1"

values = { 'zipcode' : "11111",
'state': "NY" }

data = urllib.parse.urlencode(values)
data = data.encode('ascii')

req = urllib.request.Request(url,data)

response = urllib.request.urlopen(req)
the_page = response.read()
print(the_page)

Answer 1

The form is posted to the “action” target for the form tag.

In this case, the form tag is:

<form name="zipForm" id="zipForm" action="/b2c/vzwfly" method="post">

And hence, the URL to submit the form to is http://www.verizonwireless.com/b2c/vzwfly

Sometimes Javascript is used to submit the form, and then you need to analyze the Javascript, but that doesn’t seem to be the case here.