Question

I have this html:

<input type="text" class="txtSearch">
<input type="submit" value="Search" class="sbtSearch">

What I need is to write in the text field and then click on submit using python. The input tags are not inside Form. How I could do that?

Answer 1

You shouldn’t have to actually populate the fields and ‘click’ submit. You can simulate the submission and get the desired results.

Use BeautifulSoup and urllib alongside firebug in Firefox. Watch the network traffic with firebug, and get the post parameters from the HTTP POST that the submit is doing. Create a dict and url-encode it. Pass it alongside your url request.

For example:

from BeautifulSoup import BeautifulSoup
import urllib

post_params = {
    param1 : val1,
    param2 : val2,
    param3 : val3
        }
post_args = urllib.urlencode(post_params)

url = 'http://www.website.com/'
fp = urllib.urlopen(url, post_args)
soup = BeautifulSoup(fp)

The parameter vals will change according to what you’re attempting to submit. Make appropriate accommodations in your code.