Question

i have a mechanize python script written for submitting forms to inquire drug information. and when i run it, it gives me no error message, but when i look at the response, it’s not what I see on my browser view-source page. i checked the urls after the submission:

here’s what I got:

http://www.accessdata.fda.gov/scripts/cder/drugsatfda/index.cfm

here’s addresses I’m supposed to get:

http://www.accessdata.fda.gov/scripts/cder/drugsatfda/index.cfm?fuseaction=Search.DrugDetails

I see that the second url does not contain my query text, does that mean i need cookies? if so, how?

this is my code snippet:

br = mechanize.Browser()
cj = cookielib.LWPCookieJar()
br.set_cookiejar(cj)
....
br.addheaders = [('User-agent', 'Mozilla/6.0 (X11; U; i686; en-US; rv:1.9.0.1) Gecko/2008071615 OS X 10.2 Firefox/3.0.1')]

fda_url2 = 'http://www.accessdata.fda.gov/scripts/cder/drugsatfda/index.cfm?fuseaction=Search.Addlsearch_drug_name'

print br.open(fda_url2).geturl()


for f in br.forms():
   print 'this is a form'
   print f

br.select_form('searchoptionB')
br.form['ApplNo'] = '018780'
html = br.submit(name = 'Search_Button')

print html.geturl()

the print form output was:

<searchoptionB POST http://www.accessdata.fda.gov/scripts/cder/drugsatfda/index.cfm application/x-www-form-urlencoded
  <HiddenControl(fuseaction=Search.SearchAction) (readonly)>
  <HiddenControl(SearchType=AddlSearch) (readonly)>
  <HiddenControl(SearchOption=B) (readonly)>
  <TextControl(ApplNo=)>
  <SubmitControl(Search_Button=Submit) (readonly)>
  <SubmitControl(clearcriteria=Clear) (readonly)>>

sorry for the long post ;p

Answer 1

UPD Regarding your comment. Here’s my test file:

#!/usr/bin/env python
import mechanize
import cookielib

br = mechanize.Browser()
cj = cookielib.LWPCookieJar()
br.set_cookiejar(cj)
br.addheaders = [('User-agent', 'Mozilla/6.0 (X11; U; i686; en-US; rv:1.9.0.1) Gecko/2008071615 OS X 10.2 Firefox/3.0.1')]

fda_url2 = 'http://www.accessdata.fda.gov/scripts/cder/drugsatfda/index.cfm?fuseaction=Search.Addlsearch_drug_name'

print br.open(fda_url2).geturl()


for f in br.forms():
   print 'this is a form'
   print f

br.select_form('searchoptionB')
br.form['ApplNo'] = '018780'
html = br.submit()

print br.response().read()

And this is what I get when running it:

$ ./test.py  | grep HUM
            <td valign="top" class="product_table" height="24"><a href="index.cfm?fuseaction=Search.Overview&DrugName=HUMULIN%20R">
 HUMULIN R
 INSULIN RECOMBINANT HUMAN
            <td valign="top" class="product_table" height="24"><a href="index.cfm?fuseaction=Search.Overview&DrugName=HUMULIN%20R%20PEN">
 HUMULIN R PEN
 INSULIN RECOMBINANT HUMAN

Maybe you’ve somehow changed your script or query a different page?

---

Have you tried to submit this form in a browser?

If I navigate to this URL, fill in “Option B” with “018780” and hit “Submit”, the browser indeed redirects me to http://www.accessdata.fda.gov/scripts/cder/drugsatfda/index.cfm, which contains the search results.

Try adding this to the end of your code snippet:

print br.response().read()

This will output the HTML of the page, and it does contain search results as expected.

I see that the second url does not contain my query text, does that mean i need cookies? if so, how?

This form is sent via POST, and in this case all parameters are embedded in request body (RFC 2616).