Home/ Questions/Q 4101576
In Process

The Archive Base Latest Questions

Editorial Team
  • 0
Asked: 2026-05-20T20:42:54+00:00

I’m trying to automate the download of some data from a webform. I’m using

  • 0

I’m trying to automate the download of some data from a webform. I’m using python’s mechanize module.

The url is here: http://www.hiv.lanl.gov/components/sequence/HIV/search/search.html
I need to fill out the Sequence Length, Subtype and Genomic Region. I’ve got the Sequence-Length and Genomic-Region figured out but I’m having trouble with selecting the Subtype. When I load the form is get an empty SelectControl and mechanize won’t let me select anything.

This code should load the website:

import mechanize
import cookielib

# Browser
br = mechanize.Browser()

# Cookie Jar
cj = cookielib.LWPCookieJar()
br.set_cookiejar(cj)

# Browser options
br.set_handle_equiv(True)
br.set_handle_gzip(True)
br.set_handle_redirect(True)
br.set_handle_referer(True)
br.set_handle_robots(False)

# Follows refresh 0 but not hangs on refresh > 0
br.set_handle_refresh(mechanize._http.HTTPRefreshProcessor(), max_time=1)

br.addheaders = [('User-agent', 'Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.9.0.1) Gecko/2008071615 Fedora/3.0.1-1.fc9 Firefox/3.0.1')]

br.open('http://www.hiv.lanl.gov/content/sequence/HIV/mainpage.html')
br.follow_link(text = 'Search Interface')

br.select_form(nr = 1)

Any help would be greatly appreciated.

-Will

EDIT:
I tried to use BeautifulSoup to re-parse the HTML (as per this SO question) but no luck there.

New Edit:
Below is an excerpt of the mechanize form.

<search POST http://www.hiv.lanl.gov/components/sequence/HIV/search/search.comp multipart/form-data
  <TextControl(value SequenceAccessions SA_GenBankAccession 1=)>
  <SelectControl(master=[Any, *HIV-1, HIV-2, SIV, SHIV, syntheticwholeDNA, NULL])>
  <TextControl(value SEQ_SAMple SSAM_common_name 1=)>
  <SelectControl(slave=[])>
  <TextControl(value SequenceEntry SE_sequenceLength 4=)>
  <CheckboxControl(sample_year_exact=[*1])>
  <TextControl(value SEQ_SAMple SSAM_Sample_year 11=)>
  <TextControl(value SEQ_SAMple SSAM_Sample_country 1=)>
  <CheckboxControl(recombinants=[Recombinants])>

For some reason the slave control is not populated with the possible choices.

Share

Leave an answer

You must login to add an answer.

Need An Account,

1 Answer

  1. Editorial Team

    The problem is that the Subtype <select> options are populated by Javascript, which mechanize does not execute. The Javascript code runs when the page loads, populating the slave options with the HIV-1 list:

    addLoadEvent(fillList(document.forms['search'].slave, lists['HIV-1']));

    However, the mapping of Virus -> Subtype is stored in this Javascript file search.js. You may need to store this mapping in your Python script and manually set the slave form value.

    • 0