Question

I use Mechanize to loop through a table, which is paginated.

I have a problem with a form that holds multiple submit inputs. The input tags are used as pagination and they are generated dynamically. When I loop through the pages I need to scrape, I need to be able to pick the right input, since only one of them will take me to the “next page”. The right tag can be identified by different attributes such as name, class, value etc. My problem is though, that I can’t find out how to tell mechanize which one to use.

I tried this:

require 'mechanize'
require 'yaml'

url = "http://www.somewhere.com"

agent = Mechanize.new
page = agent.get(url)

loop do

  puts "some content from site using nokogiri"

  if next_page = page.form_with(:action => /.*/)
    page = next_page.submit(page.form_with(:action => /.*/).submits[3])
  else
    break
  end

end

From this post, http://rubyforge.org/pipermail/mechanize-users/2008-November/000314.html, but as told the number of tags are changing so just picking a hardcoded number of the submits is not too good an idea.

What I would like to know is if there is a way like this:

loop do

  puts "some content from site using nokogiri"

  if next_page = page.form_with(:action => /.*/)
    page = next_page.submit(:name => /the_right_submit_button/)
  else
    break
  end

end

or something like that, maybe with a css or xpath selector.

Answer 1

I usually use form.button_with to select the right button to click:

form = results_page.forms[0]
results_page = form.submit(form.button_with(:name=>'ctl00$ContentBody$ResultsPager$NextButton'))