Question

I am scrapping google using following code

def google_search(query)
  page = @agent.get('http://www.google.com')
  google_form = page.form_with(:name => 'f')
  google_form.q = query
  page = @agent.submit(google_form)
  page
end

but sometimes i get NoMethodError: undefined methodq=’ for nil:NilClass` error. What might be the issue?

Answer 1

You are using the sample code from the docs, there is no form with that name in google search, try “gbqf” but it might differ depending on locale or various settings. When you are using mechanize use “pp” to pretty print the elements during development to see what you are fetching and going next.

Working .rb sample (updated from http://mechanize.rubyforge.org/EXAMPLES_rdoc.html) to get first 10 results by query.

require 'rubygems'
require 'mechanize'

a = Mechanize.new { |agent|
  agent.user_agent_alias = 'Mac Safari'
}

a.get('http://google.com/') do |page|
  search_result = page.form_with(:name => 'gbqf') do |search|
    search.q = 'ruby mechanize'
  end.submit

  search_result.search(".//a[@class='l']").each_with_index.map do |link, i|
    puts "#{i} #{link.content}"
  end
end