Question

I am using the Ruby gem mechanize to scrape some html… When I load my page and display the necessary results, the page laods fine. After a reload, I get this error when doing “search_results = @agent.submit(search_form)”:

undefined method `<=>' for {emptyelem <input name="hl" value="en" type="hidden">}:Hpricot::Elem

Before I post any code, just does this ring any bells?

Thanks.

Code:

    start = Time.now

    # initial set up
    @agent = Mechanize.new
    Mechanize.html_parser = Hpricot
    page = @agent.get("http://www.google.com/")
    search_form = page.forms.first

    # conduct initial search
    @search_term = search_form.q = params[:search].to_s
    search_results = @agent.submit(search_form)

    # helper variables
    search_qs = ""; @page_number = 1; i = 0; @flag = false;

    # get the query string structure
    search_results.links.each { |li| search_qs = li.href if li.href.match(/.*search\?q=.*start=.*/) }

    # search through all paginated pages
    while (i < 500)
      search_qs = search_qs.gsub(/start=\d+/,"start=#{i}")
      @search_url = "http://google.com#{search_qs}"
      search_results = @agent.get(@search_url)
      search_results.links.each { |li| @flag = true if li.text.match("All Bout Texas Tailgating") }
      break if @flag
      i+=10; @page_number+=1
    end

@execution_time = Time.now-start

render :layout => false

---

VIEW:

<h2>Query results for "<%= @search_term %>" on Google</h2>

<% if @flag %>
    <p>What page is this keyword found: <b><%= @page_number %></b></p>
    <p><%= link_to  "Click to see page", "#{@search_url}", {:target => "_blank"} %></p>
    <p>How long did this query take to run?: <%= @execution_time %> seconds</p>
<% else %>
    <p>Keyword not found in Google search reults</p>
<% end %>

STACK TRACE:

 NoMethodError (undefined method `<=>' for {emptyelem <input name="hl" value="en" type="hidden">}:Hpricot::Elem):
  mechanize (1.0.0) lib/mechanize/form/field.rb:30:in `<=>'
  mechanize (1.0.0) lib/mechanize/form.rb:171:in `sort'
  mechanize (1.0.0) lib/mechanize/form.rb:171:in `build_query'
  mechanize (1.0.0) lib/mechanize.rb:373:in `submit'
  app/controllers/admin/importer_controller.rb:24:in `check_page_rank'
  /opt/local/lib/ruby/1.8/webrick/httpserver.rb:104:in `service'
  /opt/local/lib/ruby/1.8/webrick/httpserver.rb:65:in `run'
  /opt/local/lib/ruby/1.8/webrick/server.rb:173:in `start_thread'
  /opt/local/lib/ruby/1.8/webrick/server.rb:162:in `start'
  /opt/local/lib/ruby/1.8/webrick/server.rb:162:in `start_thread'
  /opt/local/lib/ruby/1.8/webrick/server.rb:95:in `start'
  /opt/local/lib/ruby/1.8/webrick/server.rb:92:in `each'
  /opt/local/lib/ruby/1.8/webrick/server.rb:92:in `start'
  /opt/local/lib/ruby/1.8/webrick/server.rb:23:in `start'
  /opt/local/lib/ruby/1.8/webrick/server.rb:82:in `start'

Rendered rescues/_trace (98.4ms)
Rendered rescues/_request_and_response (1.2ms)
Rendering rescues/layout (internal_server_error)

Answer 1

So if you look through the source for mechanize in form.rb – form submitting is calling a function called build_query which sorts the fields on the form. Since sort uses the <=> operator, and it’s undefined on Hpricot elements, you are getting an exception.

It seems as if mechanize was built to use Nokogiri – it may have unfixed bugs with other parsing implementations. I did not get too deep into mechanize’s source and don’t want to blame anyone, but you may want to try switching to Nokogiri for this project (if possible). It doesn’t seem from this snippet as if you’re relying heavily on Hpricot. It seems strange to me that mechanize is throwing an exception on a hidden form field from Hpricot, but the stack trace is pretty clear in this regard.

Your other major option is to hop into the mechanize source and see if you can fix it yourself (or file a bug on the mechanize github and hope somebody gets to it).

Good luck.