Question

This might be a similar problem to my earlier two questions – see here and here but I’m trying to use the _detail command to automatically click the link so I can scrape the details page for each individual event.

The code I’m using is:

require 'rubygems'
require 'scrubyt'

nuffield_data = Scrubyt::Extractor.define do
  fetch 'http://www.nuffieldtheatre.co.uk/cn/events/event_listings.php'

  event do
    title 'The Coast of Mayo'
    link_url
    event_detail do
      dates "1-4 October"
      times "7:30pm"
    end
  end

  next_page "Next Page", :limit => 20
end

  nuffield_data.to_xml.write($stdout,1)

Is there any way to print out the URL that using the event_detail is trying to access? The error doesn’t seem to give me the URL that gave the 404.

Update: I think the link may be a relative link – could this be causing problems? Any ideas how to deal with that?

Answer 1

I had the same issue with relative links and fixed it like this… you have to set the :resolve param to the correct base url

  event do
    title 'The Coast of Mayo'
    link_url
    event_detail :resolve => 'http://www.nuffieldtheatre.co.uk/cn/events' do
      dates "1-4 October"
      times "7:30pm"
    end
  end