Question

My application scrapes information from various sites using Mechanize. Naturally, each site requires custom Mechanize code. Each site is stored in my database, including the url to scrape and the string name of an .rb file containing that site’s Mechanize code. For this case, let’s assume the scripts are available in the assets folder.

I would like to call http://example.com/site/:id, then have the show action dynamically choose which Mechanize script to run (say, @site.name + ".rb" ). The script will massage the data into a common model, so all sites can use the same show template.

I can’t find a way to dynamically load a .rb script within an action and obtain the result. It may be easier to have the scripts return a JSON string, which I can parse before passing on to the template, but I can’t see a solution for that either. Ideally, the script will run in the action’s scope. The ugly solution is an enormous if-else chain (testing the site name to determine which code block to run), but there must be a better way.

Any suggestions would be greatly appreciated, as would any general solutions to running different code dependent upon the properties of database objects.

Answer 1

If you have all the code in your app already why are you eval’ing Ruby code?

Create classes like:

class GoogleSpider < Spider; end
class NewYorkTimesSpider < Spider; end
class SomeOtherSpider < Spider; end

And the site class will hold the class name that will be used, so you would be able to easily do something like this in your controller action:

def show
  @site = Site.find(params[:id])
  # name contains SomeOtherSpider
  @process_output = @site.name.constantize.new.process
  # do something with the output here
end

And then you don’t need to mess around evaluating Ruby code, just call the class needed. You can even make them all singletons or keep them all in a hash for faster access.