Question

I’m building a Rails site that, among other things, allows users to build their own recipe repository. Recipes are entered either manually or via a link to another site (think epicurious, cooks.com, etc). I’m writing scripts that will scrape a recipe from these sites given a link from a user, and so far (legal issues notwithstanding) that part isn’t giving me any trouble.

However, I’m not sure where to put the code that I’m writing for these scraper scripts. My first thought was to put it in the recipes model, but it seems a bit too involved to go there; would a library or a helper be more appropriate?

Also, as I mentioned, I’m building several different scrapers for different food websites. It seems to me that the elegant way to do this would be to define an interface (or abstract base class) that determines a set of methods for constructing a recipe object given a link, but I’m not sure what the best approach would be here, either. How might I build out these OO relationships, and where should the code go?

Answer 1

You’ve got two sides of this thing that are obvious. The first is how you will store the recipes, which will be models. Obviously Models will not be scraping other sites, since they have a single responsibility: storing valid data. Your controller(s), which will initiate the scraping and storage process, should not contain the scraping code either (though they will call it).

While in Ruby we don’t go for abstract classes nor interfaces — it’s duck-typed, so it’s enough that your scrapers implement a known method or set of methods — your scraping engines should all be similar, especially in terms of the public methods they expose.

You will put your scrapers — and here’s the lame answer — wherever you want. lib is fine, but if you want to make a plugin that might not be a bad idea either. See my question here – with a stunning answer by famous Rails-guy Yehuda Katz – for some other ideas, but in general: there is no right answer. There are some wrong ones, though.