Question

In a MVC patterned framework where would a screen-scraping module most logically be located? In the model or the controller? Or is it completely outside of this pattern?

Answer 1

You can call it as you might a model if you design it to behave like one. Then it can be easily used within a controller:

def update
  @company = Company.find(params[:id])

  @scraper = Scraper.find(:page => some_url, :method => :rip)

  @scraper.product_details.each do |params|
    @company.products.create(params)
  end
end

It is always convenient when you design the output format of your scraper to be compatible with some other object, such as the column mapping of your persistent storage.

It’s often handy to have “model-like” objects. In this case, you can think of a scraper as an interface to another web site instead of a database, and in that respect it is not unlike ActiveResource. There’s no rule that a model has to be built off of ActiveRecord.