Question

I want to aggregate data from different sources, Twitter lastfm and that sort. I just can’t figure out how to store the data. Clearly in a database but I can’t figure out how abstract to make the table to hold all this data without compromising the logical understanding of the data in each column.
I was wondering if anybody else had experience with this and now they tackled it in rails.

Answer 1

One option, if you want to stick with SQL, would be to have a Model/Table which contains fields common to every data source (title, url, summary) which is associated to other Models/Tables which contain the fields specific to individual data sources. The associations could be regular or polymorphic. And if you wanted to get in to some metaprogramming you could use method_missing to delegate method calls for fields not present in the ‘common’ Model to the associated models. This would work best with a polymorphic join. Psudeo-code:

class DataSource
  belongs_to :data_source_extension, :polymorphic => true

  def method_missing(method)
    if data_source_extension.responds_to? method
      data_source_extension.send(method) 
    else
      super
    end
  end
end

The other option would be STI, so one table with all fields and a ‘type’ field which tells Rails which model the record should be wrapped in. This depends on how many different sources you have and how different they are from each other.

If the fields don’t need to be searchable storing a Hash in a Text field works well. See Serialize and the attr_bucket gem.

Or if you want to trendy a NoSQL type database allows on-the-fly fields to be generated.