Question

I desperately need some help. I’ve been tasked with throwing together a utility for one of our teams to use to do some basic analysis on data files. I’ve been away from coding on a daily basis for a while now, so I’m a little rusty. I’ll try to be concise in describing my problem.

Scenario: List Operations team creates data files for email marketing campaigns. Marketing campaigns are multi-segmented; each segment identified by a keycode. The total number of records averages around 450,000 and file size around 20MB.

We’re having serious junking issues right now so the marketing dept would like some basic analysis on the number of records with gmail, yahoo, and hotmail addresses by segment.

Current Status: I have a Rails 3 app created that allows the user to upload multiple files and for those records to be saved in a MySQL database.

Here’s a very basic sample of what the database records would look like, not including standard fields like id and the timestamps:

domain       eid    keycode  
YAHOO.COM    42886  GY4103  
HOTMAIL.COM  53012  GY4103  
YAHOO.COM    53952  GY4103  
AOL.COM      54327  GY4103  
COMCAST.NET  55134  MX4155  
GMAIL.COM    59157  MX4155  
AOL.COM      59823  MX4155  
AOL.COM      60384  MX4161  
MSN.COM      64720  MX4161  
YAHOO.COM    67790  MX4161
GMAIL.COM    73537  MX4161
YAHOO.COM    76747  MX4161  
HOTMAIL.COM  71467  MX4161
GMAIL.COM    84280  MX4161  

Problem: I need to present the analysis to the end user as:

Keycode
GY4103
  Domain       Count
  YAHOO.COM    2
  HOTMAIL.COM  1
  OTHER        1
MX4155
  Domain       Count
  GMAIL.COM    1
  OTHER        2
MX4161
  Domain       Count
  YAHOO.COM    2
  HOTMAIL.COM  1
  GMAIL.COM    2
  OTHER        2

If I were in something like Crystal Reports, I would just group by the keycode, then by domain, and then a count of the eid’s under that, do some tweaking to show only the 3 domains I want and everything else as “other” and I’m done! However, using CR isn’t possible here.

So I’m assuming I’m after creating some type of tree structure or hierarchy with Keycode at the top level, then domain under that, and a count under that?? I started taking a look at the Ancestry gem, but I’m not really getting it.

Any help that anyone could provide would be much appreciated, especially considering I need to deliver this by noon tomorrow.

Answer 1

I would probably mix in a little manual SQL in this since it seems to be such a large dataset to handle. Perhaps something like this:

# CONTROLLER
# Load the data from the database and group on keycode and domain.
# You will only get one record for each unique combination of domain and keycode
@records = MyModel.select("keycode, domain, COUNT(eid) AS eids").group("keycode, domain")

# convert the array of records into a hash where every key is a unique keycode
# and the value is an array of records for all the domains for that keycode
@records = @records.group_by(&:keycode)

# VIEW
<% @records.keys.each do |keycode| %>
  <h1><%= keycode %></h1>
  <% @records[keycode].each do |record| %>
    <p><%= record.domain %>: <%= record.eids %></p>
  <% end %>
<% end %>

I hope I understood everything correctly