Question

I am creating an import feature that imports CSV files into several tables. I made a module called CsvParser which parses a CSV file and creates records. My models that receive the create actions extends theCsvParser. They make a call to CsvParser.create and pass the correct attribute order and an optional lambda called value_parser. This lambda transforms values in a hash to a preffered format.

class Mutation < ActiveRecord::Base
  extend CsvParser

  def self.import_csv(csv_file)
    attribute_order = %w[reg_nr receipt_date reference_number book_date is_credit sum balance description]

    value_parser = lambda do |h|
      h["is_credit"] = ((h["is_credit"] == 'B') if h["is_credit"].present?)
      h["sum"] = -1 * h["sum"].to_f unless h["is_credit"]
      return [h]
    end

    CsvParser.create(csv_file, self, attribute_order, value_parser)    
  end
end

The reason that I’m using a lambda instead of checks inside the CsvParser.create method is because the lambda is like a business rule that belongs to this model.

My question is how i should test this lambda. Should i test it in the model or the CsvParser? Should i test the lambda itself or the result of an array of the self.import method? Maybe i should make another code structure?

My CsvParser looks as follows:

require "csv"

module CsvParser

  def self.create(csv_file, klass, attribute_order, value_parser = nil)
    parsed_csv = CSV.parse(csv_file, col_sep: "|")

    records = []    

    ActiveRecord::Base.transaction do
      parsed_csv.each do |row|
        record = Hash.new {|h, k| h[k] = []}      
        row.each_with_index do |value, index|
          record[attribute_order[index]] = value
        end 
        if value_parser.blank?
          records << klass.create(record)
        else
          value_parser.call(record).each do |parsed_record|
            records << klass.create(parsed_record)
          end
        end
      end 
    end
    return records
  end

end

I’m testing the module itself:
require ‘spec_helper’

describe CsvParser do

  it "should create relations" do
    file = File.new(Rails.root.join('spec/fixtures/files/importrelaties.txt'))
    Relation.should_receive(:create).at_least(:once)
    Relation.import_csv(file).should be_kind_of Array 
  end

  it "should create mutations" do
    file = File.new(Rails.root.join('spec/fixtures/files/importmutaties.txt'))
    Mutation.should_receive(:create).at_least(:once)    
    Mutation.import_csv(file).should be_kind_of Array
  end

  it "should create strategies" do
    file = File.new(Rails.root.join('spec/fixtures/files/importplan.txt'))
    Strategy.should_receive(:create).at_least(:once)
    Strategy.import_csv(file).should be_kind_of Array
  end

  it "should create reservations" do
    file = File.new(Rails.root.join('spec/fixtures/files/importreservering.txt'))
    Reservation.should_receive(:create).at_least(:once)
    Reservation.import_csv(file).should be_kind_of Array
  end

end

Answer 1

Some interesting questions. A couple of notes:

1. You probably shouldn’t have a return within the lambda. Just make the last statement [h].

2. If I understand the code correctly, the first and second lines of your lambda are overcomplicated. Reduce them to make them more readable and easier to refactor:

   h["is_credit"] = (h['is_credit'] == 'B') # I *think* that will do the same
   h['sum'] = h['sum'].to_f   # Your original code would have left this a string
   h['sum'] *= -1 unless h['is_credit']
   

3. It looks like your lambda doesn’t depend on anything external (aside from h), so I would test it separately. You could even make it a constant:

   class Mutation < ActiveRecord::Base
     extend CsvParser    # <== See point 5 below
   
     PARSE_CREDIT_AND_SUM = lambda do |h|
       h["is_credit"] = (h['is_credit'] == 'B') 
       h['sum'] = h['sum'].to_f
       h['sum'] *= -1 unless h['is_credit']
       [h]
     end
   

4. Without knowing the rationale, it’s hard to say where you should put this code. My gut instinct is that it is not the job of the CSV parser (although a good parser may detect floating point numbers and convert them from strings?) Keep your CSV parser reusable. (Note: Re-reading, I think you’ve answered this question yourself – it is business logic, tied to the model. Go with your gut!)

5. Lastly, you are defining and the method CsvParser.create. You don’t need to extend CsvParser to get access to it, although if you have other facilities in CsvParser, consider making CsvParser.create a normal module method called something like create_from_csv_file