Question

I’m definitely a newbie to ruby (and using 1.9.1), so any help is appreciated. Everything I’ve learned about Ruby has been from using google. I’m trying to compare two arrays of hashes and due to the sizes, it’s taking way to long and flirts with running out of memory. Any help would be appreciated.

I have a Class (ParseCSV) with multiple methods (initialize, open, compare, strip, output).
The way I have it working right now is as follows (and this does pass the tests I’ve written, just using a much smaller data set):

file1 = ParseCSV.new(“some_file”)
file2 = ParseCSV.new(“some_other_file”)

file1.open #this reads the file contents into an Array of Hash’s through the CSV library 
file1.strip #This is just removing extra hash’s from each array index.  So normally there are fifty hash’s in each array index, this is just done to help reduce memory consumption.  

file2.open 
file2.compare(“file1.storage”) #@storage is The array of hash’s from the open method

file2.output

Now what I’m struggling with is the compare method. Working on smaller data sets it’s not a big deal at all, works fast enough. However in this case I’m comparing about 400,000 records (all read into the array of hashes) against one that has about 450,000 records. I’m trying to speed this up. Also I can’t run the strip method on file2. Here is how I’m doing it now:

def compare(x)
    #obviously just a verbose message
    puts "Comparing and leaving behind non matching entries"

    x.each do |row|
        #@storage is the array of hashes
        @storage.each_index do |y|       
            if row[@opts[:field]] == @storage[y][@opts[:field]]
               @storage.delete_at(y)
            end
       end
    end
end

Hopefully that makes sense. I know it’s going to be a slow process just because it has to iterate 400,000 rows 440,000 times each. But do you have any other ideas on how to speed it up and possibly reduce memory consumption?

Answer 1

Yikes, that’ll be O(n^2) runtime. Nasty.

A better bet would be to use the built in Set class.

Code would look something like:

require 'set'

file1_content = load_file_content_into_array_here("some_file")
file2_content = load_file_content_into_array_here("some_other_file")

file1_set = Set[file1_content]

unique_elements = file1_set - file2_content

That assumes that the files themselves have unique content. Should work in the generic case, but may have quirks depending on what your data looks like and how you parse it, but as long as the lines can be compared with == it should help you out.

Using a set will be MUCH faster than doing a nested loop to iterate over the file content.

(and yes, I have actually done this to process files with ~2 million lines, so it should be able to handle your case – eventually. If you’re doing heavy data munging, Ruby may not be the best choice of tool though)