Question

I have a simple text file that is ~150mb. My code will read each line, and if it matches certain regexes, it gets written to an output file.
But right now, it just takes a long time to iterate through all of the lines of the file (several minutes) doing it like

File.open(filename).each do |line|
  # do some stuff
end

I know that it is the looping through the lines of the file that is taking a while because even if I do nothing with the data in “#do some stuff”, it still takes a long time.

I know that some unix programs can parse large files like this almost instantly (like grep), so I am wondering why ruby (MRI 1.9) takes so long to read the file, and is there some way to make it faster?

Answer 1

File.readlines.each do |line|
  #do stuff with each line
end

Will read the whole file into one array of lines. It should be a lot faster, but it takes more memory.