Question

I am reading a file that is 10mb in size and which contains some id’s. I read them into a list in ruby. I am concerned that it might cause memory issues in the future, when the number of id’s in file might increase. Is there a effective way of reading a large file in batches?

Thank you

Answer 1

there’s no universal way.

1) you can read file by chunks:

File.open('filename','r') do |f|
  chunk = f.read(2048)
  ...
end

disadvantage: you can miss a substring if it’d be between chunks, i.e. you look for “SOME_TEXT”, but “SOME_” is a last 5 bytes of 1st 2048-byte chunk, and “TEXT” is a 4 bytes of 2nd chunk

2) you can read file line-by-line

File.open('filename','r') do |f|
  line = f.gets
  ...
end

disadvantage: this way it’d be 2x..5x slower than first method