Question

I’ve got a remote CSV that updates every few hours, it has 1m+ rows and I’m trying to work out if I can download the file line by line and parse it (I’m only interested in the first 500,000 lines) rather than have to download the whole file then process it.

Answer 1

You could try something like this…

require 'csv'
require 'open-uri'

def read(url)
  open(url) do |f|
    f.each_line do |l|
      CSV.parse(l) do |row|
        p [row] 
      end
    end 
  end
end