Question

We’re building a CSV export mechanism in a CMS we just built and we’re having trouble keeping memory down on result sets with millions of rows. What is the best way to write a 1MM+ row result from MySQL (via the mysql2 gem) to CSV. If it makes any difference, we’re on Ruby 1.9.3p194 and MySQL 5.5. Our main goal is to keep memory consumption down on the Ruby process.

Answer 1

By default mysql2 caches the ruby objects that it creates for its result set. If you turn this off then memory consumption should be greatly reduced. for example you might do

require 'mysql2'
client = Mysql2::Client.new(:host => "localhost", :username => "someuser")
client.query 'use some_db'

CSV.open('output.csv', 'w') do |csv|
  results = client.query("SELECT * FROM big_table")
  results.each(:cache_rows => false) do |row| 
    csv << [row['column_a'], row['column_b']]
  end
end