Question

I’m having an issue with an app I’m working on.
The app allows a user to upload a CSV file which gets processed by the app and in turn creates records into a number of tables. In order to improve performance, for one of the tables, it produces a new CSV file in order to use the mysql LOAD DATA INFILE functionality.

Instead, it seems to be increasing the time it takes to process.
I’m pushing all processing into the background using sidekiq. It seems to be creating the CSV without any problems, however when I execute the load data query it just sits there and I have no idea what it’s doing.

My processing function does the following :

CSV.open(output_path, 'w+', { force_quotes: true }) do |writer|
    writer << headers

    while rows.count > 0
      ....
      data_sets.each do |ds|
        writer << [UUIDTools::UUID.random_create, resp, row[set], ds.id, now, now]
        set += 1
      end
      resp += 1
    end
end

sql = "LOAD DATA LOCAL INFILE '#{output_path}'
       INTO TABLE data_set_responses
       FIELDS TERMINATED BY ',' ENCLOSED BY '\"'
       LINES TERMINATED BY '\n'
       (id, response_number, response, data_set_id, created_at, updated_at)"

con = ActiveRecord::Base.connection
con.execute("SET autocommit = 0;")
con.execute("SET unique_checks = 0;")
con.execute("SET foreign_key_checks = 0;")
con.execute("LOCK TABLES data_set_responses WRITE;")
con.execute(sql)
con.execute("UNLOCK TABLES;")
con.execute("COMMIT;")
con.execute("SET autocommit = 1;")
con.execute("SET unique_checks = 1;")
con.execute("SET foreign_key_checks = 1;")

As of right now, my sidekiq process has been running for 22 minutes and still hasn’t finished. It should be inserting around 700k rows which shouldn’t be taking anywhere near this long!

The table I’m inserting into has a binary field for it’s primary key (uuid) so I don’t know if that’s slowing it down?

Any ideas?

Answer 1

I ended up changing my data structure to one that didn’t require the vast number of rows that this structure did. I’ve got it down to a matter of seconds 🙂