Question

https://devcenter.heroku.com/articles/request-timeout

---

30 Seconds and a timeout error fires according to their documentation.

I’m uploading and parsing a CSV file to save to my database. One of those files is 1.7MB in size and has 37000 rows.

This process takes a bit long to process, certainly more than 30 seconds.

What can I do in these cases? What options do I have?

require 'csv'

class DatabaseImporterController < ApplicationController
  def index
  end

  def import
    # Receive the uploaded CSV file and import to the database.
    csv_file = params[:csv_file].tempfile

    i = 0
    CSV.foreach(csv_file) do |row|
      # Structure for CSV file: Year, Make, Model, Trim
      if i > 0 then
        make = Make.find_or_create_by_name(row[1])
        model = make.model.create(:year => row[0], :name => row[2], :trim => row[3])
      end

      i += 1
    end
    redirect_to :action => 'list'
  end

  def list
    @models = Model.all
  end
end

Answer 1

Instead of processing your CSV file in the controller have it push a notification to a queue with the location of the uploaded file. Then have a worker dyno handle that processing.

You’ll pay a little bit more, especially if you’re trying to stick with the free single dyno tier, but this is a scalable design (which is why I’d imagine there’s a 30 second timeout on HTTP processing).

An alternative is to push the data directly into a table and execute a stored procedure asynchronously. This pushes the work off to Postgres to handle off the HTTP thread and may place your request under the 30 second time limit, though with larger files you may breech this cap anyway.

Before you bother restructuring your entire application you’ll want to run a test to ensure that Heroku has not disabled libpq-asynch.

The big cost here in your code above is Make.find_or_create_by_name which is invoking 37,000 separate – as per your example input – SELECT and possibly an INSERT for each row in your CSV. If libpq-asynch is not an option you’ll have to create a stored procedure that will perform this functionality in batches of 100 or 1000 rows at a time – that way your controller code isn’t making so many round trips to the database. Postgres supports arrays in the classical ordinal index style as well as arrays of row types so this is actually much less painful than it sounds.