Question

I have a problem with file uploading on Heroku server.
(Also a hint about the right way of doing such type of things with rails would be greatly appreciated – I’m very new in RoR).

All this code is about uploading some CSV file, then allowing user to tweak couple of settings, and parse file after all. This usually work on localhost (several times I get troubles with value stored in session), but on Heroku it is always die on upload.

In one of the neighborhood questions was written that Heroku store file only during singe instance run, but I still couldn’t find anything about this in Heroku’s docs. Should I store file data in the db right after upload, so in such case it always be available? The downside – files could be pretty big, about 10-20Mb, it isn’t looks nice.

Heroku logs say:

 2012-05-21T19:27:20+00:00 app[web.1]: Started POST "/products/upload" for 46.119
.175.140 at 2012-05-21 19:27:20 +0000
2012-05-21T19:27:20+00:00 app[web.1]:   Processing by ProductsController#upload
as HTML
2012-05-21T19:27:20+00:00 app[web.1]:   Parameters: {"utf8"=>"тЬУ", "authenticit
y_token"=>"aqJFg3aqENfxS2lKCE4o4txxkZTJgPx36SZ7r3nyZBw=", "upload"=>{"my_file"=>
#<ActionDispatch::Http::UploadedFile:0x000000053af020 @original_filename="marina
-AutoPalmaPriceList_2011-07-30.txt", @content_type="text/plain", @headers="Conte
nt-Disposition: form-data; name=\"upload[my_file]\"; filename=\"marina-AutoPalma
PriceList_2011-07-30.txt\"\r\nContent-Type: text/plain\r\n", @tempfile=#<File:/t
mp/RackMultipart20120521-1-10g8xmx>>}, "commit"=>"Upload"}
2012-05-21T19:27:20+00:00 app[web.1]:
2012-05-21T19:27:20+00:00 app[web.1]: LoadError (no such file to load -- CSV):
2012-05-21T19:27:20+00:00 app[web.1]:   app/controllers/products_controller.rb:8
2:in `upload'
2012-05-21T19:27:20+00:00 app[web.1]:
2012-05-21T19:27:20+00:00 app[web.1]:
2012-05-21T19:27:20+00:00 app[web.1]: cache: [POST /products/upload] invalidate,
 pass

The code itself:

ProductsController:

def import
  respond_to do |format|
    format.html
  end
end

def import_adjust
  case params[:commit]
    when "Adjust"
      @col_default = params[:col_data]
      #abort @col_default.to_yaml
      #update csv reader with form data, restore filters  from params
    when "Complete"
      #all ok, read the whole file
      #abort params.to_yaml
      redirect_to import_complete
    else
      @col_default = nil
  end
  #read first part of the file
  @tmp = session[:import_file]
  @csv = []
  source = CSV.open @tmp, {col_sep: ";"}

  5.times do
    line = source.readline
    if line.size>0
      @line_size = line.size
      @csv.push line
    end
  end

  #generate a selection array
  #selection = select_tag 'col_data[]', options_for_select([['name','name'], ['brand','brand'], ['delivery_time','delivery_time'], ['price','price']])
  #@csv = [selection * line_size] + @csv
end

def import_complete
  #remove all items
  #todo check products with line items will not be destroyed.
  Product.destroy_all
  #abort params.to_yaml
  map = {}
  cnt = 0
  #todo check for params count.
  params[:col_data].each do |val|
    map[cnt] = val if val != 'ignore'
    cnt += 1
  end

  source = CSV.open session[:import_file], {col_sep: ';'}
  source.each do |row|
    cnt += 1
    if row.size > 0
      item = Product.new
      map.each do |col, attr|
        item[attr] = row[col]
      end
      item[:provider_id] = params[:adjust][:provider]
      item.save
      #abort item.to_yaml
    end
  end

  #abort map.to_yaml
  #todo response needed.
end

def upload
  require 'CSV' #looks like I dont need this in fact.
  @tmp = params[:upload][:my_file].path #tempfile

  @csv = []
  #source = CSV.open @tmp, {col_sep: ";"}

  session[:import_file] = params[:upload][:my_file].path

  respond_to do |format|
    format.html { redirect_to action: 'import_adjust' }
  end
end

upload.html.erb:

<h1>Uploaded</h1>
<%= @tmp %>

<% @csv.each do |val| %>
    <%= val %>
<% end %>

_form_import.html.erb:

<%= form_for :upload, :html => {:multipart => true}, :url => {action: "upload"} do |f| %>
<%= f.file_field :my_file %>
<%= f.submit "Upload" %>
<% end %>

import_adjust.html.erb:

<h1>New product</h1>


<%= form_for :adjust, :url => {action: "import_adjust"} do |f| %>
<% if @csv %>
<table>
  <tr>
    <% @line_size.times do |cnt| %>
        <td>
        <%= select_tag 'col_data[]',
               options_for_select([
                  ['--ignore--', 'ignore'],
                  ['name','name'],
                  ['brand','brand'],
                  ['delivery_time','delivery_time'],
                  ['price','price']
        ], @col_default!=nil ? @col_default[cnt] : nil) %>
        </td>
    <% end %>
  </tr>

  <% @csv.each do |val| %>
     <tr>
     <% val.each do |cell| %>
       <td>
         <%= cell %>
       </td>
     <% end %>
     </tr>
  <% end %>
</table>
<% end %>


  <%= f.label :delimiter, 'Разделитель' %>
  <%= f.text_field :delimiter %>
  <br>
  <%= f.label :provider, 'Поставщик' %>
  <%#todo default empty option needed! Human mistakes warning! %>
  <%= f.select :provider, Provider.all.collect { |item| [item.name, item.id] } %>
  <br>
  <%= f.label :delimiter, 'Разделитель' %>
  <%= f.text_field :delimiter %>
  <br>
  <%# Adjust for proceed adjusting or Complete  for parsing %>
  <%= f.submit "Adjust" %>
  <%= f.submit "Complete" %>
<% end %>


<%= link_to 'Back', products_path %>

Answer 1

I have an identical scenario as Lifecoder where a user uploads a file, names the columns using a map_fields plugin (by Andrew Timberlake), and then the file is parsed and processed. Here’s how I handle it:

  file_field = params[options[:file_field]]
  map_fields_file_name = "map_fields_#{Time.now.to_i}_#{$$}"

  bucket = S3.buckets[CSV_COUPON_BUCKET_NAME]    # gets an existing bucket
  obj = bucket.objects[map_fields_file_name]
  obj.write( file_field.read )

  # Save the name and bucket to retrieve on second pass
  session[:map_fields][:bucket_name] = map_fields_file_name

Then on the second pass to process the file, I open the file and read it back into temp for the dyno to process:

    # Get CSV data out of bucket and stick it back into temp, so we pick up where
    # we left off as far as map_fields is concerned.
    bucket = S3.buckets[CSV_COUPON_BUCKET_NAME]
    obj = bucket.objects[session[:map_fields][:bucket_name]]
    temp_path = File.join(Dir::tmpdir, "map_fields_#{Time.now.to_i}_#{$$}")
    File.open(temp_path, 'wb') do |f|
      f.write obj.read
    end

I had to use the plugin so I could modify the code, as obviously the gem is handled by Heroku and doesn’t allow for modifications.