Question

I’m currently developing an application that needs to parse some (a big file of) csv data. I have to perform several steps to manipulate and map the data to ActiveRecord objects. This has to be done in several steps, and i have to display the results in a view.

So i start at imports#index. Here i let the users upload the csv file, nothing fancy

Then i go trough imports#create to create the import object.

Then i’m at step2 imports#show. Here i display the results. So in import#show i parsed all the csv data and delete the rows that i dont need. (It’s one big array with structs in it.)

Then i want to go to step 3 to manipulate the same data further. But how in the world can i get the big array from step2 in step3 without a session (because of the Structs i cant insert it in a session it gives me the following error: TypeError (can't dump anonymous class #<Class:0x105477128>):

And when i put it in a session with .inspect it raises this error: ActionDispatch::Cookies::CookieOverflow. But then it’s all one big string and i want to maintain the ruby functionality (e.g. it’s 1 big array with struct objects in it…)

So to wrap it all up

Step 1
upload file
Step2
parse data & manipulate it.
Step3
Use the data from step2 (without a session..)
Step4
User data from step3

How in the world can i get this to work… I can’t seem to figure it out :<

Answer 1

You should keep in mind different principles:

• Controllers are not meant to deal with data, Models or Classes do

• parsing a big file shouldn’t be made inside the controller but put inside some queue (Delayed Job handles this properly, see how it works here). Just imagine your user waiting endlessly for something to happen…

To sum up:

• Step 1: upload file => ok I guess you save it, great

• Step2 parse data & manipulate it => should be triggered by step 1 and launched in a Delayed Job queue

• Step3 Use the data from step2 (without a session..) => all data processed the way I suggest could be stored in cache, database or wherever you find it useful

• Step4 User data from step3 => you want to display the result of your processing, I suggest you implement an AJAX request checking if the job was done or not (to do that, I highly recommend you store the info in cache in order not to query your database too often).