Question

I am facing some issues when downloading large files in Heroku. I have to download and parse files greater than 1Gb. What I am trying to do right now, is use curl to download them into /tmp folder (of a Rails application).

The curl command is: “curl –retry 999 -o #{destination} #{uri} 2> /dev/null” and destination is Rails.root.join(“tmp”, “file.example”)

The problem is that after a few minutes downloading, the “curl” process that is downloading the file is finished, way far from the download is finished. Before being finished, the logs show lots of “Memory exceeded”. This led me to the thinking that when I am saving to /tmp folder, it is storing the downloaded content in the memory and when it memory hit its limit, the process is killed.

I would like to know if any of you have already experienced a similar issue on Heroku and if saving to /tmp folder really works like this. If so, do you have any suggestions to get this working at Heroku?

thanks,
Elvio

Answer 1

You are probably better off saving the file in an external cloud provider like S3 using the fog gem. In any case, Heroku is a read only filesystem, so they won’t allow you to curl, must less write to it.