In my Rails 3.1 project, deployed to Heroku, I have a rake task that generates a few hundred thousand small PNGs based on records in the database and uploads the results to S3 using fog, one file at a time. (Before I was using heroku, the task saved the files locally and everything worked fine.) When I run the task using heroku run:detached, it runs for a while, and uploads a bunch of files, but eventually fails with Connection reset by peer (Errno::ECONNRESET). What’s weird about this is that I’ve modified the script to catch and retry that exception like so:
file_exists = false
retry_count = 10
begin
file_exists = directory.files.head(filekey)
rescue Errno::ECONNRESET => e
if retry_count > 0 then
logger.warning("Connection reset: #{e.inspect}")
retry_count -= 1
sleep(1)
retry
else
raise
end
end
According to the stack trace, file_exists = directory.files.head(filekey) is the line of my code that leads to the exception, so I feel like I’m missing something about Ruby exception handling. I’d love to know what I’m doing wrong, since I’m not 100% sure if this is a problem with my begin/rescue block, the way fog handles network errors, or what.
Here’s the complete stack trace:
Nov 07 12:08:01 ... app/run.1: Connection reset by peer (Errno::ECONNRESET)
Nov 07 12:08:01 ... app/run.1: /app/vendor/ruby-1.8.7/lib/ruby/1.8/openssl/buffering.rb:36:in `sysread'
Nov 07 12:08:01 ... app/run.1: /app/vendor/ruby-1.8.7/lib/ruby/1.8/openssl/buffering.rb:36:in `fill_rbuff'
Nov 07 12:08:01 ... app/run.1: /app/vendor/ruby-1.8.7/lib/ruby/1.8/openssl/buffering.rb:68:in `read'
Nov 07 12:08:01 ... app/run.1: /app/vendor/bundle/ruby/1.8/gems/excon-0.16.7/lib/excon/socket.rb:121:in `read'
Nov 07 12:08:01 ... app/run.1: /app/vendor/ruby-1.8.7/lib/ruby/1.8/timeout.rb:67:in `timeout'
Nov 07 12:08:01 ... app/run.1: /app/vendor/bundle/ruby/1.8/gems/excon-0.16.7/lib/excon/socket.rb:120:in `read'
Nov 07 12:08:01 ... app/run.1: /app/vendor/bundle/ruby/1.8/gems/excon-0.16.7/lib/excon/ssl_socket.rb:79:in `read'
Nov 07 12:08:01 ... app/run.1: /app/vendor/bundle/ruby/1.8/gems/excon-0.16.7/lib/excon/response.rb:21:in `parse'
Nov 07 12:08:01 ... app/run.1: /app/vendor/bundle/ruby/1.8/gems/excon-0.16.7/lib/excon/connection.rb:275:in `request_kernel'
Nov 07 12:08:01 ... app/run.1: /app/vendor/bundle/ruby/1.8/gems/excon-0.16.7/lib/excon/connection.rb:103:in `request'
Nov 07 12:08:01 ... app/run.1: /app/vendor/bundle/ruby/1.8/gems/fog-1.6.0/lib/fog/core/connection.rb:20:in `request'
Nov 07 12:08:01 ... app/run.1: /app/vendor/bundle/ruby/1.8/gems/fog-1.6.0/lib/fog/aws/storage.rb:392:in `request'
Nov 07 12:08:01 ... app/run.1: /app/vendor/bundle/ruby/1.8/gems/fog-1.6.0/lib/fog/aws/requests/storage/head_object.rb:53:in `head_object'
Nov 07 12:08:01 ... app/run.1: /app/vendor/bundle/ruby/1.8/gems/fog-1.6.0/lib/fog/aws/models/storage/files.rb:98:in `head'
Nov 07 12:08:01 ... app/run.1: /app/lib/tasks/assets.rake:45
Nov 07 12:08:01 ... app/run.1: /app/lib/tasks/assets.rake:37:in `each'
Nov 07 12:08:01 ... app/run.1: /app/lib/tasks/assets.rake:37
Nov 07 12:08:01 ... app/run.1: /app/vendor/bundle/ruby/1.8/gems/activerecord-3.1.8/lib/active_record/relation/batches.rb:21:in `find_each'
Nov 07 12:08:01 ... app/run.1: /app/vendor/bundle/ruby/1.8/gems/activerecord-3.1.8/lib/active_record/relation/batches.rb:21:in `each'
Nov 07 12:08:01 ... app/run.1: /app/vendor/bundle/ruby/1.8/gems/activerecord-3.1.8/lib/active_record/relation/batches.rb:21:in `find_each'
Nov 07 12:08:01 ... app/run.1: /app/vendor/bundle/ruby/1.8/gems/activerecord-3.1.8/lib/active_record/relation/batches.rb:69:in `find_in_batches'
Nov 07 12:08:01 ... app/run.1: /app/vendor/bundle/ruby/1.8/gems/activerecord-3.1.8/lib/active_record/relation/batches.rb:20:in `find_each'
Nov 07 12:08:01 ... app/run.1: /app/lib/tasks/assets.rake:35
Nov 07 12:08:01 ... app/run.1: /app/vendor/bundle/ruby/1.8/gems/rake-0.9.2.2/lib/rake/task.rb:205:in `call'
Nov 07 12:08:01 ... app/run.1: /app/vendor/bundle/ruby/1.8/gems/rake-0.9.2.2/lib/rake/task.rb:205:in `execute'
Nov 07 12:08:01 ... app/run.1: /app/vendor/bundle/ruby/1.8/gems/rake-0.9.2.2/lib/rake/task.rb:200:in `each'
Nov 07 12:08:01 ... app/run.1: /app/vendor/bundle/ruby/1.8/gems/rake-0.9.2.2/lib/rake/task.rb:200:in `execute'
Nov 07 12:08:01 ... app/run.1: /app/vendor/bundle/ruby/1.8/gems/rake-0.9.2.2/lib/rake/task.rb:158:in `invoke_with_call_chain'
Nov 07 12:08:01 ... app/run.1: /app/vendor/ruby-1.8.7/lib/ruby/1.8/monitor.rb:242:in `synchronize'
Nov 07 12:08:01 ... app/run.1: /app/vendor/bundle/ruby/1.8/gems/rake-0.9.2.2/lib/rake/task.rb:151:in `invoke_with_call_chain'
Nov 07 12:08:01 ... app/run.1: /app/vendor/bundle/ruby/1.8/gems/rake-0.9.2.2/lib/rake/task.rb:144:in `invoke'
Nov 07 12:08:01 ... app/run.1: /app/vendor/bundle/ruby/1.8/gems/rake-0.9.2.2/lib/rake/application.rb:116:in `invoke_task'
Nov 07 12:08:01 ... app/run.1: /app/vendor/bundle/ruby/1.8/gems/rake-0.9.2.2/lib/rake/application.rb:94:in `top_level'
Nov 07 12:08:01 ... app/run.1: /app/vendor/bundle/ruby/1.8/gems/rake-0.9.2.2/lib/rake/application.rb:94:in `each'
Nov 07 12:08:01 ... app/run.1: /app/vendor/bundle/ruby/1.8/gems/rake-0.9.2.2/lib/rake/application.rb:94:in `top_level'
Nov 07 12:08:01 ... app/run.1: /app/vendor/bundle/ruby/1.8/gems/rake-0.9.2.2/lib/rake/application.rb:133:in `standard_exception_handling'
Nov 07 12:08:01 ... app/run.1: /app/vendor/bundle/ruby/1.8/gems/rake-0.9.2.2/lib/rake/application.rb:88:in `top_level'
Nov 07 12:08:01 ... app/run.1: /app/vendor/bundle/ruby/1.8/gems/rake-0.9.2.2/lib/rake/application.rb:66:in `run'
Nov 07 12:08:01 ... app/run.1: /app/vendor/bundle/ruby/1.8/gems/rake-0.9.2.2/lib/rake/application.rb:133:in `standard_exception_handling'
Nov 07 12:08:01 ... app/run.1: /app/vendor/bundle/ruby/1.8/gems/rake-0.9.2.2/lib/rake/application.rb:63:in `run'
Nov 07 12:08:01 ... app/run.1: /app/vendor/bundle/ruby/1.8/gems/rake-0.9.2.2/bin/rake:33
Nov 07 12:08:01 ... app/run.1: /app/vendor/bundle/ruby/1.8/bin/rake:23:in `load'
Nov 07 12:08:01 ... app/run.1: /app/vendor/bundle/ruby/1.8/bin/rake:23
Nov 07 12:08:01 ... app/run.1: Tasks: TOP => ...:make_outlines
Nov 07 12:08:02 ... heroku/run.1: Process exited with status 1
Nov 07 12:08:02 ... heroku/run.1: State changed from up to complete
perhaps try like this: