Question

I’m writing an app which needs to send many emails and creates many user notifications because of these emails. This task produces a timeout in Heroku. To solve this, I decided to use Resque and RedistToGo.

What I did was to send the email (it’s actually just one email because we use Sendgrid to handle this) and create the notifications using a Resque worker. The email is already created, so I send its id to the worker, along with all the recipients.

This works fine locally. In production, unless we restart our app in Heroku, it only works once. I will post some of my code and the error message:

#lib/tasks/resque.rake
require 'resque/tasks'

task "resque:setup" => :environment do
  ENV['QUEUE'] = '*'
end

desc "Alias for resque:work (To run workers on Heroku)"
task "jobs:work" => "resque:work"


#config/initalizers/resque.rb
ENV["REDISTOGO_URL"] ||= "redis://redistogo:some_hash@some_url:some_number/"

uri = URI.parse(ENV["REDISTOGO_URL"])
Resque.redis = Redis.new(:host => uri.host, :port => uri.port, :password => uri.password)

Dir["#{Rails.root}/app/workers/*.rb"].each { |file| require file }


#app/workers/massive_email_sender.rb
class MassiveEmailSender 
  @queue = :massive_email_queue

  def self.perform(email_id, recipients)
    email = Email.find(email_id.to_i)
    email.recipients = recipients
    email.send_email
  end

end

I’ve got an Email model which has an after_create that enqueues the worker:

class Email < ActiveRecord::Base

...

after_create :enqueue_email

  def enqueue_email
    Resque.enqueue(MassiveEmailSender, self.id, self.recipients)
  end

...

end

This Email model also has the send_email method which does what I said before

I’m getting the following error message. I’m gonna post all the information Resque gives to me:

Worker
    9dddd06a-2158-464a-b3d9-b2d16380afcf:1 on massive_email_queue at just now
    Retry or Remove
Class
    MassiveEmailSender
Arguments

    21
    ["[e-mail]", "[e-mail]"]

Exception
    ActiveRecord::StatementInvalid
Error
    PG::Error: SSL error: decryption failed or bad record mac : SELECT a.attname, format_type(a.atttypid, a.atttypmod), d.adsrc, a.attnotnull FROM pg_attribute a LEFT JOIN pg_attrdef d ON a.attrelid = d.adrelid AND a.attnum = d.adnum WHERE a.attrelid = '"emails"'::regclass AND a.attnum > 0 AND NOT a.attisdropped ORDER BY a.attnum

    /app/vendor/bundle/ruby/1.9.1/gems/activerecord-3.2.2/lib/active_record/connection_adapters/postgresql_adapter.rb:1139:in `async_exec'
    /app/vendor/bundle/ruby/1.9.1/gems/activerecord-3.2.2/lib/active_record/connection_adapters/postgresql_adapter.rb:1139:in `exec_no_cache'
    /app/vendor/bundle/ruby/1.9.1/gems/activerecord-3.2.2/lib/active_record/connection_adapters/postgresql_adapter.rb:663:in `block in exec_query'
    /app/vendor/bundle/ruby/1.9.1/gems/activerecord-3.2.2/lib/active_record/connection_adapters/abstract_adapter.rb:280:in `block in log'
    /app/vendor/bundle/ruby/1.9.1/gems/activesupport-3.2.2/lib/active_support/notifications/instrumenter.rb:20:in `instrument'
    /app/vendor/bundle/ruby/1.9.1/gems/activerecord-3.2.2/lib/active_record/connection_adapters/abstract_adapter.rb:275:in `log'
    /app/vendor/bundle/ruby/1.9.1/gems/newrelic_rpm-3.3.2/lib/new_relic/agent/instrumentation/active_record.rb:31:in `block in log_with_newrelic_instrumentation'
    /app/vendor/bundle/ruby/1.9.1/gems/newrelic_rpm-3.3.2/lib/new_relic/agent/method_tracer.rb:242:in `trace_execution_scoped'
    /app/vendor/bundle/ruby/1.9.1/gems/newrelic_rpm-3.3.2/lib/new_relic/agent/instrumentation/active_record.rb:28:in `log_with_newrelic_instrumentation'
    /app/vendor/bundle/ruby/1.9.1/gems/activerecord-3.2.2/lib/active_record/connection_adapters/postgresql_adapter.rb:662:in `exec_query'
    /app/vendor/bundle/ruby/1.9.1/gems/activerecord-3.2.2/lib/active_record/connection_adapters/postgresql_adapter.rb:1264:in `column_definitions'
    /app/vendor/bundle/ruby/1.9.1/gems/activerecord-3.2.2/lib/active_record/connection_adapters/postgresql_adapter.rb:858:in `columns'
    /app/vendor/bundle/ruby/1.9.1/gems/activerecord-3.2.2/lib/active_record/connection_adapters/schema_cache.rb:12:in `block in initialize'
    /app/vendor/bundle/ruby/1.9.1/gems/activerecord-3.2.2/lib/active_record/model_schema.rb:228:in `yield'
    /app/vendor/bundle/ruby/1.9.1/gems/activerecord-3.2.2/lib/active_record/model_schema.rb:228:in `default'
    /app/vendor/bundle/ruby/1.9.1/gems/activerecord-3.2.2/lib/active_record/model_schema.rb:228:in `columns'
    /app/vendor/bundle/ruby/1.9.1/gems/activerecord-3.2.2/lib/active_record/model_schema.rb:237:in `columns_hash'
    /app/vendor/bundle/ruby/1.9.1/gems/activerecord-3.2.2/lib/active_record/relation/delegation.rb:7:in `columns_hash'
    /app/vendor/bundle/ruby/1.9.1/gems/activerecord-3.2.2/lib/active_record/relation/finder_methods.rb:330:in `find_one'
    /app/vendor/bundle/ruby/1.9.1/gems/activerecord-3.2.2/lib/active_record/relation/finder_methods.rb:311:in `find_with_ids'
    /app/vendor/bundle/ruby/1.9.1/gems/activerecord-3.2.2/lib/active_record/relation/finder_methods.rb:107:in `find'
    /app/vendor/bundle/ruby/1.9.1/gems/activerecord-3.2.2/lib/active_record/querying.rb:5:in `find'
    /app/app/workers/massive_email_sender.rb:5:in `perform'

According to this, the first argument is the email id, and the second one is the list of all recipients… exactly as it should be.

Can anyone help me? Thanks!

Answer 1

I’ve run into the same problem. Assuming you’re using Active Record you have to call ActiveRecord::Base.establish_connection for each forked Resque worker to make sure it doesn’t have a stale database connection. Try putting this in your lib/tasks/resque.rake

task "resque:setup" => :environment do
  ENV['QUEUE'] = '*'
  Resque.after_fork = Proc.new { ActiveRecord::Base.establish_connection }
end