Question

We have a Rails application that is running in a MySQL master-slave set-up for a while now, using the master_slave_adapter plugin. Recently, there was a need for background processing of long running tasks. So we settled on DelayedJob.

DelayedJob’s table/model uses the same master-slave adapter. And it keeps the slave connection alive by polling the table. But the master connection remains idle for long periods of time, closes overnight, and the next time someone activates a job this happens:

Mysql::Error: MySQL server has gone away: UPDATE `delayed_jobs` SET locked_by = null, locked_at = null WHERE (locked_by = 'delayed_job host:[snip] pid:20481')

I’ve heard bad things about using the reconnect option in my database.yml, because it allegedly doesn’t set the connection character set after a reconnect, like it does on the first connection initialization.

What’s the proper way to make this work?

Answer 1

FWIW, we now monkey patch Delayed::Job in the two places it matters. Here’s the blob:

module Delayed
  class Job < ActiveRecord::Base
    class << self
      def refresh_connections_for_delayed_job
        # Do a cheap check to see if we're actually using master-slave.
        if (c = self.connection).respond_to? :master_connection
          c.master_connection.reconnect! unless c.master_connection.active?
        end
      end

      def clear_locks_with_connection_refresh!(worker_name)
        self.refresh_connections_for_delayed_job
        self.clear_locks_without_connection_refresh!(worker_name)
      end

      alias_method_chain :clear_locks!, :connection_refresh
    end

    def lock_exclusively_with_connection_refresh!(max_run_time, worker)
      self.class.refresh_connections_for_delayed_job
      self.lock_exclusively_without_connection_refresh!(max_run_time, worker)
    end
    alias_method_chain :lock_exclusively!, :connection_refresh
  end
end