Question

I am using Resque on a production website.

When I deploy, I want GOD to stop all of the workers and then restart them since sometimes we change the code of a class and requeue the failed jobs.

The problem is, that when I do god stop resque, the rakes does not actually stop, the workers still stay alive and working with older code, which creates all sorts of issues for me.

even when I do ‘god terminate’ it won’t kill the workers.

Right now, I am using a shell script to kill workers, but since I have more then one server, it’s pretty much a pain in the ass doing in on all production servers.

This is my god config file:

rails_env   = ENV['RAILS_ENV']  || "production"
rails_root  = ENV['RAILS_ROOT'] || "/mnt/data-store/html/gogobot/current"
num_workers = rails_env == 'production' ? 5 : 2

num_workers.times do |num|
  God.watch do |w|
    w.dir      = "#{rails_root}"
    w.name     = "resque-#{num}"
    w.group    = "resque"
    w.interval = 2.minutes
    w.env      = {"QUEUE"=>"duplicate_merging,facebook_wall_posts,generic,mailer,notifications,realtime,scoring_system,signup,social_graph_facebook,social_graph_foursquare,social_graph_twitter,user_info,user_score", "RAILS_ENV"=>rails_env, "PIDFILE" => "#{rails_root}/tmp/resque_#{w}.pid"}
    w.pid_file = "#{rails_root}/tmp/resque_#{w}.pid"
    w.start    = "cd #{rails_root}/ && rake environment resque:work QUEUE=duplicate_merging,facebook_wall_posts,generic,mailer,notifications,realtime,scoring_system,signup,social_graph_facebook,social_graph_foursquare,social_graph_twitter,user_info,user_score RAILS_ENV=#{rails_env}"
    w.log      = "#{rails_root}/log/resque_god.log"

    w.uid = 'root'
    w.gid = 'root'

    # restart if memory gets too high
    w.transition(:up, :restart) do |on|
      on.condition(:memory_usage) do |c|
        c.above = 350.megabytes
        c.times = 2
      end
    end

    # determine the state on startup
    w.transition(:init, { true => :up, false => :start }) do |on|
      on.condition(:process_running) do |c|
        c.running = true
      end
    end

    # determine when process has finished starting
    w.transition([:start, :restart], :up) do |on|
      on.condition(:process_running) do |c|
        c.running = true
        c.interval = 5.seconds
      end

      # failsafe
      on.condition(:tries) do |c|
        c.times = 5
        c.transition = :start
        c.interval = 5.seconds
      end
    end

    # start if process is not running
    w.transition(:up, :start) do |on|
      on.condition(:process_running) do |c|
        c.running = false
      end
    end
  end
end

1.times do |num|
  God.watch do |w|
    w.name     = "dj-#{num}"
    w.group    = 'dj'
    w.interval = 30.seconds
    w.start    = "cd #{rails_root} && rake jobs:work"

    w.uid = 'root'
    w.gid = 'root'

    # retart if memory gets too high
    w.transition(:up, :restart) do |on|
      on.condition(:memory_usage) do |c|
        c.above = 300.megabytes
        c.times = 2
      end
    end

    # determine the state on startup
    w.transition(:init, { true => :up, false => :start }) do |on|
      on.condition(:process_running) do |c|
        c.running = true
      end
    end

    # determine when process has finished starting
    w.transition([:start, :restart], :up) do |on|
      on.condition(:process_running) do |c|
        c.running = true
        c.interval = 5.seconds
      end

      # failsafe
      on.condition(:tries) do |c|
        c.times = 5
        c.transition = :start
        c.interval = 5.seconds
      end
    end

    # start if process is not running
    w.transition(:up, :start) do |on|
      on.condition(:process_running) do |c|
        c.running = false
      end
    end
  end
end

Would appreciate any help in how I can stop rake jobs using GOD.

Thanks.

Answer 1

The solution is to send a SIGQUIT to the process
so for example you can run

god signal resque SIGQUIT