Question

When sshing into the webserver I can restart delayed_job all day without a problem. It brings down the existing worker, starts a new one and writes tmp/pids/delayed_job.pid with its process id. (I am also restarting passenger to mimic what I’m about to do with capistrano)

app@StagingServer:/app/current$ touch tmp/restart.txt; RAILS_ENV=staging script/delayed_job restart
delayed_job: trying to stop process with pid 22170...
delayed_job: process with pid 22170 successfully stopped.
delayed_job: process with pid 22284 started.
app@StagingServer:/app/current$ touch tmp/restart.txt; RAILS_ENV=staging script/delayed_job restart
delayed_job: trying to stop process with pid 22284...
delayed_job: process with pid 22284 successfully stopped.
delayed_job: process with pid 22355 started.
app@StagingServer:/app/current$ touch tmp/restart.txt; RAILS_ENV=staging script/delayed_job restart
delayed_job: trying to stop process with pid 22355...
delayed_job: process with pid 22355 successfully stopped.
delayed_job: process with pid 22427 started.
app@StagingServer:/app/current$

However, when I deploy using capistrano

dev@ubuntu:~/app-site$ cap passenger:restart
    triggering start callbacks for `passenger:restart'
  * executing `multistage:ensure'
*** Defaulting to `staging'
  * executing `staging'
  * executing `passenger:restart'
  * executing "touch /app/current/tmp/restart.txt"
    servers: ["staging.app.com"]
    [staging.app.com] executing command
    command finished in 242ms
  * executing "cd /app/current;RAILS_ENV=staging script/delayed_job restart"
    servers: ["staging.app.com"]
    [staging.app.com] executing command
 ** [out :: staging.app.com] delayed_job: trying to stop process with pid 21646...
 ** [out :: staging.app.com] delayed_job: process with pid 21646 successfully stopped.
    command finished in 11889ms

Seems fine? While the last line from delayed_job isn’t printed (I think due to it not ending in a newline) this does successfully create a new process. However, it doesn’t create the .pid file, so when I try and restart again:

dev@ubuntu:~/app-site$ cap passenger:restart
    triggering start callbacks for `passenger:restart'
  * executing `multistage:ensure'
*** Defaulting to `staging'
  * executing `staging'
  * executing `passenger:restart'
  * executing "touch /app/current/tmp/restart.txt"
    servers: ["staging.app.com"]
    [staging.app.com] executing command
    command finished in 398ms
  * executing "cd /app/current;RAILS_ENV=staging script/delayed_job restart"
    servers: ["staging.app.com"]
    [staging.app.com] executing command
 ** [out :: staging.app.com] Warning: no instances running. Starting...
 ** [out :: staging.app.com] delayed_job: process with pid 21950 started.
    command finished in 6758ms

It doesn’t stop the existing process. Strangely, this time it will create a new process and it’s .pid file.

This leaves me with 2 delayed_jobs processes running and only one in the .pid file. Every 2 cap deploys I do will add another delayed_job process. The previous processes are using the old version of the app, essentially breaking it.

config/deploy.rb:

namespace :passenger do
  desc "Restart Application"  
  task :restart do  
    run "touch #{current_path}/tmp/restart.txt"
    run "cd #{current_path};RAILS_ENV=#{deploy_env} script/delayed_job restart"
  end
end
after :deploy, "passenger:restart"

• Ubuntu, nginx, passenger
• daemons (1.1.4)
• delayed_job (2.1.4)
• rails (3.0.9)

And locally

• capistrano (2.9.0)
• capistrano-ext (1.2.1)

Update:

Reading around, it seems it could be to do with a race condition inside daemons. However I’m a bit confused why it’s showing only (and consistently) when using capistrano. I will try changing the command to a stop, sleep then start.

Answer 1

Solved by using stop and start instead of restart. Works due to a race condition probably caused by the daemons gem.

I would still like to know if anyone else has a better solution.