Nick Hammond

Software Developer, Cyclist, & Traveler.

Monitoring multiple delayed job workers with Bluepill

In devops

It's really simple to have multiple workers going through your Delayed::Job queue and it's just as easy to start monitoring them properly with Bluepill.

The script/delayed_job command has a few options, one of them to just start up multiple workers(-n) and another one to start a worker based on an identifier(-i). In order to monitor multiple workers we need to specify a unique identifer for each worker so that Bluepill can monitor each one individually and tie it back to a pid.

Once I have multiple workers for an application I prefer to isolate them into their own Bluepill application group. Once you do this it's easy to stop and restart delayed_job when deploying new code, whether you're using Capistrano, Vlad or something else.

  Bluepill.application("delayed_job", :log_file => "/var/log/bluepill_delayed_job.log") do |app|
    <% 3.times do |i| %>
      app.process("delayed_job.<%= i %>") do |process|
        process.working_dir = "/srv/"

        process.start_grace_time    = 30.seconds
        process.stop_grace_time     = 30.seconds
        process.restart_grace_time  = 45.seconds

        process.start_command = "sudo RAILS_ENV=<%= node.chef_environment %> script/delayed_job start -i <%= i %>"
        process.stop_command  = "sudo RAILS_ENV=<%= node.chef_environment %> script/delayed_job stop -i <%= i %>"

        process.pid_file = "/srv/<%= i %>.pid"
    <% end %>

delayed_job daemonizes itself so you don't need to have Bluepill do that, we just need to point it to the specific pid location. Now you just need to load up the Bluepill application group like normal and you should see it start to bring up your workers.

To get delayed_job restarting with deploys you can reference the application group now instead of tieing it to restarting the specific process. Capistrano would look something like this:

after "deploy:stop" do
  run "sudo bluepill delayed_job stop"

after "deploy:start" do
  run "sudo bluepill delayed_job start"

after "deploy:restart" do
  run "sudo bluepill delayed_job restart"

Make sure that you've removed the delayedjob Capistrano tasks as those are no longer relevant for stopping/restarting delayedjob anymore.

P.S. We should keep in touch, subscribe below.