How to restart God when you deploy a new release via Capistrano

I don’t have time to read and I want to jump immediately to the solution →.

Like the 90% of Rails developers, I use Capistrano to manage the deployment of my Rails applications. I also use God to monitor several application processes, such as DelayedJob workers.

The problem

Each time I execute a deployment, I need the restart all the Rails instances, including rake tasks and workers, otherwise they will continue to run with the previous (old) application version.

I cannot simply send a command to God via Capistrano because God is omnipotent and runs with root privileges, while Capistrano doesn’t (for security reasons I don’t deploy with a sudoer user).

Here comes the problem. How to restart God when you deploy a new release? I already discussed this problem in the past and the solution I adopted since today was to execute a command as root without using sudo. The problem with this solution, is that it requires you to type the root password each time you execute a new deploy. And this is extremely awkward.

Last week I was configuring a new Rails application and I decided to investigate a new solution. I really like the way how Phusion Passenger works: whenever you touch the /tmp/restart.txt file, Passenger restarts all the Rails processes.

Why not implementing the same behavior in God? Luckily, someone else already had the same idea before.

The solution

I found the following recipe on Gist.

module God
  module Conditions
    class RestartFileTouched < PollCondition
      attr_accessor :restart_file
      def initialize
        super
      end

      def process_start_time
        Time.parse(`ps -o lstart  -p #{self.watch.pid} --no-heading`)
      end

      def restart_file_modification_time
        File.mtime(self.restart_file)
      end

      def valid?
        valid = true
        valid &= complain("Attribute 'restart_file' must be specified", self) if self.restart_file.nil?
        valid
      end

      def test
        process_start_time < restart_file_modification_time
      end
    end
  end
end

Save the code in your God application or on the server and make sure God loads it. My suggestion is to add the following line in the God configuration script (e.g. /etc/god/config)

# Load in all God shared configs.
# Share configs apply to all instances.
God.load "/etc/god/conf.d/*.god"

and save the file in the /etc/god/conf.d folder. In this way, when you’ll start God with

$ god -c /etc/god/config

all the shared recipes will be loaded.

Then, configure the monitored process to watch the restart file. The configuration is available as a state transition condition.

rails_root = "/path/to/rails"

restart.condition(:restart_file_touched) do |c|
  c.interval = 5.seconds
  c.restart_file = File.join(rails_root, 'tmp', 'restart.txt')
end

Taking the GitHub DelayedJob recipe as example, you should change

# restart if memory gets too high
w.transition(:up, :restart) do |on|
  on.condition(:memory_usage) do |c|
    c.above = 300.megabytes
    c.times = 2
  end
end

to

# restart if memory gets too high
w.transition(:up, :restart) do |on|
  on.condition(:memory_usage) do |c|
    c.above = 300.megabytes
    c.times = 2
  end

  on.condition(:restart_file_touched) do |c|
    c.interval = 5.seconds
    c.restart_file = File.join(rails_root, 'tmp', 'restart.txt')
  end
end

The restart_file can be whichever file you want. I strongly encourage to use the same /tmp/restart.txt Passenger file, in this way your processes will restart each time you restart the application.

Thanks to Amedeo for his help and thanks to Nathan Humbert for creating the original God configuration.