I’m a Rails developer but I’m not much of an Ops guy. So when it comes to server monitoring, I want a simpler solution than God. God is a “Process Monitoring Framework in Ruby” and while I know it’s awesome, sometimes it’s not quick and dirty enough for my liking.

Fail Loudly

Usually, if one of my application’s dependencies (PostgreSQL, Redis) goes down, it fails loudly. We love Honeybadger for error notifications so we hear about these failures right away and we can snap into action. Perfect.

Background Jobs

Background jobs are similar in that no news is good news. The major difference is that if your background job process were to go down, you still get… no news. So how do we make sure that our background job process is always running?

Dead Man’s Snitch

Dead Man’s Snitch is a process monitoring service for periodic events, like cron jobs. Signing up is easy and your first “snitch” is free. Each snitch has its own URL that you ping every so often to confirm your process is still up and running.

For this case, I created an hourly snitch. Now I can just define a worker and queue up a job!

class SnitchWorker
  include Sidekiq::Worker

  def perform(token)
    `curl https://nosnch.in/#{token}`
    SnitchWorker.perform_in(1.hour, token)
  end
end

 > SnitchWorker.perform_async("c2354d53d2")
=> "d273eb429dc8877aedf6eb0c"

This worker just pings the snitch and then tell itself to do it all again in an hour.

Now if my background job process ever goes down (or even gets really backed up), my snitch will stop receiving pings and Dead Man’s Snitch will let me know.

God not required.