String interpolation in Ruby is great. It’s one of those niceties that I really miss whenever I venture into JavaScript land. But watch out! There are some lesser known “features” of Ruby’s string interpolation that can burn you… badly.

You’re probably familiar with Ruby’s standard string interpolation:

"Hello, my name is #{@name}."

However, you may not be familiar with Ruby’s shorthand for interpolating instance, class and global variables:

"Hello, my name is #@name."
"Hello, my name is #@@name."
"Hello, my name is #$name."

That’s kind of cool!

Well, maybe not as cool as you think!

Let’s look at the instance variable notation. Specifically, let’s look at how use of this shorthand recently burned CloudAMQP. CloudAMQP is a managed RabbitMQ service that we use heavily for our periodic task monitoring service, Dead Man’s Snitch. CloudAMQP had a major incident that resulted in termination of a large portion of their EC2 instances all at once[1]. CloudAMQP explains the incident in detail but I’d like to focus on their last bullet point describing what went wrong:

It turns out that we had a string interpolation bug.

The short version is that there was code responsible for identifying which EC2 instances need to be terminated by matching on instance name:

"CloudAMQP-#@name-*"

which is equivalent to:

"CloudAMQP-#{@name}-*"

In an effort to match additional EC2 instances, the following pattern was added:

"CloudAMQP-#@name0*"

Note that the - was replaced with a 0. The intent was:

"CloudAMQP-#{@name}0*"

Unfortunately, because 0 is a valid variable name character, the actual equivalent is:

"CloudAMQP-#{@name0}*"

Because the instance variable doesn’t exist, it’s interpreted as nil and the effect is that all instances that begin with CloudAMQP- were terminated.

How to avoid such a bug:

In discussing the CloudAMQP post-mortem around the office, we’ve come up with a few ways that you might be able to catch a bug like this before it becomes a huge problem:

1. Always use Ruby’s standard string interpolation style.[2] The two added characters are worth it.
2. Make sure your code is sufficiently unit-tested. An instance variable typo should appear as a failure in your test suite.
3. Run your application with Ruby’s warnings. Use of unknown instance variables will throw a warning. You can use rspec --warn to see warnings in your RSpec test suite.

[1] Dead Man’s Snitch automatically transitioned to use IronMQ during CloudAMQP’s outage and experienced no downtime.

[2] We’re calling for the deprecation and eventual removal of Ruby’s string interpolation shorthands.