We noticed in one of our projects that Travis was giving us green when the tests themselves were actually failing. After some digging we figured out that Travis CI was using the result of the CodeClimate coverage tool over the test suite. After some digging through code and Issues we found the following Issues which describe our problem:

codeclimate/ruby-test-reporter#22

colszowka/simplecov#269

The bug is in how SimpleCov, the code coverage tool used by CodeClimate, handles exit status codes. Some changes were made in the 0.8 series that force a certain exit status, clobbering any other code that would be set. The only solution at this time is to use an earlier version of SimpleCov, 0.7.1. If you’re using Bundler, add the following line to the test group:

gem "simplecov", "~> 0.7.1", :require => false

We’ll update this post when a fixed version of SimpleCov is available.

UPDATE: This issue is fixed as of version 0.9.0.