Cucumber exiting with 0 on failure?
features/support/env.rb, if you have this:
at_exit do cleanup_after_tests end
at_exit do begin e = $! # last exception cleanup_after_tests ensure raise e if $! != e end end
After upgrading Rails from 4.0 to 4.1, my Cucumber tests were failing, but also exiting with a status code of 0, which means successful execution. Because of that rake thinks everything is fine and moves on. After searching and finding a lot of what wasn’t the problem, I went spelunking.
After this change in minitest 5,
exit is unconditionally called as part of the autorun. This turned out to be where the exit code was changing to 0. But why?
multi_test overrides the minitest runner so that it can report the exit status from Cucumber. It uses
$! which should be set to a
SystemExit exception and contain the status code it needs to report, but
The problem was our
at_exit was triggering something that cleared
$!, making it so that multi_test couldn’t do its job. It turned out to be a call to
FileUtils.mkdir_p that was resetting
$!, but I added the
ensure block so that we hopefully don’t end up accidentally triggering this again.