Cucumber exiting with 0 on failure?

Now what?

https://www.pexels.com/photo/food-vegetables-cucumbers-gherkins-8694/

The solution:

In your features/support/env.rb, if you have this:

at_exit do
  cleanup_after_tests
end

Use this:

at_exit do
  begin
    e = $! # last exception
    cleanup_after_tests
  ensure
    raise e if $! != e
  end
end

What happened?

After upgrading Rails from 4.0 to 4.1, my Cucumber tests were failing, but also exiting with a status code of 0, which means successful execution. Because of that rake thinks everything is fine and moves on. After searching and finding a lot of what wasn’t the problem, I went spelunking.

The tunnels

minitest

After this change in minitest 5, exit is unconditionally called as part of the autorun. This turned out to be where the exit code was changing to 0. But why?

multi_test

multi_test overrides the minitest runner so that it can report the exit status from Cucumber. It uses $! which should be set to a SystemExit exception and contain the status code it needs to report, but $! was nil. Why?

features/support/env.rb

The problem was our at_exit was triggering something that cleared $!, making it so that multi_test couldn’t do its job. It turned out to be a call to FileUtils.mkdir_p that was resetting $!, but I added the ensure block so that we hopefully don’t end up accidentally triggering this again.


Vegetable cucumbers is licensed under Creative Commons Zero

Photo of David Genord II

David has been building websites since 8th grade. After working to integrate third-party automotive data with internal systems, he now brings his vast and wide knowledge to Collective Idea.

Comments

Add a Comment

Hmm...that didn't work.

Something went wrong while adding your comment. If you don't mind, please try submitting it again.

Comment Added!

Your comment has been added to this post. Please refresh this page to view it.

Optional. If added, we will display a link to the website in your comment.
Optional. Never shared or displayed in your comment.
  1. August 09, 2016 at 20:17 PM

    Great post - this solves a really important issue for us, ensuring Cuke fails will fail our CI build. We adapted your code based on HoundCI comments like so:

    ```
    at_exit do
    begin
    e = $ERROR_INFO # last exception
    cleanup_after_tests
    ensure
    fail e if $ERROR_INFO != e
    end
    end
    ```

    but get:

    AgileVentures/LocalSupport/features/support/env.rb:163:in `fail’: exception object expected (TypeError)

    we get the same with ‘raise’. I think that where there are no execeptions, e is nil, so perhaps the code could be:

    ```
    at_exit do
    begin
    e = $ERROR_INFO # last exception
    cleanup_after_tests
    ensure
    fail e if not e.nil? and $ERROR_INFO != e
    end
    end
    ```

  2. August 09, 2016 at 20:46 PM

    actually I’m finding that the following seems to work better for us:

    ```
    at_exit do
    exit 1 if Cucumber.wants_to_quit
    end
    ```