Test Your API with Cucumber and json_spec
At Collective Idea, we do a lot of work with RESTful JSON APIs. They can be a joy to build but a pain to test. We’re currently working on a project that’s all API all the time, so we developed some reusable Cucumber steps for testing. Now, we’ve abstracted all that goodness out into its own gem… json_spec.
The Gem
The json_spec gem is a collection of RSpec matchers and a collection of Cucumber steps to wrap them up. In an API-centric project a Cucumber scenario may look like this:
Scenario: User list
Given I post to "/users.json" with:
"""
{
"first_name": "Steve",
"last_name": "Richert"
}
"""
And I keep the JSON response at "id" as "USER_ID"
When I get "/users.json"
Then the JSON response should have 1 user
And the JSON response at "0" should be:
"""
{
"id": %{USER_ID},
"first_name": "Steve",
"last_name": "Richert"
}
"""
Besides the “I post” and “I get” steps, the steps above are provided by json_spec.
The Steps
There are only a handful of json_spec steps but they’re versatile, taking any of the following shapes:
the JSON should have "path"
I keep the JSON as "name"
I keep the JSON at "path" as "name"
the JSON should be:
the JSON should be {"key":"value"}
the JSON should be ["entry"]
the JSON should be %{name}
the JSON at "path" should be:
the JSON at "path" should be {"key":"value"}
the JSON at "path" should be ["entry"]
the JSON at "path" should be %{name}
the JSON at "path" should be "string"
the JSON at "path" should be 10
the JSON at "path" should be 10.0
the JSON at "path" should be true
the JSON at "path" should be false
the JSON at "path" should be null
the JSON should be a hash
the JSON at "path" should be an array
the JSON should have 1 entry
the JSON at "path" should have 2 entries
the JSON should have 3 keys
the JSON at "path" should have 4 whatevers
On top of these variations, each “JSON” can be written as “JSON response” and can be lowercase if you prefer. Each “should” can be “should not” as well.
Usage
In order to use json_spec in your Cucumber suite, in env.rb you must:
require "json_spec/cucumber"
Lastly you’ll need to define a last\_json method so that json_spec knows what to evaluate. If you’re using Capybara to access your API, this may look like:
def last_json
page.source
end
JSON Paths
You’ll see references to “path” in the steps above. These paths are slash-separated strings and they refer to specific locations in a JSON document. For instance, given a list of users:
[
{
"id": 1,
"first_name": "Steve",
"last_name": "Richert"
}
]
You could access the first name of the first user at the path:
0/first_name
And More…
The json_spec gem is young and features are still being added quickly. One of these features (used above) is the ability to store snippets of JSON for use in later steps.
Keep an eye on the json_spec repository to stay up on the latest and greatest developments. The README also has more information on the underlying RSpec matchers and how can they be used directly.
This approach has worked well for us but we’re very open to suggestions and contributions to improve json_spec. Check out the issue tracker to help. We hope you’ll get as much use out of json_spec as we do.
Comments
WHY would I want to use Cucumber to test this instead of writing plain old rspec controller specs? If you think about what Cucumber aims to be (although it rarely seems to succeed at it), it should first and foremost be a tool to describe the behavior of your application in your customer’s domain language. I don’t know about your customers but if my (end) customers actually know what JSON is, they usually only know that you can use it “for APIs and stuff” – so it’s not Cucumber’s domain.
I feel that you as experienced Ruby developers (as you probably are) should be encouraging what’s right and not just go with the hype of using Cucumber for every little thing.
Just my 2 cents, of course.
Clemens : I’ll give you two reasons:
1.) The application in question (as well as gems we’ve developed) is 100% API, so the “customer” is an API user.
2.) We find Cucumber to be much easier to figure out intent later. You don’t question why you’re testing some strange edge case, you have built the reasoning into the test.
Personally, I (and I think I can speak for the rest of the team) love writing Cucumber. We don’t think it is the right tool for every job, but it makes a lot of what we do much easier, faster, and cleaner.
I’ve done something similar with cucumber to test API’s before. Nice to see that something’s been factored out.
To date I had done somewhat rigid blanket tests, like:
Then the response should contain Foo
and the I test the contents of Foo int he step_definition
Really good to see this! I have created similar steps for my own projects, but never found the time to extract it to a gem. I also think you found a nice solution for things like id’s. How do you treat timestamps? I use timecop and steps like:
And what about response codes? I have a step like:
Clemens: When working on the API for http://www.zwapp.com it is really helpful for our IPhone developer to exactly see what he can expect of the API. Since we are a young project we cannot afford the time for the duplication in the form of documentation and an other form of tests. To me testing API’s is a lot different than testing a webpage. Whereas Capybara etc. code might not become ugly for webpages
fill\_in 'username'. For API’s I find it very hard to read and have an overview. Cucumber makes it feel as if you are automating curl. I often find myself using the features as documentation for myself.On the other hand I would love the be convinced to do it differently by someone having a good example using something else than Cucumber.
I’ll keep an eye on your repo and see if we can migrate to it and extend it.
Jeroen: Thanks! As far as timestamps, json_spec allows you to ignore certain keys when comparing JSON hashes. By default, it ignores id, created_at and updated_at. README for more info. But in the few cases where we need to assert timestamp values, we do use Timecop.
We also assert status codes like you mention but in separate steps from our JSON assertions. So although the status code step is useful, it’s unrelated to JSON so it falls outside the scope of json_spec.
The fact that ruby hashes are embedded in the scenarios indicates a problem to me.
You’re also building variable assignment step definitions, “And I keep the JSON response at “id” as “USER_ID””, makes me cringe. Why not just write something like this
I can’t help but feel like projects like this are the reason Cucumber gets a bad reputation.
Daniel: Seeing that the “customer” is an API user, is it not conceivable that they would understand code scenarios?
Michael: Our choice to use cucumber here wasn’t because of the customer, it is because we find Cucumber quicker to write and easier to understand, especially when you have to come back to a project or chunk of code after a period of time.
It isn’t for everyone, and certainly this gem isn’t for every project, but we’ve found it helpful.
Definitely check out the rspec matchers, as that’s what everything is built on. You can use them without cucumber at all!
Daniel: “Our choice to use cucumber here wasn’t because of the customer”, that is interesting, seeing as that is your first reason in response to Clemens.
Your second reason is that the reasoning is built into the tests. I don’t see any additional context in the Cucumber scenarios versus what is in my gist. Maybe these examples are too simple to start to see the benefits? I’d love to see some more real world examples with this extra reasoning built into the scenarios.
I’m also not sure how typing more words to describe the same thing is any quicker to write.
Michael: Good catch, I wasn’t being very clear. Our “customer” in this case is indeed both an API and a client that understands APIs. That said, we always want to pick the best tool for the job. In this case, we chose Cucumber because we find it clearer and easier to maintain for all involved. Your mileage may vary. ;)
Daniel: I’m still trying to understand where the additional context/reasoning comes from. I don’t see any additional context in the Cucumber scenarios versus what is in my gist. Maybe these examples are too simple to start to see the benefits? I’d love to see some more real world examples with this extra reasoning built into the scenarios. Does the extra reasoning come from the feature descriptions at the top of a .feature file?
Michael: It is a readability thing. Your example works great, but as it grows to tens or hundreds of lines (which unfortunately happens as you find complex use cases and workflows) you lose track of the “why”.
With Cucumber, you can more clearly see the flow because you’re reading plain English (well, not quite in our case with this gem) as opposed to raw code.
Many of our test suites grow huge over time, and when you come back to code even a month or two later, Cucumber is easier for us to grok; we remember what the test case was there for, maybe a quirky bug or a very specific case.
Do you have an aversion to Cucumber in general? I find many people do, but after using it for 2+ years, I can honestly say it is a huge competitive advantage for our business and makes us work better and happier.
Michael: One big win for Cucumber is that you’re dealing with actual JSON and posted data. In your gist you’re doing a ruby abstraction of it. It’s easy to take for granted that you give #post a hash and it gets converted to JSON.
Ideally someone could copy the data out of the scenario and post it to the actual application with curl. It leads to much better documentation when you’re dealing with a separate team that is trying to figure out exactly what a request and response will look like.
Daniel: When using RSpec to specify JSON API behavior, I’m able to remember the “why” because of the context surrounding it. If the examples read exactly the same and the context describes the “why”, I’m still not sure where the benefit comes from…
Are you wrapping these step definitions in other steps? How are you managing the duplication between scenarios that happens when you get the type of complexity and growth you describe? Maybe the cost of duplication outweighs the benefits of generated documentation.
I do not have an aversion to Cucumber, I actually really like it and use it everyday. I have not used Cucumber to drive the design of an HTTP/JSON API, I typically use a combination of RSpec integration and controller specs.
Chris: Thanks for the insight, it does make sense that the Cucumber scenarios are operating at a higher-level. I think I have a better understanding now, you’re using this as a communication tool for your customers. Daniel has wavered back and forth whether it was for the customer or the developer. I think it is very important to make the distinction between developer tests and customer tests.
Cucumber scenarios can be used as example in documentation. You can see an example at Chargify site.
I used the same approach to develop application with XML API.
For that I developed test_xml gem.
It can be used with Cucumber, TestUnit or RSpec.
alovak: You hit the nail on the head. Clear documentation is a big part of its usefulness for us.
I hadn’t seen your test_xml gem. Well done. And strangely similar to json_spec. We prefer JSON when it’s up to us… but it’s not always up to us, so we’ll keep an eye on test_xml. Thanks!
alovak: which parts of the chargify documentation is being generated from Cucumber?
Michale Guteri, Example:
http://docs.chargify.com/api-products#api-usage-json-products-list
json_spec looks great. I’m trying to get it to work, but the step definitions don’t seem to be picked up; all the json_spec steps are “undefined step”. I made the changes to env.rb. Anything else I should be checking? I’m using 0.7.0.
json_spec looks great. I’m trying to get it to work, but the step definitions don’t seem to be picked up; all the json_spec steps are “undefined step”. I made the changes to env.rb. Anything else I should be checking? I’m using 0.7.0.
FYI: to define last_json if you are using something a gem like rest-client or http-party which returns a ‘response’ object then, in the step that makes the call capture the response along these lines
@last_response = RestClient.post(url, request_body, :content_type => ‘application/json’)
Then wherever you are maintaining world extensions (usually a file under features\support) add something like the following
module JSONSpecInterface
def last_json
@last_response.body
end
end
World(JSONSpecInterface)
I like the gem. Very helpful.
However, I do not like the cucumber step definition part. It does not provide sufficient debug information. When I use the statement “the JSON response at ‘xxx’ should be ‘yyyy’, the function returns me something like ‘Can’t convert Hash to String’. That is all I get. Clearly, the path part works. The return value may be a hash with one value instead of a plain value. That is fine. The bad thing is that it should spit out the return value to me so that I can debug and change my expectation.
Meanwhile, should a utility gem include cucumber steps? I am debating on it. Maybe overkill.
Therefore, I switch to jsonpath. I use httparty to submit requests and use jsonpath to analyze the response. Everything is under my control.
I have been looking this gem over. Good work. This greatly helps testing APIs and JSON. I am trying to understand how to use it for testing JSON files opposed to responses from requests. Can you suggest how to do this? How does the last_json that needs to be defined?
Thank You!
Hi folks!
Thank you for writing json_spec! It has inspired me to write jason_spec: https://github.com/organisedminds/jason_spec, something I would not have done without your good work.