Going Rambo: Contract and Collaboration Testing in Ruby

End-to-end testing of a microservices architecture is costly and can be unreliable, leading to test suites that add limited value to a project. Contract and collaboration tests are becoming the accepted way to solve this problem, relying on data contracts and documentation to create fast, robust, and informative tests to make sure each API works as advertised.

Rambo is a new open-source Ruby tool that generates contract tests from API documentation in RAML, enabling documentation to become a single source of truth about service APIs. In this talk, I will discuss contract and collaboration testing and how Rambo tackles these problems for Ruby services. Additionally, I will give generalized suggestions about how to incorporate these practices into your or your organization’s workflows.

Transcript

2. CONTRACT AND COLLABORATION TESTING IN RUBY
3. Who Is This Lady? • Ruby developer and quality engineer • Rambo author • Cucumber core team • (Former?) opera singer • Huge Trekkie • @DanaScheider on Twitter
4. Definitions • API: The means through which software applications talk to each other • Web API: An application that serves resources to client applications over HTTP • Client: An application that contacts and consumes data from a web API • RAML: The REST API Modeling Language, a YAML-based language designed to describe web APIs in a standardized way
5. Definitions • Endpoint: A unique URI that represents a resource or collection of resources • Microservices Architecture: An architecture characterized by many small services communicating with each other via (usually web) APIs • Data Contract: A document detailing the way a service is guaranteed to behave toward its client or clients
6. A microservices architecture is the logical extension of the single-responsibility principle.
7. Microservices lend themselves to fast, reliable, and low-maintenance test suites.
8. Characteristics of Web APIs • Accepted protocols • Available endpoints • Accepted methods • Authorization and authentication schemes • URI parameters
9. Characteristics of Web APIs • Request formats and requirements – Required/permitted headers – Data format (JSON, XML, etc.) – Keys and possible values, including: • Data types • Length • Formats (e.g., string must be valid date) • Required/forbidden keys
10. Characteristics of Web APIs • Response characteristics – Format (JSON, XML, etc.) – Keys and possible values – Headers – Status codes
11. Data Contracts • Exist between two services • Can be provider- or consumer-driven • Guarantee that a particular version of the API will have certain characteristics • Facilitate interoperability • Specify how the APIs meet business needs • Can be aggregated into a single contract representing the totality of the API’s behavior
12. Data Contracts • Client-driven – Most common for small back-end services internal to an organization – Good for services with few clients • Provider-driven – Good for services with many clients – Most common for large public APIs (e.g., Facebook, Twitter, Gmail, etc.)
13. End-to-End Testing • Ensures that the system as a whole meets business needs • Slow • Costly in time and money • Lengthens feedback cycle for developers • Tests can be unreliable • Difficult to determine source of failures
14. Integration Testing • Tests an entire service or cluster of services • Ensures that pieces of the system work together to achieve the desired results • Is implementation agnostic
15. Unit Testing • Tests internal components • Is not implementation agnostic • May use doubles, mocks, stubs, and spies to isolate units under test • Is fast, cheap, and reliable • Does not verify that the system will meet business needs
16. The distinction between end-to-end, integration, and unit tests is a distinction of degree, not kind.
17. Contract Testing • Tests consumer-facing API as described by the data contract • Typically works by making actual or faked network requests to the service • Verifies the service is meeting business needs • Keeps tests relevant to consumer needs • Ensures the service works as documented
18. When writing automated tests of the modules which interact with external components, the goal is to verify that the module can communicate sufficiently rather than to acceptance test the external component. –Toby Clemson
19. To a given service in a microservices context, all other services in the ecosystem are considered external.
20. Collaboration Testing • Tests an API client • Verifies that the client is making correctly formed requests to the correct service • Relies on stubbing to capture outgoing requests without actually involving an external service
21. Contract & Collaboration Testing
22. Summary • Data contracts establish expectations between two services • Unit tests test internal logic • Contract tests verify that the API behavior is in line with the stipulations of the contract • Collaboration tests verify that the client is formulating requests correctly
23. Caveat: Red means theory!
24. Web Services: Challenges • Multiple sources of truth – API documentation – Data contracts – Test suites – App code • Multiple points of failure • Lack of clarity about how services meet business goals
25. Generated Contract Tests • Enforce adherence to data contracts • Ensure up-to-date tests • Test what matters to consumers (i.e., business)
26. Going Rambo • Generate RSpec tests from contracts in RAML • Use JSON Schema for examples and request/response schemas • Generate and run tests in CI • Easily view request bodies and compare actual response bodies to schemas
27. How Rambo Works • Three entry points/interfaces: – Command line – Rake task – Ruby API • Options: – Command line flags (--no-rails) – .rambo.yml – Ruby option hash
28. Under the Hood • DocumentGenerator class creates paths and files from ERB templates: – spec/rambo_helper.rb – spec/contract/api_name_spec.rb – spec/contract/output – spec/support/examples – spec/support/matchers/rambo_matchers.rb • match_schema matcher
29. Under the Hood • Options captured at entry points and passed through to generator • JsonTestDatagem generates request bodies from given schemas if no examples given • Requests made using Rack::Test • raml-rb gem parses RAML • RamlModels module adds abstraction layer for RAML nodes
30. Under the Hood • Generated request bodies and response schemas go in spec/support/examples • Actual response bodies received are saved in spec/contract/output for troubleshooting
31. Summary • Robust contract and collaboration tests reduce the need for costly end-to-end testing • Use Rambo to generate contract tests from your contracts – Eliminate problems stemming from multiple sources of truth – Ensure tests and docs are up-to-date
32. Contributing to Rambo • GitHub repo is /danascheider/rambo • Look at the contributor docs in the wiki • Read README • Add or improve README or wiki • Browse issue reports and pick one • File a new issue report – Request a feature – Report a bug • Contribute to the JSON Test Data gem • Contribute to the raml-rb gem
33. MORE INFO RAML: http://raml.org Toby Clemson on testing microservices: http://martinfowler.com/articles/microservice-testing/
34. MORE INFO Git: https://github.com/danascheider/rambo Rubygems: rambo_rubyv. 0.2.3 Twitter: @DanaScheider