Move Fast and Consumer Driven Contract Test Things

Dd1e13d25a79220cb09fa0c1140df9d8?s=47 Alon Pe'er
February 17, 2017

Move Fast and Consumer Driven Contract Test Things

At SoundCloud, we've found that teams move faster when we've moved away from a monolith architecture to one based around microservices. Unfortunately, this new type of architecture has been prone to cascading failures when breaking changes go unnoticed in one of our services' API's. These failures have had a devastating impact on our system's uptime, but we've found that we can mitigate some of this risk by introducing consumer driven contract tests.

Consumer driven contract tests allow each consumer service and client to define their expectations and interactions with each provider service upstream, and for provider services to verify all of these contracts as part of their build pipeline. Breakage avoided.

In this talk we’ll go through SoundCloud’s process of breaking the monolith into microservices, then see how PACT-based contract tests were introduced, and discuss some of the challenges of adopting such tests in an already-established microservices culture.

Video available here: https://www.youtube.com/watch?v=nQ0UGY2-YYI

Dd1e13d25a79220cb09fa0c1140df9d8?s=128

Alon Pe'er

February 17, 2017
Tweet

Transcript

  1. Making SoundCloud’s µ-services safe to deploy Move Fast and Consumer-Driven-

    Contract-Test Things microXchg Berlin 2017
  2. 1.5 years @ SoundCloud Discovery Cluster Backend Engineer @alonpeer

  3. About SoundCloud a cloud full of sounds 135M tracks, 175M

    listeners, 12 hours/minute 300+ services / 50+ teams
  4. Agenda • Testing, monolith style • My first steps @

    SoundCloud • Introducing contract tests • Pactifying our services • Caveats • QA Q&A
  5. Testing, monolith style

  6. • One service, one client • API changes are easy

    • One client team to sync with for migrations • API deprecation is easy Testing, monolith style The good ol’ days
  7. • Different requirements per client => Code complexity increases •

    Harder to deploy without breaking at least one client Testing, monolith style Mo clients, mo problems
  8. • More manual QA expensive, slow, prone to human error,

    doesn’t scale • More end-to-end tests Maintenance nightmare, flaky, slow, creates bottlenecks Testing, monolith style Mo clients, mo problems
  9. My first steps @ SoundCloud

  10. My first steps @ SoundCloud Whoa, micro-services

  11. My first steps @ SoundCloud Testing strategy E2E Unit Integration

  12. My first steps @ SoundCloud Digging into post-mortems © ofsmallthings

  13. My first steps @ SoundCloud “A lack of trust in

    the acceptance tests caused us to largely ignore the warnings they generated.” “We couldn’t figure out the impact of the broken acceptance tests, and assumed the only problem was the tests themselves, rather than the underlying code.” “The commit went through as there weren't any tests for the serialisation from the client to the backend.” Digging into post-mortems © ofsmallthings
  14. Introducing contract tests

  15. Introducing contract tests My daily routine © LockeSteerpike

  16. Introducing contract tests My daily routine © rickandmorty.com

  17. Unit tests are not enough Introducing contract tests “#cake” >>

    { result = call(“cake”) assert(result == “lie”) } GET “/cake”: return “lie”
  18. Unit tests are not enough Introducing contract tests GET “/cake”:

    return “truth” “#cake” >> { result = call(“cake”) assert(result == “lie”) }
  19. Unit tests are not enough Introducing contract tests import JsonLibFoo

    GET “/cake”: return toJson(current_state) // { “current_state” : “lie” } import JsonLibFoo “#cake” >> { result = fromJson(call(“cake”)) assert(result == “lie”) }
  20. Unit tests are not enough Introducing contract tests import JsonLibBar

    GET “/cake”: return toJson(current_state) // { “currentState” : “lie” } import JsonLibBar “#cake” >> { result = fromJson(call(“cake”)) assert(result == “lie”) }
  21. Introducing contract tests How this works? Consumer Provider End-to-End Test

  22. Introducing contract tests How this works? Consumer Provider Unit Test

    Unit Test
  23. Introducing contract tests How this works? Consumer Provider Unit Test

    Unit Test
  24. Introducing contract tests How this works? ➢ Requesting “/cake”. ➢

    Expecting a JSON response with a key “current_state”. ➢ Asserting deserialization of the response succeeds. import JsonLibFoo GET “/cake”: return toJson(current_state) // { “current_state” : “lie” } Consumer Provider
  25. Pactifying our services http://pact.io

  26. What’s Pact? Pactifying our services • A family of frameworks

    designed for contract testing. • Supports JSON over HTTP. • Impl. for Java, Scala, Go, Ruby, .Net, Swift, JS etc. • Tests run locally, no external dependencies.
  27. What’s Pact? Pactifying our services Step 1: Define consumer expectations

    © pact.io
  28. What’s Pact? Pactifying our services Step 2: Verify expectations on

    provider © pact.io
  29. Consumer: client code

  30. Consumer: client code

  31. Consumer: client code

  32. Consumer: client code

  33. Consumer: client code

  34. Consumer: client code

  35. Consumer: client unit test

  36. Consumer: client unit test

  37. Consumer: client unit test

  38. Consumer: client unit test

  39. Consumer: client contract unit test

  40. Consumer: client contract unit test

  41. Consumer: client contract unit test

  42. Consumer: client contract unit test

  43. Consumer: client contract unit test

  44. Consumer: client contract unit test

  45. Consumer: client contract unit test

  46. Consumer: client contract unit test

  47. Consumer: my-consumer-likes.json

  48. Consumer: my-consumer-likes.json

  49. Consumer: my-consumer-likes.json

  50. Consumer: my-consumer-likes.json

  51. Consumer: my-consumer-likes.json

  52. Consumer: my-consumer-likes.json

  53. Consumer: my-consumer-likes.json

  54. Pactifying our services Provider side verification • Collect pact files

    from consumers and verify them all. • Keep your provider environment isolated. ◦ Dockerized databases. ◦ Test doubles for upstream services.
  55. Pactifying our services Provider states

  56. Pactifying our services Provider states Likes App Likes DB

  57. Pactifying our services Provider states Set state: “User 1000 has

    liked 2 tracks” Likes App Likes DB
  58. Pactifying our services Provider states INSERT likes Set state: “User

    1000 has liked 2 tracks” Likes App Likes DB
  59. Pactifying our services Provider states INSERT likes Set state: “User

    1000 has liked 2 tracks” HTTP Request: GET /likes/1000 Likes App Likes DB
  60. Pactifying our services Provider states HTTP Request: GET /likes/1000 HTTP

    Response: 200 OK INSERT likes Set state: “User 1000 has liked 2 tracks” Likes App Likes DB
  61. Pactifying our services Provider states Likes App Likes DB HTTP

    Response: 200 OK INSERT likes Set state: “User 1000 has liked 2 tracks” HTTP Request: GET /likes/2000 HTTP Response: 404 Not Found Set state: “User 2000 has liked no tracks” HTTP Request: GET /likes/1000 DELETE likes
  62. Pactifying our services Pact broker • Share pact files between

    projects. • API for pact frameworks to fetch all pacts by provider. • Support for versioning and tagging. • Useful UI and visualization of the network.
  63. Pactifying our services Pact broker © pact broker

  64. Pactifying our services Pact broker © pact broker

  65. Pactifying our services Pact broker © pact broker

  66. Pactifying our services Our CI pipeline Push new change Generate

    consumer contracts Deploy Upload pacts & tag with ‘prod’ Consumer pipeline
  67. Pactifying our services Our CI pipeline Push new change Deploy

    Upload pacts & tag with ‘prod’ Consumer pipeline Push new change Verify all ‘prod’-tagged pacts Deploy Provider pipeline Generate consumer contracts
  68. • Communication is key. Caveats

  69. • Communication is key. • New moving parts. Caveats

  70. • Communication is key. • New moving parts. • Learning

    curve per pact framework. Caveats
  71. • Communication is key. • New moving parts. • Learning

    curve per pact framework. • Frameworks are WIP. Caveats
  72. • Communication is key. • New moving parts. • Learning

    curve per pact framework. • Frameworks are WIP. • Postel's law (the robustness principle). Caveats
  73. • Communication is key. • New moving parts. • Learning

    curve per pact framework. • Frameworks are WIP. • Postel's law (the robustness principle). • Provider side setup is time consuming. Caveats
  74. • Communication is key. • New moving parts. • Learning

    curve per pact framework. • Frameworks are WIP. • Postel's law (the robustness principle). • Provider side setup is time consuming. • Automating consumer-triggered provider verification. Caveats
  75. • Communication is key. • New moving parts. • Learning

    curve per pact framework. • Frameworks are WIP. • Postel's law (the robustness principle). • Provider side setup is time consuming. • Automating consumer-triggered provider verification. • Adoption is essential. Caveats
  76. Recap

  77. Recap Write contract tests Consum ers

  78. Recap Write contract tests Generate pact files Consum ers

  79. Recap Write contract tests Generate pact files Publish to broker

    Consum ers
  80. Recap Write contract tests Generate pact files Publish to broker

    Verify pacts Consum ers Providers
  81. Recap Write contract tests Generate pact files Publish to broker

    Verify pacts Consum ers Providers
  82. QA Q&A @alonpeer