Move Fast and Consumer Driven Contract Test Things

At SoundCloud, we've found that teams move faster when we've moved away from a monolith architecture to one based around microservices. Unfortunately, this new type of architecture has been prone to cascading failures when breaking changes go unnoticed in one of our services' API's. These failures have had a devastating impact on our system's uptime, but we've found that we can mitigate some of this risk by introducing consumer driven contract tests.

Consumer driven contract tests allow each consumer service and client to define their expectations and interactions with each provider service upstream, and for provider services to verify all of these contracts as part of their build pipeline. Breakage avoided.

In this talk we’ll go through SoundCloud’s process of breaking the monolith into microservices, then see how PACT-based contract tests were introduced, and discuss some of the challenges of adopting such tests in an already-established microservices culture.

Video available here: https://www.youtube.com/watch?v=nQ0UGY2-YYI

Transcript

1. Making SoundCloud’s µ-services safe to deploy
   Move Fast and Consumer-Driven-
   Contract-Test Things
   microXchg Berlin 2017
   

   View Slide

2. 1.5 years @ SoundCloud Discovery Cluster Backend Engineer
   @alonpeer
   

   View Slide

3. About SoundCloud
   a cloud full of sounds
   135M tracks, 175M listeners, 12 hours/minute
   300+ services / 50+ teams
   

   View Slide

4. Agenda
   ● Testing, monolith style
   ● My first steps @ SoundCloud
   ● Introducing contract tests
   ● Pactifying our services
   ● Caveats
   ● QA Q&A
   

   View Slide

5. Testing, monolith style
   

   View Slide

6. ● One service, one client
   ● API changes are easy
   ● One client team to sync with for migrations
   ● API deprecation is easy
   Testing, monolith style
   The good ol’ days
   

   View Slide

7. ● Different requirements per client
   => Code complexity increases
   ● Harder to deploy without breaking at least one client
   Testing, monolith style
   Mo clients, mo problems
   

   View Slide

8. ● More manual QA
   expensive, slow, prone to human error,
   doesn’t scale
   ● More end-to-end tests
   Maintenance nightmare, flaky, slow,
   creates bottlenecks
   Testing, monolith style
   Mo clients, mo problems
   

   View Slide

9. My first steps @ SoundCloud
   

   View Slide

10. My first steps @ SoundCloud
    Whoa, micro-services
    

    View Slide

11. My first steps @ SoundCloud
    Testing strategy
    E2E
    Unit
    Integration
    

    View Slide

12. My first steps @ SoundCloud
    Digging into post-mortems
    © ofsmallthings
    

    View Slide

13. My first steps @ SoundCloud
    “A lack of trust in the acceptance tests caused us to largely
    ignore the warnings they generated.”
    “We couldn’t figure out the impact of the broken acceptance
    tests, and assumed the only problem was the tests themselves,
    rather than the underlying code.”
    “The commit went through as there weren't any tests for
    the serialisation from the client to the backend.”
    Digging into post-mortems
    © ofsmallthings
    

    View Slide

14. Introducing contract tests
    

    View Slide

15. Introducing contract tests
    My daily routine
    © LockeSteerpike
    

    View Slide

16. Introducing contract tests
    My daily routine
    © rickandmorty.com
    

    View Slide

17. Unit tests are not enough
    Introducing contract tests
    “#cake” >> {
    result = call(“cake”)
    assert(result == “lie”)
    }
    GET “/cake”:
    return “lie”
    

    View Slide

18. Unit tests are not enough
    Introducing contract tests
    GET “/cake”:
    return “truth”
    “#cake” >> {
    result = call(“cake”)
    assert(result == “lie”)
    }
    

    View Slide

19. Unit tests are not enough
    Introducing contract tests
    import JsonLibFoo
    GET “/cake”:
    return toJson(current_state)
    // { “current_state” : “lie” }
    import JsonLibFoo
    “#cake” >> {
    result = fromJson(call(“cake”))
    assert(result == “lie”)
    }
    

    View Slide

20. Unit tests are not enough
    Introducing contract tests
    import JsonLibBar
    GET “/cake”:
    return toJson(current_state)
    // { “currentState” : “lie” }
    import JsonLibBar
    “#cake” >> {
    result = fromJson(call(“cake”))
    assert(result == “lie”)
    }
    

    View Slide

21. Introducing contract tests
    How this works?
    Consumer Provider
    End-to-End Test
    

    View Slide

22. Introducing contract tests
    How this works?
    Consumer Provider
    Unit Test Unit Test
    

    View Slide

23. Introducing contract tests
    How this works?
    Consumer Provider
    Unit Test Unit Test
    

    View Slide

24. Introducing contract tests
    How this works?
    ➢ Requesting “/cake”.
    ➢ Expecting a JSON response
    with a key “current_state”.
    ➢ Asserting deserialization
    of the response succeeds.
    import JsonLibFoo
    GET “/cake”:
    return toJson(current_state)
    // { “current_state” : “lie” }
    Consumer
    Provider
    

    View Slide

25. Pactifying our services
    http://pact.io
    

    View Slide

26. What’s Pact?
    Pactifying our services
    ● A family of frameworks designed for contract testing.
    ● Supports JSON over HTTP.
    ● Impl. for Java, Scala, Go, Ruby, .Net, Swift, JS etc.
    ● Tests run locally, no external dependencies.
    

    View Slide

27. What’s Pact?
    Pactifying our services
    Step 1: Define consumer expectations
    © pact.io
    

    View Slide

28. What’s Pact?
    Pactifying our services
    Step 2: Verify expectations on provider
    © pact.io
    

    View Slide

29. Consumer: client code
    

    View Slide

30. Consumer: client code
    

    View Slide

31. Consumer: client code
    

    View Slide

32. Consumer: client code
    

    View Slide

33. Consumer: client code
    

    View Slide

34. Consumer: client code
    

    View Slide

35. Consumer: client unit test
    

    View Slide

36. Consumer: client unit test
    

    View Slide

37. Consumer: client unit test
    

    View Slide

38. Consumer: client unit test
    

    View Slide

39. Consumer: client contract unit test
    

    View Slide

40. Consumer: client contract unit test
    

    View Slide

41. Consumer: client contract unit test
    

    View Slide

42. Consumer: client contract unit test
    

    View Slide

43. Consumer: client contract unit test
    

    View Slide

44. Consumer: client contract unit test
    

    View Slide

45. Consumer: client contract unit test
    

    View Slide

46. Consumer: client contract unit test
    

    View Slide

47. Consumer: my-consumer-likes.json
    

    View Slide

48. Consumer: my-consumer-likes.json
    

    View Slide

49. Consumer: my-consumer-likes.json
    

    View Slide

50. Consumer: my-consumer-likes.json
    

    View Slide

51. Consumer: my-consumer-likes.json
    

    View Slide

52. Consumer: my-consumer-likes.json
    

    View Slide

53. Consumer: my-consumer-likes.json
    

    View Slide

54. Pactifying our services
    Provider side verification
    ● Collect pact files from consumers and verify them all.
    ● Keep your provider environment isolated.
    ○ Dockerized databases.
    ○ Test doubles for upstream services.
    

    View Slide

55. Pactifying our services
    Provider states
    

    View Slide

56. Pactifying our services
    Provider states
    Likes
    App
    Likes
    DB
    

    View Slide

57. Pactifying our services
    Provider states
    Set state:
    “User 1000 has liked 2 tracks”
    Likes
    App
    Likes
    DB
    

    View Slide

58. Pactifying our services
    Provider states
    INSERT
    likes
    Set state:
    “User 1000 has liked 2 tracks”
    Likes
    App
    Likes
    DB
    

    View Slide

59. Pactifying our services
    Provider states
    INSERT
    likes
    Set state:
    “User 1000 has liked 2 tracks”
    HTTP Request: GET /likes/1000
    Likes
    App
    Likes
    DB
    

    View Slide

60. Pactifying our services
    Provider states
    HTTP Request: GET /likes/1000
    HTTP Response: 200 OK
    INSERT
    likes
    Set state:
    “User 1000 has liked 2 tracks”
    Likes
    App
    Likes
    DB
    

    View Slide

61. Pactifying our services
    Provider states
    Likes
    App
    Likes
    DB
    HTTP Response: 200 OK
    INSERT
    likes
    Set state:
    “User 1000 has liked 2 tracks”
    HTTP Request: GET /likes/2000
    HTTP Response: 404 Not Found
    Set state:
    “User 2000 has liked no tracks”
    HTTP Request: GET /likes/1000
    DELETE
    likes
    

    View Slide

62. Pactifying our services
    Pact broker
    ● Share pact files between projects.
    ● API for pact frameworks to fetch all pacts by provider.
    ● Support for versioning and tagging.
    ● Useful UI and visualization of the network.
    

    View Slide

63. Pactifying our services
    Pact broker
    © pact broker
    

    View Slide

64. Pactifying our services
    Pact broker
    © pact broker
    

    View Slide

65. Pactifying our services
    Pact broker
    © pact broker
    

    View Slide

66. Pactifying our services
    Our CI pipeline
    Push new
    change
    Generate
    consumer
    contracts
    Deploy
    Upload
    pacts & tag
    with ‘prod’
    Consumer pipeline
    

    View Slide

67. Pactifying our services
    Our CI pipeline
    Push new
    change
    Deploy
    Upload
    pacts & tag
    with ‘prod’
    Consumer pipeline
    Push new
    change
    Verify all
    ‘prod’-tagged
    pacts
    Deploy
    Provider pipeline
    Generate
    consumer
    contracts
    

    View Slide

68. ● Communication is key.
    Caveats
    

    View Slide

69. ● Communication is key.
    ● New moving parts.
    Caveats
    

    View Slide

70. ● Communication is key.
    ● New moving parts.
    ● Learning curve per pact framework.
    Caveats
    

    View Slide

71. ● Communication is key.
    ● New moving parts.
    ● Learning curve per pact framework.
    ● Frameworks are WIP.
    Caveats
    

    View Slide

72. ● Communication is key.
    ● New moving parts.
    ● Learning curve per pact framework.
    ● Frameworks are WIP.
    ● Postel's law (the robustness principle).
    Caveats
    

    View Slide

73. ● Communication is key.
    ● New moving parts.
    ● Learning curve per pact framework.
    ● Frameworks are WIP.
    ● Postel's law (the robustness principle).
    ● Provider side setup is time consuming.
    Caveats
    

    View Slide

74. ● Communication is key.
    ● New moving parts.
    ● Learning curve per pact framework.
    ● Frameworks are WIP.
    ● Postel's law (the robustness principle).
    ● Provider side setup is time consuming.
    ● Automating consumer-triggered provider verification.
    Caveats
    

    View Slide

75. ● Communication is key.
    ● New moving parts.
    ● Learning curve per pact framework.
    ● Frameworks are WIP.
    ● Postel's law (the robustness principle).
    ● Provider side setup is time consuming.
    ● Automating consumer-triggered provider verification.
    ● Adoption is essential.
    Caveats
    

    View Slide

76. Recap
    

    View Slide

77. Recap
    Write contract tests
    Consum
    ers
    

    View Slide

78. Recap
    Write contract tests Generate pact files
    Consum
    ers
    

    View Slide

79. Recap
    Write contract tests Generate pact files
    Publish to broker
    Consum
    ers
    

    View Slide

80. Recap
    Write contract tests Generate pact files
    Publish to broker
    Verify pacts
    Consum
    ers
    Providers
    

    View Slide

81. Recap
    Write contract tests Generate pact files
    Publish to broker
    Verify pacts
    Consum
    ers
    Providers
    

    View Slide

82. QA Q&A
    @alonpeer
    

    View Slide