$30 off During Our Annual Pro Sale. View Details »

Move Fast and Consumer Driven Contract Test Things

Alon Pe'er
February 17, 2017

Move Fast and Consumer Driven Contract Test Things

At SoundCloud, we've found that teams move faster when we've moved away from a monolith architecture to one based around microservices. Unfortunately, this new type of architecture has been prone to cascading failures when breaking changes go unnoticed in one of our services' API's. These failures have had a devastating impact on our system's uptime, but we've found that we can mitigate some of this risk by introducing consumer driven contract tests.

Consumer driven contract tests allow each consumer service and client to define their expectations and interactions with each provider service upstream, and for provider services to verify all of these contracts as part of their build pipeline. Breakage avoided.

In this talk we’ll go through SoundCloud’s process of breaking the monolith into microservices, then see how PACT-based contract tests were introduced, and discuss some of the challenges of adopting such tests in an already-established microservices culture.

Video available here: https://www.youtube.com/watch?v=nQ0UGY2-YYI

Alon Pe'er

February 17, 2017
Tweet

Other Decks in Technology

Transcript

  1. Making SoundCloud’s µ-services safe to deploy
    Move Fast and Consumer-Driven-
    Contract-Test Things
    microXchg Berlin 2017

    View Slide

  2. 1.5 years @ SoundCloud Discovery Cluster Backend Engineer
    @alonpeer

    View Slide

  3. About SoundCloud
    a cloud full of sounds
    135M tracks, 175M listeners, 12 hours/minute
    300+ services / 50+ teams

    View Slide

  4. Agenda
    ● Testing, monolith style
    ● My first steps @ SoundCloud
    ● Introducing contract tests
    ● Pactifying our services
    ● Caveats
    ● QA Q&A

    View Slide

  5. Testing, monolith style

    View Slide

  6. ● One service, one client
    ● API changes are easy
    ● One client team to sync with for migrations
    ● API deprecation is easy
    Testing, monolith style
    The good ol’ days

    View Slide

  7. ● Different requirements per client
    => Code complexity increases
    ● Harder to deploy without breaking at least one client
    Testing, monolith style
    Mo clients, mo problems

    View Slide

  8. ● More manual QA
    expensive, slow, prone to human error,
    doesn’t scale
    ● More end-to-end tests
    Maintenance nightmare, flaky, slow,
    creates bottlenecks
    Testing, monolith style
    Mo clients, mo problems

    View Slide

  9. My first steps @ SoundCloud

    View Slide

  10. My first steps @ SoundCloud
    Whoa, micro-services

    View Slide

  11. My first steps @ SoundCloud
    Testing strategy
    E2E
    Unit
    Integration

    View Slide

  12. My first steps @ SoundCloud
    Digging into post-mortems
    © ofsmallthings

    View Slide

  13. My first steps @ SoundCloud
    “A lack of trust in the acceptance tests caused us to largely
    ignore the warnings they generated.”
    “We couldn’t figure out the impact of the broken acceptance
    tests, and assumed the only problem was the tests themselves,
    rather than the underlying code.”
    “The commit went through as there weren't any tests for
    the serialisation from the client to the backend.”
    Digging into post-mortems
    © ofsmallthings

    View Slide

  14. Introducing contract tests

    View Slide

  15. Introducing contract tests
    My daily routine
    © LockeSteerpike

    View Slide

  16. Introducing contract tests
    My daily routine
    © rickandmorty.com

    View Slide

  17. Unit tests are not enough
    Introducing contract tests
    “#cake” >> {
    result = call(“cake”)
    assert(result == “lie”)
    }
    GET “/cake”:
    return “lie”

    View Slide

  18. Unit tests are not enough
    Introducing contract tests
    GET “/cake”:
    return “truth”
    “#cake” >> {
    result = call(“cake”)
    assert(result == “lie”)
    }

    View Slide

  19. Unit tests are not enough
    Introducing contract tests
    import JsonLibFoo
    GET “/cake”:
    return toJson(current_state)
    // { “current_state” : “lie” }
    import JsonLibFoo
    “#cake” >> {
    result = fromJson(call(“cake”))
    assert(result == “lie”)
    }

    View Slide

  20. Unit tests are not enough
    Introducing contract tests
    import JsonLibBar
    GET “/cake”:
    return toJson(current_state)
    // { “currentState” : “lie” }
    import JsonLibBar
    “#cake” >> {
    result = fromJson(call(“cake”))
    assert(result == “lie”)
    }

    View Slide

  21. Introducing contract tests
    How this works?
    Consumer Provider
    End-to-End Test

    View Slide

  22. Introducing contract tests
    How this works?
    Consumer Provider
    Unit Test Unit Test

    View Slide

  23. Introducing contract tests
    How this works?
    Consumer Provider
    Unit Test Unit Test

    View Slide

  24. Introducing contract tests
    How this works?
    ➢ Requesting “/cake”.
    ➢ Expecting a JSON response
    with a key “current_state”.
    ➢ Asserting deserialization
    of the response succeeds.
    import JsonLibFoo
    GET “/cake”:
    return toJson(current_state)
    // { “current_state” : “lie” }
    Consumer
    Provider

    View Slide

  25. Pactifying our services
    http://pact.io

    View Slide

  26. What’s Pact?
    Pactifying our services
    ● A family of frameworks designed for contract testing.
    ● Supports JSON over HTTP.
    ● Impl. for Java, Scala, Go, Ruby, .Net, Swift, JS etc.
    ● Tests run locally, no external dependencies.

    View Slide

  27. What’s Pact?
    Pactifying our services
    Step 1: Define consumer expectations
    © pact.io

    View Slide

  28. What’s Pact?
    Pactifying our services
    Step 2: Verify expectations on provider
    © pact.io

    View Slide

  29. Consumer: client code

    View Slide

  30. Consumer: client code

    View Slide

  31. Consumer: client code

    View Slide

  32. Consumer: client code

    View Slide

  33. Consumer: client code

    View Slide

  34. Consumer: client code

    View Slide

  35. Consumer: client unit test

    View Slide

  36. Consumer: client unit test

    View Slide

  37. Consumer: client unit test

    View Slide

  38. Consumer: client unit test

    View Slide

  39. Consumer: client contract unit test

    View Slide

  40. Consumer: client contract unit test

    View Slide

  41. Consumer: client contract unit test

    View Slide

  42. Consumer: client contract unit test

    View Slide

  43. Consumer: client contract unit test

    View Slide

  44. Consumer: client contract unit test

    View Slide

  45. Consumer: client contract unit test

    View Slide

  46. Consumer: client contract unit test

    View Slide

  47. Consumer: my-consumer-likes.json

    View Slide

  48. Consumer: my-consumer-likes.json

    View Slide

  49. Consumer: my-consumer-likes.json

    View Slide

  50. Consumer: my-consumer-likes.json

    View Slide

  51. Consumer: my-consumer-likes.json

    View Slide

  52. Consumer: my-consumer-likes.json

    View Slide

  53. Consumer: my-consumer-likes.json

    View Slide

  54. Pactifying our services
    Provider side verification
    ● Collect pact files from consumers and verify them all.
    ● Keep your provider environment isolated.
    ○ Dockerized databases.
    ○ Test doubles for upstream services.

    View Slide

  55. Pactifying our services
    Provider states

    View Slide

  56. Pactifying our services
    Provider states
    Likes
    App
    Likes
    DB

    View Slide

  57. Pactifying our services
    Provider states
    Set state:
    “User 1000 has liked 2 tracks”
    Likes
    App
    Likes
    DB

    View Slide

  58. Pactifying our services
    Provider states
    INSERT
    likes
    Set state:
    “User 1000 has liked 2 tracks”
    Likes
    App
    Likes
    DB

    View Slide

  59. Pactifying our services
    Provider states
    INSERT
    likes
    Set state:
    “User 1000 has liked 2 tracks”
    HTTP Request: GET /likes/1000
    Likes
    App
    Likes
    DB

    View Slide

  60. Pactifying our services
    Provider states
    HTTP Request: GET /likes/1000
    HTTP Response: 200 OK
    INSERT
    likes
    Set state:
    “User 1000 has liked 2 tracks”
    Likes
    App
    Likes
    DB

    View Slide

  61. Pactifying our services
    Provider states
    Likes
    App
    Likes
    DB
    HTTP Response: 200 OK
    INSERT
    likes
    Set state:
    “User 1000 has liked 2 tracks”
    HTTP Request: GET /likes/2000
    HTTP Response: 404 Not Found
    Set state:
    “User 2000 has liked no tracks”
    HTTP Request: GET /likes/1000
    DELETE
    likes

    View Slide

  62. Pactifying our services
    Pact broker
    ● Share pact files between projects.
    ● API for pact frameworks to fetch all pacts by provider.
    ● Support for versioning and tagging.
    ● Useful UI and visualization of the network.

    View Slide

  63. Pactifying our services
    Pact broker
    © pact broker

    View Slide

  64. Pactifying our services
    Pact broker
    © pact broker

    View Slide

  65. Pactifying our services
    Pact broker
    © pact broker

    View Slide

  66. Pactifying our services
    Our CI pipeline
    Push new
    change
    Generate
    consumer
    contracts
    Deploy
    Upload
    pacts & tag
    with ‘prod’
    Consumer pipeline

    View Slide

  67. Pactifying our services
    Our CI pipeline
    Push new
    change
    Deploy
    Upload
    pacts & tag
    with ‘prod’
    Consumer pipeline
    Push new
    change
    Verify all
    ‘prod’-tagged
    pacts
    Deploy
    Provider pipeline
    Generate
    consumer
    contracts

    View Slide

  68. ● Communication is key.
    Caveats

    View Slide

  69. ● Communication is key.
    ● New moving parts.
    Caveats

    View Slide

  70. ● Communication is key.
    ● New moving parts.
    ● Learning curve per pact framework.
    Caveats

    View Slide

  71. ● Communication is key.
    ● New moving parts.
    ● Learning curve per pact framework.
    ● Frameworks are WIP.
    Caveats

    View Slide

  72. ● Communication is key.
    ● New moving parts.
    ● Learning curve per pact framework.
    ● Frameworks are WIP.
    ● Postel's law (the robustness principle).
    Caveats

    View Slide

  73. ● Communication is key.
    ● New moving parts.
    ● Learning curve per pact framework.
    ● Frameworks are WIP.
    ● Postel's law (the robustness principle).
    ● Provider side setup is time consuming.
    Caveats

    View Slide

  74. ● Communication is key.
    ● New moving parts.
    ● Learning curve per pact framework.
    ● Frameworks are WIP.
    ● Postel's law (the robustness principle).
    ● Provider side setup is time consuming.
    ● Automating consumer-triggered provider verification.
    Caveats

    View Slide

  75. ● Communication is key.
    ● New moving parts.
    ● Learning curve per pact framework.
    ● Frameworks are WIP.
    ● Postel's law (the robustness principle).
    ● Provider side setup is time consuming.
    ● Automating consumer-triggered provider verification.
    ● Adoption is essential.
    Caveats

    View Slide

  76. Recap

    View Slide

  77. Recap
    Write contract tests
    Consum
    ers

    View Slide

  78. Recap
    Write contract tests Generate pact files
    Consum
    ers

    View Slide

  79. Recap
    Write contract tests Generate pact files
    Publish to broker
    Consum
    ers

    View Slide

  80. Recap
    Write contract tests Generate pact files
    Publish to broker
    Verify pacts
    Consum
    ers
    Providers

    View Slide

  81. Recap
    Write contract tests Generate pact files
    Publish to broker
    Verify pacts
    Consum
    ers
    Providers

    View Slide

  82. QA Q&A
    @alonpeer

    View Slide