Avoiding common pitfalls with modern microservices testing

Have you ever wondered why your microservices break in production after all the tests have passed during CI/CD? Have you ever wondered if you’re testing too much or too little? If you want to learn about common pitfalls and how to catch them at build time rather than at runtime, this session is for you!

In this session we’ll share some common battle scars from our experiences in the field. Additionally, we will introduce the testing pyramid and explain why it is important in helping to continuously deploy microservices independently without fear. We will demonstrate, through a test-driven development approach, how the combination of Pact contract testing combined with Quarkus developer joy can help prevent your microservices from breaking in production.

Transcript

1. Holly Cummins Sr. Principal Software Engineer Red Hat @holly_cummins Eric Deandrea Sr. Principal Developer Advocate Red Hat @edeandrea Avoiding common pitfalls with modern microservices testing
2. @holly_cummins @edeandrea #DevoxxUK #RedHat “Every time we touch one microservice, all the others break.”
3. @holly_cummins @edeandrea #DevoxxUK #RedHat “My end-to-end test environment is only up the third weekend every month.”
4. @holly_cummins @edeandrea #DevoxxUK #RedHat Imposter Disclaimer!
5. @holly_cummins @edeandrea #DevoxxUK #RedHat Imposter Disclaimer! We are not testers. (or even test experts)
6. @holly_cummins @edeandrea #DevoxxUK #RedHat Imposter Disclaimer! We are not testers. (or even test experts) That’s ok– testing is every developer’s job!
7. @holly_cummins @edeandrea #DevoxxUK #RedHat Why is testing microservices so hard?
8. @holly_cummins @edeandrea #DevoxxUK #RedHat Netflix microservice architecture
9. @holly_cummins @edeandrea #DevoxxUK #RedHat Death star “That’s no moon. It’s our architecture.”
10. @holly_cummins @edeandrea #DevoxxUK #RedHat Do microservices solve all problems?
11. @holly_cummins @edeandrea #DevoxxUK #RedHat A microservice system: a portrait
14. @holly_cummins @edeandrea #DevoxxUK #RedHat Netflix microservice architecture
15. @holly_cummins @edeandrea #DevoxxUK #RedHat Netflix microservice architecture Testing with microservices is easy!
16. @holly_cummins @edeandrea #DevoxxUK #RedHat Netflix microservice architecture Testing with microservices is easy! My services are so small!
17. @holly_cummins @edeandrea #DevoxxUK #RedHat Netflix microservice architecture Testing with microservices is easy! My services are so small! Everything is decoupled! But…..
18. @holly_cummins @edeandrea #DevoxxUK #RedHat Netflix microservice architecture Testing with microservices is easy! My services are so small! Everything is decoupled! But….. If you only test the points, And not the lines…. Does the thing actually work?
19. @holly_cummins @edeandrea #DevoxxUK #RedHat distributed != decoupled
20. @holly_cummins @edeandrea #DevoxxUK #RedHat Understand the coupling
21. @holly_cummins @edeandrea #DevoxxUK #RedHat “It is all that is left unsaid upon which tragedies are built.” - Kreia, Star Wars Knights of the Old Republic II: The Sith Lords
22. @holly_cummins @edeandrea #DevoxxUK #RedHat
23. @holly_cummins @edeandrea #DevoxxUK #RedHat This is not the answer.
24. @holly_cummins @edeandrea #DevoxxUK #RedHat “You lied to me. How many other lies have I been told by the Council? And how do you know that you even have the truth?” – Anakin Skywalker
25. @holly_cummins @edeandrea #DevoxxUK #RedHat “In my book, experience outranks everything.” — Captain Rex, The Clone Wars
26. @holly_cummins @edeandrea #DevoxxUK #RedHat Ok, so how do we try these out?
27. @holly_cummins @edeandrea #DevoxxUK #RedHat Get good at testing.
28. @holly_cummins @edeandrea #DevoxxUK #RedHat Get good at testing. And automation.
30. end-to-end tests high realism high value
31. end-to-end tests unit tests low effort high realism high value
32. end-to-end tests unit tests integration tests low effort high realism high value
33. end-to-end tests unit tests integration tests low effort high realism
34. end-to-end tests unit tests integration tests low effort high realism tests with application server test REST endpoints tests with a database
35. @holly_cummins @edeandrea #DevoxxUK #RedHat their code our code Why mocks aren’t enough
36. @holly_cummins @edeandrea #DevoxxUK #RedHat our mock our code Why mocks aren’t enough
37. @holly_cummins @edeandrea #DevoxxUK #RedHat our mock our code ✔ tests Why mocks aren’t enough
38. @holly_cummins @edeandrea #DevoxxUK #RedHat our mock our code Why mocks aren’t enough
39. @holly_cummins @edeandrea #DevoxxUK #RedHat our mock our code ✔ tests ✘ reality our code their actual code Why mocks aren’t enough
40. @holly_cummins @edeandrea #DevoxxUK #RedHat a mock stormtrooper
41. @holly_cummins @edeandrea #DevoxxUK #RedHat a mock stormtrooper “Aren’t you a little short for a stormtrooper?” – Princess Leia
42. @holly_cummins @edeandrea #DevoxxUK #RedHat https://www.flickr.com/photos/jdhancock/4811282106 Source: https://medium.com/jung-skywalker/star-wars-symbology-luke-han-and-how-to-face-the-abyss-e9fa966319b5
43. @holly_cummins @edeandrea #DevoxxUK #RedHat Demo The limitations of unit tests https://github.com/edeandrea/pact-quarkus-wookie-carpet-demo
44. @holly_cummins @edeandrea #DevoxxUK #RedHat “Will someone get this big walking carpet out of my way?” – Princess Leia Demo The limitations of unit tests https://github.com/edeandrea/pact-quarkus-wookie-carpet-demo
45. @holly_cummins @edeandrea #DevoxxUK #RedHat “Will someone get this big walking carpet out of my way?” – Princess Leia Demo The limitations of unit tests https://github.com/edeandrea/pact-quarkus-wookie-carpet-demo
46. @holly_cummins @edeandrea #DevoxxUK #RedHat A potential solution
47. @holly_cummins @edeandrea #DevoxxUK #RedHat How to test a fire alarm?
48. @holly_cummins @edeandrea #DevoxxUK #RedHat
49. @holly_cummins @edeandrea #DevoxxUK #RedHat How NOT to test a fire alarm!
50. @holly_cummins @edeandrea #DevoxxUK #RedHat
52. Unit testing a fire alarm
53. @holly_cummins @edeandrea #DevoxxUK #RedHat uh … is that enough?
54. @holly_cummins @edeandrea #DevoxxUK #RedHat
55. @holly_cummins @edeandrea #DevoxxUK #RedHat contract testing a fire alarm
56. Photo: https://www.westcoastfirepits.com/shop/death-star
57. Photo: https://www.westcoastfirepits.com/shop/death-star How to SAFELY burn a Death Star!
58. end-to-end tests unit tests integration tests low effort high realism high value
59. end-to-end tests unit tests integration tests low effort high realism high value contract tests
60. @holly_cummins @edeandrea #DevoxxUK #RedHat our code their code A better mock
61. @holly_cummins @edeandrea #DevoxxUK #RedHat our code their code contract test mock functional test A better mock
62. @holly_cummins @edeandrea #DevoxxUK #RedHat ✔ our tests ✔ their tests ✔ reality our code their code contract test mock functional test A better mock
63. @holly_cummins @edeandrea #DevoxxUK #RedHat our code their code A better mock
64. @holly_cummins @edeandrea #DevoxxUK #RedHat our code their code contract test mock functional test A better mock
65. @holly_cummins @edeandrea #DevoxxUK #RedHat ✔ our tests ✘ their tests ✘ reality our code their code contract test mock functional test A better mock
66. @holly_cummins @edeandrea #DevoxxUK #RedHat A better mock our code their code
67. @holly_cummins @edeandrea #DevoxxUK #RedHat A better mock our code their code contract test mock functional test
68. @holly_cummins @edeandrea #DevoxxUK #RedHat ✘ our tests ✔ their tests ✘ reality A better mock our code their code contract test mock functional test
69. @holly_cummins @edeandrea #DevoxxUK #RedHat Pact Contract Testing
70. @holly_cummins @edeandrea #DevoxxUK #RedHat Demo
71. @holly_cummins @edeandrea #DevoxxUK #RedHat “Help me, Pact! You’re my only hope!” – Princess Leia (almost) Demo
72. @holly_cummins @edeandrea #DevoxxUK #RedHat “Uh, you’re telling me that consumers can break a provider’s tests?”
74. @holly_cummins @edeandrea #DevoxxUK #RedHat Consumer-driven contract testing
75. @holly_cummins @edeandrea #DevoxxUK #RedHat consumer Consumer-driven contract testing
76. @holly_cummins @edeandrea #DevoxxUK #RedHat contract consumer tests declare (Pact) Pact file Consumer-driven contract testing Spring Cloud Contract
77. @holly_cummins @edeandrea #DevoxxUK #RedHat contract consumer tests mock declare (Pact) Pact file Consumer-driven contract testing Spring Cloud Contract Pact
78. @holly_cummins @edeandrea #DevoxxUK #RedHat contract consumer tests provider mock declare (Pact) Pact file Consumer-driven contract testing Spring Cloud Contract Pact
79. @holly_cummins @edeandrea #DevoxxUK #RedHat contract consumer tests provider mock declare (Pact) contract tests verify Pact file Consumer-driven contract testing Spring Cloud Contract Pact Pact
80. @holly_cummins @edeandrea #DevoxxUK #RedHat contract consumer tests provider mock declare (Pact) contract tests verify Pact file Consumer-driven contract testing ☑ rich semantic testing Spring Cloud Contract Pact Pact
81. @holly_cummins @edeandrea #DevoxxUK #RedHat contract consumer tests provider mock declare (Pact) contract tests verify Pact file Consumer-driven contract testing ☑ rich semantic testing ☑ provider can ‘develop to the contract’ instead of developing speculatively (TDD) Spring Cloud Contract Pact Pact
82. @holly_cummins @edeandrea #DevoxxUK #RedHat contract consumer tests provider mock declare (Pact) contract tests verify Pact file Consumer-driven contract testing ☑ rich semantic testing ☑ provider can ‘develop to the contract’ instead of developing speculatively (TDD) ⛔ existing spec assets cannot be re-purposed Spring Cloud Contract Pact Pact
83. @holly_cummins @edeandrea #DevoxxUK #RedHat contract consumer tests provider mock declare (Pact) contract tests verify Pact file Consumer-driven contract testing ☑ rich semantic testing ☑ provider can ‘develop to the contract’ instead of developing speculatively (TDD) ⛔ existing spec assets cannot be re-purposed ⛔ provider needs to know who consumers are Spring Cloud Contract Pact Pact
84. @holly_cummins @edeandrea #DevoxxUK #RedHat contract consumer tests provider mock declare (Pact) contract tests verify Pact file Consumer-driven contract testing ☑ rich semantic testing ☑ provider can ‘develop to the contract’ instead of developing speculatively (TDD) ⛔ existing spec assets cannot be re-purposed ⛔ provider needs to know who consumers are ⛔ consumers can break provider CI Spring Cloud Contract Pact Pact
85. @holly_cummins @edeandrea #DevoxxUK #RedHat …but I want to just use Swagger/OpenAPI
86. @holly_cummins @edeandrea #DevoxxUK #RedHat Is Pact always the best choice?
87. @holly_cummins @edeandrea #DevoxxUK #RedHat Provider-driven contract testing
88. @holly_cummins @edeandrea #DevoxxUK #RedHat provider Provider-driven contract testing
89. @holly_cummins @edeandrea #DevoxxUK #RedHat contract provider OpenAPI Provider-driven contract testing generate (OpenAPI snapshot of current implementation) other protocol specs
90. @holly_cummins @edeandrea #DevoxxUK #RedHat contract provider contract tests verify OpenAPI Provider-driven contract testing generate (OpenAPI snapshot of current implementation) other protocol specs Schemathesis Microcks
91. @holly_cummins @edeandrea #DevoxxUK #RedHat contract consumer provider contract tests verify OpenAPI Provider-driven contract testing generate (OpenAPI snapshot of current implementation) other protocol specs Schemathesis Microcks
92. @holly_cummins @edeandrea #DevoxxUK #RedHat contract consumer tests provider mock contract tests verify OpenAPI Provider-driven contract testing generate (OpenAPI snapshot of current implementation) other protocol specs Prism Microcks Schemathesis Microcks
93. @holly_cummins @edeandrea #DevoxxUK #RedHat contract consumer tests provider mock contract tests verify OpenAPI Provider-driven contract testing generate (OpenAPI snapshot of current implementation) other protocol specs ☑ familiar contract format Prism Microcks Schemathesis Microcks
94. @holly_cummins @edeandrea #DevoxxUK #RedHat contract consumer tests provider mock contract tests verify OpenAPI Provider-driven contract testing generate (OpenAPI snapshot of current implementation) other protocol specs ☑ familiar contract format ☑ provider does not need comms with consumers Prism Microcks Schemathesis Microcks
95. @holly_cummins @edeandrea #DevoxxUK #RedHat contract consumer tests provider mock contract tests verify OpenAPI Provider-driven contract testing generate (OpenAPI snapshot of current implementation) other protocol specs ☑ familiar contract format ☑ provider does not need comms with consumers ⛔ provider does not get insight into how API is being used (deprecation is harder) Prism Microcks Schemathesis Microcks
96. @holly_cummins @edeandrea #DevoxxUK #RedHat contract consumer tests provider mock contract tests verify OpenAPI Provider-driven contract testing generate (OpenAPI snapshot of current implementation) other protocol specs ☑ familiar contract format ☑ provider does not need comms with consumers ⛔ provider does not get insight into how API is being used (deprecation is harder) ⛔ spec-based tests lack functional depth Prism Microcks Schemathesis Microcks
97. @holly_cummins @edeandrea #DevoxxUK #RedHat Is OpenAPI enough? It has the same syntax, but the semantics are different.
98. @holly_cummins @edeandrea #DevoxxUK #RedHat Is OpenAPI enough? It has the same syntax, but the semantics are different. It looks the same, but the behaviour is different.
99. @holly_cummins @edeandrea #DevoxxUK #RedHat Is OpenAPI enough? It has the same syntax, but the semantics are different. It looks the same, but the behaviour is different. Looks like a storm trooper. Behaves like Luke. (because it it is Luke)
100. @holly_cummins @edeandrea #DevoxxUK #RedHat Demo: Semantic testing with Pact
101. @holly_cummins @edeandrea #DevoxxUK #RedHat how annoying, the provider is returning 204. the contract expects 404, so my test is broken. quarkus will return 204 by default. 204 vs 404 - it’s more about the theory behind REST...if a caller asks for a resource the server doesn't have, it should be a 404 - same way if a user types in an invalid URL into the browser ugh, it’s too late at night to care about REST theory. I’m just going to update the expectations in the test to match reality. what?! why is my other contract test broken now? 
 oh … because what I’m changing is a mock…. Based on a true story….
102. @holly_cummins @edeandrea #DevoxxUK #RedHat
103. @holly_cummins @edeandrea #DevoxxUK #RedHat “contract-first”
104. @holly_cummins @edeandrea #DevoxxUK #RedHat “contract-first”
105. @holly_cummins @edeandrea #DevoxxUK #RedHat “contract-first” It’s like TDD …
106. @holly_cummins @edeandrea #DevoxxUK #RedHat “contract-first” It’s like TDD … … between teams for consumer-driven
107. @holly_cummins @edeandrea #DevoxxUK #RedHat “contract-first” It’s like TDD … … between teams for consumer-driven … with yourself, for provider-driven
108. @holly_cummins @edeandrea #DevoxxUK #RedHat
109. @holly_cummins @edeandrea #DevoxxUK #RedHat soo …
110. @holly_cummins @edeandrea #DevoxxUK #RedHat soo …
111. @holly_cummins @edeandrea #DevoxxUK #RedHat soo … microservices testing is hard
112. @holly_cummins @edeandrea #DevoxxUK #RedHat soo … microservices testing is hard limit end-to-end testing
113. @holly_cummins @edeandrea #DevoxxUK #RedHat soo … microservices testing is hard limit end-to-end testing mocks will not be enough
114. @holly_cummins @edeandrea #DevoxxUK #RedHat soo … microservices testing is hard limit end-to-end testing mocks will not be enough contract testing can rescue you
115. @holly_cummins @edeandrea #DevoxxUK #RedHat ® Thank you! @holly_cummins 
 @edeandrea slides
116. @holly_cummins @edeandrea #DevoxxUK #RedHat Additional Resources
117. @holly_cummins @edeandrea #DevoxxUK #RedHat https://github.com/quarkusio/quarkus-super-heroes More realistic sample application (fewer wookies)
118. @holly_cummins @edeandrea #DevoxxUK #RedHat Kubernetes Native Java with Quarkus Deployment Developer Joy • Quickly & easily build Kubernetes-Native Java applications • Industry leader in startup time and memory utilization for native and JVM applications • One step native executable generation • Leverage existing Java, Jakarta, MicroProfile, & Spring expertise • Imperative & reactive APIs side-by-side • Live coding with zero config https://www.infoq.com/articles/native-java-quarkus
119. @holly_cummins @edeandrea #DevoxxUK #RedHat https://www.oreilly.com/library/view/reactive-systems-in/9781492091714 https://developers.redhat.com/e-books/understanding-quarkus https://developers.redhat.com/e-books/practising-quarkus