Demystifying the Test Pyramid

Back in 2009, Mike Cohn in his "Succeeding with Agile" book, introduced the pyramid of tests. Since then, it has been criticised (mostly for being unsuitable for many real life situations), slightly updated, but its core concept is still seen in many articles. So what value can we get out of the test pyramid, and how can you adapt it to better fit with your use cases, workflows and setup.

After explaining the original concept of the testing pyramid, we will discuss its pitfalls and caveats, then provide more depth and a new perspective on this tool while providing tips and guidelines that engineers can use to improve their test suites.

Transcript

1. Demystifying the test pyramid
2. Outline - History of the Test pyramid - Original pyramid - What is it ? - Flaws and evolutions - A word about coverage - How to look at the pyramid in 2023 - A 3d Pyramid with 4 sides: What, Why, How, When·re - Conclusion
3. Xavier F. Gouchet Senior Software Engineer at Datadog @xgouchet / @datadoghq
6. About the Test Pyramid…
7. “ ” An effective test automation strategy calls for automating tests at three different levels […] . If the automated test suite is growing daily and has run without error every night of this sprint, the team is probably in good shape. — Mike Cohn Succeeding with Agile Mike Cohn (2009)
8. Succeeding with Agile Mike Cohn (2009)
9. Unit Testing a “unit” independantly from any other code. + fast and reliable in CI + lots of mocks - tight coupling with implementation
10. Service Testing two or more units interacting together. + validates integration before end-to-end tests + mock network and DB - non exhaustive - fault localization is difficult a.k.a. Integration Test
11. UI Test the product as the end user would. + validates user experience - fragility and flakiness - hard to debug a.k.a. End to End Test
13. • Popularized the idea of automated tests ◦ But like all popular things, it’s often taken at face value • Gives the wrong idea that the volume is important ◦ number Unit Tests > number of Integration Tests > number of E2E Tests ◦ driven by the cost of writing/running/maintaining tests • Doesn’t match all types of projects • It’s a triangle, not a pyramid… ◦ easy to understand, but lacks nuances What’s up with the test pyramid
14. “ ” All models are approximations. Essentially, all models are wrong, but some are useful. — Georges Box
15. The Pyramid revisited
16. Noah Sussman (2017)
17. “ ” I interpret the agile testing pyramid as a beginning to a discussion about test strategy in an agile team. — Sharon Robson More Agile Testing Lisa Crispin, Janet Gregory (2014)
18. “ ” A pyramid is a self-supporting structure that cannot exist without each of its sides. Each has a different focus (or direction), but each ties into the whole unit. — Sharon Robson More Agile Testing Lisa Crispin, Janet Gregory (2014)
19. More Agile Testing Lisa Crispin, Janet Gregory (2014)
20. The four sides of the pyramid • What are we testing? • How are we testing it? • Why are we testing? What are we looking to gain more confidence on? • When are we running the tests? And Where?
21. What…? End to End Tests Integration Tests Unit Tests Static Analysis
22. What…? End to End Tests Integration Tests Unit Tests Static Analysis • Levels size don’t relate to… ◦ Number of tests ◦ Importance ◦ Speed • Add a level for… ◦ Business Rules ◦ APIs / Data Format ◦ Security
23. Why…? Usability Functionality Reliability Performance Code Quality Maintenance End to End Tests Integration Tests Unit Tests Static Analysis
24. Why…? Usability Functionality Reliability Performance Code Quality Maintenance • And much more: ◦ “Verify that the code works” ◦ Make the contract explicit ◦ Guide the development (TDD) ◦ Prevent regression caused by ▪ Other devs ▪ Ourselves in the future ◦ Ensure retrocompatibility
25. How…? Datadog, Maestro, … JUnit, Espresso, … JUnit, Spek, … Detekt Ktlint Usability Functionality Reliability Performance Code quality Maintenance
26. How…? Datadog, Maestro, … JUnit, Espresso, … JUnit, Spek, … Detekt Ktlint
27. When…? Nightly/pre release Per PR/Nightly Per Commit/PR Per commit Datadog, Maestro, … JUnit, Espresso, … JUnit, Spek, … Detekt Ktlint
28. When…? Nightly/pre release Per PR/Nightly Per Commit/PR Per commit • Local ◦ Reproducible environment setup ◦ Easy to run locally • CI ◦ VCS provided (Github Actions, Gitlab, …) ◦ Third Party Vendor (Bitrise, Circle CI, Travis, …) ◦ Custom hosted (Jenkins, …)
29. About tests roles…
30. “ ” Different tests also play different roles in a project. The traditional role of automated tests is to check that an application behaves correctly. However, tests can also play a role as "living documentation". — John Ferguson Smart A Test Pyramid Heresy John Ferguson Smart (2017)
31. Tests that Discover • Example based • TDD/BDD • Discover the shape of a feature ◦ API, interface, signatures, …
32. Tests that Describe • Make the contract explicit • Explain Business Rule / Domain Logic • Living documentation
33. Tests that Demonstrate • Complex or global scenarios • Detail oriented • Edge cases
34. About coverage…
35. Feature Compliance Input Coverage Compiler Static Analysis Monkey Tests Unit Tests Integration Tests E2E Tests Fuzzy Tests PBT
36. SUCCESS: 26/26 (100%) Tests passed Bernd Lörwald @bloerwald
37. “ ” The more any quantitative indicator is used for decision-making, the more subject it will be to corruption pressure. — Donald T. Campbell
38. “ ” When code coverage is at 80%, it doesn’t mean 80% of your code is tested; it means at least 20% is not tested.
39. Wrapping up
40. “ ” The test pyramid is more what you’d call guidelines than actual rules.
41. • You don’t need to have 4 levels exactly • Have as many tests as you can, for each level • Discuss the pyramid regularly • You can (should) have conventions for tests Takeaway
42. “ ” You need to be as confident in the tests you code as you are in the code you test.
43. Thank you! Any questions?
44. A 4 sided pyramid LEVEL 01 LEVEL 02 LEVEL 03 LEVEL 04 Lorem Ipsum Lorem Ipsum Lorem Ipsum Lorem Ipsum
45. A 4 sided pyramid LEVEL 01 LEVEL 02 LEVEL 03 LEVEL 04 Lorem Ipsum Lorem Ipsum Lorem Ipsum Lorem Ipsum