Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Demystifying the Test Pyramid

Demystifying the Test Pyramid

Back in 2009, Mike Cohn in his "Succeeding with Agile" book, introduced the pyramid of tests. Since then, it has been criticised (mostly for being unsuitable for many real life situations), slightly updated, but its core concept is still seen in many articles. So what value can we get out of the test pyramid, and how can you adapt it to better fit with your use cases, workflows and setup.

After explaining the original concept of the testing pyramid, we will discuss its pitfalls and caveats, then provide more depth and a new perspective on this tool while providing tips and guidelines that engineers can use to improve their test suites.

Xavier Gouchet

April 27, 2023

More Decks by Xavier Gouchet

Other Decks in Programming


  1. Outline - History of the Test pyramid - Original pyramid

    - What is it ? - Flaws and evolutions - A word about coverage - How to look at the pyramid in 2023 - A 3d Pyramid with 4 sides: What, Why, How, When·re - Conclusion
  2. 4

  3. “ ” An effective test automation strategy calls for automating

    tests at three different levels […] . If the automated test suite is growing daily and has run without error every night of this sprint, the team is probably in good shape. — Mike Cohn Succeeding with Agile Mike Cohn (2009)
  4. Unit Testing a “unit” independantly from any other code. +

    fast and reliable in CI + lots of mocks - tight coupling with implementation
  5. Service Testing two or more units interacting together. + validates

    integration before end-to-end tests + mock network and DB - non exhaustive - fault localization is difficult a.k.a. Integration Test
  6. UI Test the product as the end user would. +

    validates user experience - fragility and flakiness - hard to debug a.k.a. End to End Test
  7. • Popularized the idea of automated tests ◦ But like

    all popular things, it’s often taken at face value • Gives the wrong idea that the volume is important ◦ number Unit Tests > number of Integration Tests > number of E2E Tests ◦ driven by the cost of writing/running/maintaining tests • Doesn’t match all types of projects • It’s a triangle, not a pyramid… ◦ easy to understand, but lacks nuances What’s up with the test pyramid
  8. “ ” All models are approximations. Essentially, all models are

    wrong, but some are useful. — Georges Box
  9. “ ” I interpret the agile testing pyramid as a

    beginning to a discussion about test strategy in an agile team. — Sharon Robson More Agile Testing Lisa Crispin, Janet Gregory (2014)
  10. “ ” A pyramid is a self-supporting structure that cannot

    exist without each of its sides. Each has a different focus (or direction), but each ties into the whole unit. — Sharon Robson More Agile Testing Lisa Crispin, Janet Gregory (2014)
  11. The four sides of the pyramid • What are we

    testing? • How are we testing it? • Why are we testing? What are we looking to gain more confidence on? • When are we running the tests? And Where?
  12. What…? End to End Tests Integration Tests Unit Tests Static

    Analysis • Levels size don’t relate to… ◦ Number of tests ◦ Importance ◦ Speed • Add a level for… ◦ Business Rules ◦ APIs / Data Format ◦ Security
  13. Why…? Usability Functionality Reliability Performance Code Quality Maintenance • And

    much more: ◦ “Verify that the code works” ◦ Make the contract explicit ◦ Guide the development (TDD) ◦ Prevent regression caused by ▪ Other devs ▪ Ourselves in the future ◦ Ensure retrocompatibility
  14. How…? Datadog, Maestro, … JUnit, Espresso, … JUnit, Spek, …

    Detekt Ktlint Usability Functionality Reliability Performance Code quality Maintenance
  15. When…? Nightly/pre release Per PR/Nightly Per Commit/PR Per commit Datadog,

    Maestro, … JUnit, Espresso, … JUnit, Spek, … Detekt Ktlint
  16. When…? Nightly/pre release Per PR/Nightly Per Commit/PR Per commit •

    Local ◦ Reproducible environment setup ◦ Easy to run locally • CI ◦ VCS provided (Github Actions, Gitlab, …) ◦ Third Party Vendor (Bitrise, Circle CI, Travis, …) ◦ Custom hosted (Jenkins, …)
  17. “ ” Different tests also play different roles in a

    project. The traditional role of automated tests is to check that an application behaves correctly. However, tests can also play a role as "living documentation". — John Ferguson Smart A Test Pyramid Heresy John Ferguson Smart (2017)
  18. Tests that Discover • Example based • TDD/BDD • Discover

    the shape of a feature ◦ API, interface, signatures, …
  19. Tests that Describe • Make the contract explicit • Explain

    Business Rule / Domain Logic • Living documentation
  20. Feature Compliance Input Coverage Compiler Static Analysis Monkey Tests Unit

    Tests Integration Tests E2E Tests Fuzzy Tests PBT
  21. “ ” The more any quantitative indicator is used for

    decision-making, the more subject it will be to corruption pressure. — Donald T. Campbell
  22. “ ” When code coverage is at 80%, it doesn’t

    mean 80% of your code is tested; it means at least 20% is not tested.
  23. • You don’t need to have 4 levels exactly •

    Have as many tests as you can, for each level • Discuss the pyramid regularly • You can (should) have conventions for tests Takeaway
  24. “ ” You need to be as confident in the

    tests you code as you are in the code you test.
  25. A 4 sided pyramid LEVEL 01 LEVEL 02 LEVEL 03

    LEVEL 04 Lorem Ipsum Lorem Ipsum Lorem Ipsum Lorem Ipsum
  26. A 4 sided pyramid LEVEL 01 LEVEL 02 LEVEL 03

    LEVEL 04 Lorem Ipsum Lorem Ipsum Lorem Ipsum Lorem Ipsum