Demystifying the Test Pyramid

Back in 2009, Mike Cohn in his "Succeeding with Agile" book, introduced the pyramid of tests. Since then, it has been criticised (mostly for being unsuitable for many real life situations), slightly updated, but its core concept is still seen in many articles. So what value can we get out of the test pyramid, and how can you adapt it to better fit with your use cases, workflows and setup.

After explaining the original concept of the testing pyramid, we will discuss its pitfalls and caveats, then provide more depth and a new perspective on this tool while providing tips and guidelines that engineers can use to improve their test suites.

Transcript

1. Demystifying
   the test
   pyramid
   

   View Slide

2. Outline
   - History of the Test pyramid
   - Original pyramid
   - What is it ?
   - Flaws and evolutions
   - A word about coverage
   - How to look at the pyramid in 2023
   - A 3d Pyramid with 4 sides: What, Why, How, When·re
   - Conclusion
   

   View Slide

3. Xavier F. Gouchet
   Senior Software Engineer
   at Datadog
   @xgouchet / @datadoghq
   

   View Slide

4. View Slide

5. View Slide

6. About the Test Pyramid…
   

   View Slide

7. “
   ”
   An effective test automation strategy calls for
   automating tests at three different levels […] .
   If the automated test suite is growing daily and has
   run without error every night of this sprint, the
   team is probably in good shape.
   — Mike Cohn
   Succeeding with Agile
   Mike Cohn (2009)
   

   View Slide

8. Succeeding with Agile
   Mike Cohn (2009)
   

   View Slide

9. Unit
   Testing a “unit” independantly
   from any other code.
   + fast and reliable in CI
   + lots of mocks
   - tight coupling with
   implementation
   

   View Slide

10. Service
    Testing two or more units
    interacting together.
    + validates integration before
    end-to-end tests
    + mock network and DB
    - non exhaustive
    - fault localization is difficult
    a.k.a. Integration Test
    

    View Slide

11. UI
    Test the product as the end user
    would.
    + validates user experience
    - fragility and flakiness
    - hard to debug a.k.a. End to End Test
    

    View Slide

12. View Slide

13. ● Popularized the idea of automated tests
    ○ But like all popular things, it’s often taken at face value
    ● Gives the wrong idea that the volume is important
    ○ number Unit Tests > number of Integration Tests > number of E2E Tests
    ○ driven by the cost of writing/running/maintaining tests
    ● Doesn’t match all types of projects
    ● It’s a triangle, not a pyramid…
    ○ easy to understand, but lacks nuances
    What’s up with the test pyramid
    

    View Slide

14. “
    ”
    All models are approximations. Essentially, all
    models are wrong, but some are useful.
    — Georges Box
    

    View Slide

15. The Pyramid revisited
    

    View Slide

16. Noah Sussman (2017)
    

    View Slide

17. “
    ”
    I interpret the agile testing pyramid as a beginning
    to a discussion about test strategy in an agile
    team.
    — Sharon Robson
    More Agile Testing
    Lisa Crispin, Janet Gregory (2014)
    

    View Slide

18. “
    ”
    A pyramid is a self-supporting structure that
    cannot exist without each of its sides. Each has a
    different focus (or direction), but each ties into the
    whole unit.
    — Sharon Robson
    More Agile Testing
    Lisa Crispin, Janet Gregory (2014)
    

    View Slide

19. More Agile Testing
    Lisa Crispin, Janet Gregory (2014)
    

    View Slide

20. The four sides of the pyramid
    ● What are we testing?
    ● How are we testing it?
    ● Why are we testing? What are we looking to gain more confidence
    on?
    ● When are we running the tests? And Where?
    

    View Slide

21. What…?
    End to End Tests
    Integration Tests
    Unit Tests
    Static
    Analysis
    

    View Slide

22. What…?
    End to End Tests
    Integration Tests
    Unit Tests
    Static
    Analysis
    ● Levels size don’t relate to…
    ○ Number of tests
    ○ Importance
    ○ Speed
    ● Add a level for…
    ○ Business Rules
    ○ APIs / Data Format
    ○ Security
    

    View Slide

23. Why…?
    Usability
    Functionality
    Reliability
    Performance
    Code Quality
    Maintenance
    End to End Tests
    Integration Tests
    Unit Tests
    Static
    Analysis
    

    View Slide

24. Why…?
    Usability
    Functionality
    Reliability
    Performance
    Code Quality
    Maintenance
    ● And much more:
    ○ “Verify that the code works”
    ○ Make the contract explicit
    ○ Guide the development (TDD)
    ○ Prevent regression caused by
    ■ Other devs
    ■ Ourselves in the future
    ○ Ensure retrocompatibility
    

    View Slide

25. How…?
    Datadog, Maestro, …
    JUnit, Espresso, …
    JUnit, Spek, …
    Detekt
    Ktlint
    Usability
    Functionality
    Reliability
    Performance
    Code quality
    Maintenance
    

    View Slide

26. How…?
    Datadog, Maestro, …
    JUnit, Espresso, …
    JUnit, Spek, …
    Detekt
    Ktlint
    

    View Slide

27. When…?
    Nightly/pre release
    Per PR/Nightly
    Per Commit/PR
    Per commit
    Datadog, Maestro, …
    JUnit, Espresso, …
    JUnit, Spek, …
    Detekt
    Ktlint
    

    View Slide

28. When…?
    Nightly/pre release
    Per PR/Nightly
    Per Commit/PR
    Per commit
    ● Local
    ○ Reproducible environment
    setup
    ○ Easy to run locally
    ● CI
    ○ VCS provided (Github Actions,
    Gitlab, …)
    ○ Third Party Vendor (Bitrise,
    Circle CI, Travis, …)
    ○ Custom hosted (Jenkins, …)
    

    View Slide

29. About tests roles…
    

    View Slide

30. “
    ”
    Different tests also play different roles in a project.
    The traditional role of automated tests is to check
    that an application behaves correctly. However,
    tests can also play a role as "living
    documentation".
    — John Ferguson Smart
    A Test Pyramid Heresy
    John Ferguson Smart (2017)
    

    View Slide

31. Tests that Discover
    ● Example based
    ● TDD/BDD
    ● Discover the shape of a feature
    ○ API, interface, signatures, …
    

    View Slide

32. Tests that Describe
    ● Make the contract explicit
    ● Explain Business Rule / Domain Logic
    ● Living documentation
    

    View Slide

33. Tests that Demonstrate
    ● Complex or global scenarios
    ● Detail oriented
    ● Edge cases
    

    View Slide

34. About coverage…
    

    View Slide

35. Feature
    Compliance
    Input
    Coverage
    Compiler
    Static
    Analysis
    Monkey
    Tests
    Unit Tests
    Integration
    Tests
    E2E Tests
    Fuzzy Tests
    PBT
    

    View Slide

36. SUCCESS: 26/26 (100%) Tests passed
    Bernd Lörwald @bloerwald
    

    View Slide

37. “
    ”
    The more any quantitative indicator is used for
    decision-making, the more subject it will be to
    corruption pressure.
    — Donald T. Campbell
    

    View Slide

38. “
    ”
    When code coverage is at 80%,
    it doesn’t mean 80% of your code is tested;
    it means at least 20% is not tested.
    

    View Slide

39. Wrapping up
    

    View Slide

40. “
    ”
    The test pyramid is more
    what you’d call guidelines
    than actual rules.
    

    View Slide

41. ● You don’t need to have 4 levels exactly
    ● Have as many tests as you can, for each level
    ● Discuss the pyramid regularly
    ● You can (should) have conventions for tests
    Takeaway
    

    View Slide

42. “
    ”
    You need to be as confident
    in the tests you code as you are in
    the code you test.
    

    View Slide

43. Thank you!
    Any questions?
    

    View Slide

44. A 4 sided pyramid
    LEVEL 01
    LEVEL 02
    LEVEL 03
    LEVEL 04
    Lorem Ipsum
    Lorem Ipsum
    Lorem Ipsum
    Lorem Ipsum
    

    View Slide

45. A 4 sided pyramid
    LEVEL 01
    LEVEL 02
    LEVEL 03
    LEVEL 04
    Lorem Ipsum
    Lorem Ipsum
    Lorem Ipsum
    Lorem Ipsum
    

    View Slide