Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Demystifying the Test Pyramid

Demystifying the Test Pyramid

Back in 2009, Mike Cohn in his "Succeeding with Agile" book, introduced the pyramid of tests. Since then, it has been criticised (mostly for being unsuitable for many real life situations), slightly updated, but its core concept is still seen in many articles. So what value can we get out of the test pyramid, and how can you adapt it to better fit with your use cases, workflows and setup.

After explaining the original concept of the testing pyramid, we will discuss its pitfalls and caveats, then provide more depth and a new perspective on this tool while providing tips and guidelines that engineers can use to improve their test suites.

Xavier Gouchet

April 27, 2023
Tweet

More Decks by Xavier Gouchet

Other Decks in Programming

Transcript

  1. Demystifying
    the test
    pyramid

    View full-size slide

  2. Outline
    - History of the Test pyramid
    - Original pyramid
    - What is it ?
    - Flaws and evolutions
    - A word about coverage
    - How to look at the pyramid in 2023
    - A 3d Pyramid with 4 sides: What, Why, How, When·re
    - Conclusion

    View full-size slide

  3. Xavier F. Gouchet
    Senior Software Engineer
    at Datadog
    @xgouchet / @datadoghq

    View full-size slide

  4. About the Test Pyramid…

    View full-size slide



  5. An effective test automation strategy calls for
    automating tests at three different levels […] .
    If the automated test suite is growing daily and has
    run without error every night of this sprint, the
    team is probably in good shape.
    — Mike Cohn
    Succeeding with Agile
    Mike Cohn (2009)

    View full-size slide

  6. Succeeding with Agile
    Mike Cohn (2009)

    View full-size slide

  7. Unit
    Testing a “unit” independantly
    from any other code.
    + fast and reliable in CI
    + lots of mocks
    - tight coupling with
    implementation

    View full-size slide

  8. Service
    Testing two or more units
    interacting together.
    + validates integration before
    end-to-end tests
    + mock network and DB
    - non exhaustive
    - fault localization is difficult
    a.k.a. Integration Test

    View full-size slide

  9. UI
    Test the product as the end user
    would.
    + validates user experience
    - fragility and flakiness
    - hard to debug a.k.a. End to End Test

    View full-size slide

  10. ● Popularized the idea of automated tests
    ○ But like all popular things, it’s often taken at face value
    ● Gives the wrong idea that the volume is important
    ○ number Unit Tests > number of Integration Tests > number of E2E Tests
    ○ driven by the cost of writing/running/maintaining tests
    ● Doesn’t match all types of projects
    ● It’s a triangle, not a pyramid…
    ○ easy to understand, but lacks nuances
    What’s up with the test pyramid

    View full-size slide



  11. All models are approximations. Essentially, all
    models are wrong, but some are useful.
    — Georges Box

    View full-size slide

  12. The Pyramid revisited

    View full-size slide

  13. Noah Sussman (2017)

    View full-size slide



  14. I interpret the agile testing pyramid as a beginning
    to a discussion about test strategy in an agile
    team.
    — Sharon Robson
    More Agile Testing
    Lisa Crispin, Janet Gregory (2014)

    View full-size slide



  15. A pyramid is a self-supporting structure that
    cannot exist without each of its sides. Each has a
    different focus (or direction), but each ties into the
    whole unit.
    — Sharon Robson
    More Agile Testing
    Lisa Crispin, Janet Gregory (2014)

    View full-size slide

  16. More Agile Testing
    Lisa Crispin, Janet Gregory (2014)

    View full-size slide

  17. The four sides of the pyramid
    ● What are we testing?
    ● How are we testing it?
    ● Why are we testing? What are we looking to gain more confidence
    on?
    ● When are we running the tests? And Where?

    View full-size slide

  18. What…?
    End to End Tests
    Integration Tests
    Unit Tests
    Static
    Analysis

    View full-size slide

  19. What…?
    End to End Tests
    Integration Tests
    Unit Tests
    Static
    Analysis
    ● Levels size don’t relate to…
    ○ Number of tests
    ○ Importance
    ○ Speed
    ● Add a level for…
    ○ Business Rules
    ○ APIs / Data Format
    ○ Security

    View full-size slide

  20. Why…?
    Usability
    Functionality
    Reliability
    Performance
    Code Quality
    Maintenance
    End to End Tests
    Integration Tests
    Unit Tests
    Static
    Analysis

    View full-size slide

  21. Why…?
    Usability
    Functionality
    Reliability
    Performance
    Code Quality
    Maintenance
    ● And much more:
    ○ “Verify that the code works”
    ○ Make the contract explicit
    ○ Guide the development (TDD)
    ○ Prevent regression caused by
    ■ Other devs
    ■ Ourselves in the future
    ○ Ensure retrocompatibility

    View full-size slide

  22. How…?
    Datadog, Maestro, …
    JUnit, Espresso, …
    JUnit, Spek, …
    Detekt
    Ktlint
    Usability
    Functionality
    Reliability
    Performance
    Code quality
    Maintenance

    View full-size slide

  23. How…?
    Datadog, Maestro, …
    JUnit, Espresso, …
    JUnit, Spek, …
    Detekt
    Ktlint

    View full-size slide

  24. When…?
    Nightly/pre release
    Per PR/Nightly
    Per Commit/PR
    Per commit
    Datadog, Maestro, …
    JUnit, Espresso, …
    JUnit, Spek, …
    Detekt
    Ktlint

    View full-size slide

  25. When…?
    Nightly/pre release
    Per PR/Nightly
    Per Commit/PR
    Per commit
    ● Local
    ○ Reproducible environment
    setup
    ○ Easy to run locally
    ● CI
    ○ VCS provided (Github Actions,
    Gitlab, …)
    ○ Third Party Vendor (Bitrise,
    Circle CI, Travis, …)
    ○ Custom hosted (Jenkins, …)

    View full-size slide

  26. About tests roles…

    View full-size slide



  27. Different tests also play different roles in a project.
    The traditional role of automated tests is to check
    that an application behaves correctly. However,
    tests can also play a role as "living
    documentation".
    — John Ferguson Smart
    A Test Pyramid Heresy
    John Ferguson Smart (2017)

    View full-size slide

  28. Tests that Discover
    ● Example based
    ● TDD/BDD
    ● Discover the shape of a feature
    ○ API, interface, signatures, …

    View full-size slide

  29. Tests that Describe
    ● Make the contract explicit
    ● Explain Business Rule / Domain Logic
    ● Living documentation

    View full-size slide

  30. Tests that Demonstrate
    ● Complex or global scenarios
    ● Detail oriented
    ● Edge cases

    View full-size slide

  31. About coverage…

    View full-size slide

  32. Feature
    Compliance
    Input
    Coverage
    Compiler
    Static
    Analysis
    Monkey
    Tests
    Unit Tests
    Integration
    Tests
    E2E Tests
    Fuzzy Tests
    PBT

    View full-size slide

  33. SUCCESS: 26/26 (100%) Tests passed
    Bernd Lörwald @bloerwald

    View full-size slide



  34. The more any quantitative indicator is used for
    decision-making, the more subject it will be to
    corruption pressure.
    — Donald T. Campbell

    View full-size slide



  35. When code coverage is at 80%,
    it doesn’t mean 80% of your code is tested;
    it means at least 20% is not tested.

    View full-size slide



  36. The test pyramid is more
    what you’d call guidelines
    than actual rules.

    View full-size slide

  37. ● You don’t need to have 4 levels exactly
    ● Have as many tests as you can, for each level
    ● Discuss the pyramid regularly
    ● You can (should) have conventions for tests
    Takeaway

    View full-size slide



  38. You need to be as confident
    in the tests you code as you are in
    the code you test.

    View full-size slide

  39. Thank you!
    Any questions?

    View full-size slide

  40. A 4 sided pyramid
    LEVEL 01
    LEVEL 02
    LEVEL 03
    LEVEL 04
    Lorem Ipsum
    Lorem Ipsum
    Lorem Ipsum
    Lorem Ipsum

    View full-size slide

  41. A 4 sided pyramid
    LEVEL 01
    LEVEL 02
    LEVEL 03
    LEVEL 04
    Lorem Ipsum
    Lorem Ipsum
    Lorem Ipsum
    Lorem Ipsum

    View full-size slide