Tweet
Tweet

Slide 1

Slide 1 text

Demystifying the test pyramid

Slide 2

Slide 2 text

Outline - History of the Test pyramid - Original pyramid - What is it ? - Flaws and evolutions - A word about coverage - How to look at the pyramid in 2023 - A 3d Pyramid with 4 sides: What, Why, How, When·re - Conclusion

Slide 3

Slide 3 text

Xavier F. Gouchet Senior Software Engineer at Datadog @xgouchet / @datadoghq

Slide 4

Slide 4 text

4

Slide 5

Slide 5 text

No content

Slide 6

Slide 6 text

About the Test Pyramid…

Slide 7

Slide 7 text

“ ” An effective test automation strategy calls for automating tests at three different levels […] . If the automated test suite is growing daily and has run without error every night of this sprint, the team is probably in good shape. — Mike Cohn Succeeding with Agile Mike Cohn (2009)

Slide 8

Slide 8 text

Succeeding with Agile Mike Cohn (2009)

Slide 9

Slide 9 text

Unit Testing a “unit” independantly from any other code. + fast and reliable in CI + lots of mocks - tight coupling with implementation

Slide 10

Slide 10 text

Service Testing two or more units interacting together. + validates integration before end-to-end tests + mock network and DB - non exhaustive - fault localization is difficult a.k.a. Integration Test

Slide 11

Slide 11 text

UI Test the product as the end user would. + validates user experience - fragility and flakiness - hard to debug a.k.a. End to End Test

Slide 12

Slide 12 text

No content

Slide 13

Slide 13 text

● Popularized the idea of automated tests ○ But like all popular things, it’s often taken at face value ● Gives the wrong idea that the volume is important ○ number Unit Tests > number of Integration Tests > number of E2E Tests ○ driven by the cost of writing/running/maintaining tests ● Doesn’t match all types of projects ● It’s a triangle, not a pyramid… ○ easy to understand, but lacks nuances What’s up with the test pyramid

Slide 14

Slide 14 text

“ ” All models are approximations. Essentially, all models are wrong, but some are useful. — Georges Box

Slide 15

Slide 15 text

The Pyramid revisited

Slide 16

Slide 16 text

Noah Sussman (2017)

Slide 17

Slide 17 text

“ ” I interpret the agile testing pyramid as a beginning to a discussion about test strategy in an agile team. — Sharon Robson More Agile Testing Lisa Crispin, Janet Gregory (2014)

Slide 18

Slide 18 text

“ ” A pyramid is a self-supporting structure that cannot exist without each of its sides. Each has a different focus (or direction), but each ties into the whole unit. — Sharon Robson More Agile Testing Lisa Crispin, Janet Gregory (2014)

Slide 19

Slide 19 text

More Agile Testing Lisa Crispin, Janet Gregory (2014)

Slide 20

Slide 20 text

The four sides of the pyramid ● What are we testing? ● How are we testing it? ● Why are we testing? What are we looking to gain more confidence on? ● When are we running the tests? And Where?

Slide 21

Slide 21 text

What…? End to End Tests Integration Tests Unit Tests Static Analysis

Slide 22

Slide 22 text

What…? End to End Tests Integration Tests Unit Tests Static Analysis ● Levels size don’t relate to… ○ Number of tests ○ Importance ○ Speed ● Add a level for… ○ Business Rules ○ APIs / Data Format ○ Security

Slide 23

Slide 23 text

Why…? Usability Functionality Reliability Performance Code Quality Maintenance End to End Tests Integration Tests Unit Tests Static Analysis

Slide 24

Slide 24 text

Why…? Usability Functionality Reliability Performance Code Quality Maintenance ● And much more: ○ “Verify that the code works” ○ Make the contract explicit ○ Guide the development (TDD) ○ Prevent regression caused by ■ Other devs ■ Ourselves in the future ○ Ensure retrocompatibility

Slide 25

Slide 25 text

How…? Datadog, Maestro, … JUnit, Espresso, … JUnit, Spek, … Detekt Ktlint Usability Functionality Reliability Performance Code quality Maintenance

Slide 26

Slide 26 text

How…? Datadog, Maestro, … JUnit, Espresso, … JUnit, Spek, … Detekt Ktlint

Slide 27

Slide 27 text

When…? Nightly/pre release Per PR/Nightly Per Commit/PR Per commit Datadog, Maestro, … JUnit, Espresso, … JUnit, Spek, … Detekt Ktlint

Slide 28

Slide 28 text

When…? Nightly/pre release Per PR/Nightly Per Commit/PR Per commit ● Local ○ Reproducible environment setup ○ Easy to run locally ● CI ○ VCS provided (Github Actions, Gitlab, …) ○ Third Party Vendor (Bitrise, Circle CI, Travis, …) ○ Custom hosted (Jenkins, …)

Slide 29

Slide 29 text

About tests roles…

Slide 30

Slide 30 text

“ ” Different tests also play different roles in a project. The traditional role of automated tests is to check that an application behaves correctly. However, tests can also play a role as "living documentation". — John Ferguson Smart A Test Pyramid Heresy John Ferguson Smart (2017)

Slide 31

Slide 31 text

Tests that Discover ● Example based ● TDD/BDD ● Discover the shape of a feature ○ API, interface, signatures, …

Slide 32

Slide 32 text

Tests that Describe ● Make the contract explicit ● Explain Business Rule / Domain Logic ● Living documentation

Slide 33

Slide 33 text

Tests that Demonstrate ● Complex or global scenarios ● Detail oriented ● Edge cases

Slide 34

Slide 34 text

About coverage…

Slide 35

Slide 35 text

Feature Compliance Input Coverage Compiler Static Analysis Monkey Tests Unit Tests Integration Tests E2E Tests Fuzzy Tests PBT

Slide 36

Slide 36 text

SUCCESS: 26/26 (100%) Tests passed Bernd Lörwald @bloerwald

Slide 37

Slide 37 text

“ ” The more any quantitative indicator is used for decision-making, the more subject it will be to corruption pressure. — Donald T. Campbell

Slide 38

Slide 38 text

“ ” When code coverage is at 80%, it doesn’t mean 80% of your code is tested; it means at least 20% is not tested.

Slide 39

Slide 39 text

Wrapping up

Slide 40

Slide 40 text

“ ” The test pyramid is more what you’d call guidelines than actual rules.

Slide 41

Slide 41 text

● You don’t need to have 4 levels exactly ● Have as many tests as you can, for each level ● Discuss the pyramid regularly ● You can (should) have conventions for tests Takeaway

Slide 42

Slide 42 text

“ ” You need to be as confident in the tests you code as you are in the code you test.

Slide 43

Slide 43 text

Thank you! Any questions?

Slide 44

Slide 44 text

A 4 sided pyramid LEVEL 01 LEVEL 02 LEVEL 03 LEVEL 04 Lorem Ipsum Lorem Ipsum Lorem Ipsum Lorem Ipsum

Slide 45

Slide 45 text

A 4 sided pyramid LEVEL 01 LEVEL 02 LEVEL 03 LEVEL 04 Lorem Ipsum Lorem Ipsum Lorem Ipsum Lorem Ipsum