Slide 1
Slide 1 text
100%
Unit Test
Coverage
and beyond
Marc Sallin
27 August 2024
Slide 2
Slide 2 text
Speaker
Marc Sallin
Solution Architect
[email protected]
27. August 2024
Swiss Post – 100% Unit Test Coverage – Marc Sallin – C1 Public 2
Swiss Post Ltd
I/T, IT20.2
Wankdorfallee 4, 3014 Bern
swisspost.ch
Slide 3
Slide 3 text
Swiss Post Parcel Sorting
Daillens
Vétroz
Härkingen
PZ & BBZ
Urdorf
Frauenfeld
Untervaz
Cadenazzo
Pratteln
Ostermundigen
Rümlang
Wallisellen
Buchs
1.5 Years
3 Centers
6 Million Parcels
27. August 2024
Swiss Post – 100% Unit Test Coverage – Marc Sallin – C1 Public 3
Slide 4
Slide 4 text
Some Numbers
~5k
Deployments
0
Regressions
1 / 1,5h
Change Failure & TTR
~1.5d
Commit Lead Time
18
Dev & Arch
99.9%
Unit Test Coverage
27. August 2024
Swiss Post – 100% Unit Test Coverage – Marc Sallin – C1 Public 4
Slide 5
Slide 5 text
27. August 2024
Swiss Post – 100% Unit Test Coverage – Marc Sallin – C1 Public 5
“80% Coverage is enough.”
Too much people in our industry..
Slide 6
Slide 6 text
27. August 2024
Swiss Post – 100% Unit Test Coverage – Marc Sallin – C1 Public 6
Dear Software
Engineers, we need to
talk...
Slide 7
Slide 7 text
Software Engineering
“The systematic application of scientific and technological knowledge,
methods, and experience to the design, implementation, testing, and
documentation of software.”
IEEE Standard Glossary of Software Engineering Terminology
27. August 2024
Swiss Post – 100% Unit Test Coverage – Marc Sallin – C1 Public 7
Slide 8
Slide 8 text
Fallacy 1: Nirvana fallacy or the danger of false dichotomies
Premise: Tests do not proof the absence of defects
It follows: Even 100% test coverage doesn’t proof the absence of defects
Conclusion: it follows that 100% isn’t worth it
27. August 2024
Swiss Post – 100% Unit Test Coverage – Marc Sallin – C1 Public 8
Slide 9
Slide 9 text
Fallacy 2: There is a threshold
27. August 2024
Swiss Post – 100% Unit Test Coverage – Marc Sallin – C1 Public 9
𝑓 𝑥 = 2(1 − 𝑒−2𝑥)
Slide 10
Slide 10 text
Fallacy 3: BUT not all areas are important
27. August 2024
Swiss Post – 100% Unit Test Coverage – Marc Sallin – C1 Public 10
Premise: Not all areas are important to test
It follows: We not need to test all areas
Conclusion: We only cover 80%
To make the conclusion’s true, the following assumptions have to be true:
Assumption I: Important areas are objectively definable
Assumption II: The ration of important vs. unimportant areas stays the same
Slide 11
Slide 11 text
Fallacy 4: We just need End-2-End
27. August 2024
Swiss Post – 100% Unit Test Coverage – Marc Sallin – C1 Public 11
Slide 12
Slide 12 text
Fallacy 5: Tests are to expensive and we have no time
27. August 2024
Swiss Post – 100% Unit Test Coverage – Marc Sallin – C1 Public 12
https://martinfowler.com/bliki/DesignStaminaHypothesis.html (26.08.2024)
Slide 13
Slide 13 text
Fallacy 6: Test shouldn’t be written by devs
27. August 2024
Swiss Post – 100% Unit Test Coverage – Marc Sallin – C1 Public 13
https://builtin.com/software-engineering-perspectives/v-model (28.08.2024)
Slide 14
Slide 14 text
Beyond 100%...
But you accepted that tests do not
guarantee functional correctness! Can
we do better?
2. September 2024
Die Schweizerische Post – Titel der Präsentation – Vorname Name – C1 Öffentlich | C2 Intern | C3
Eingeschränkt | C4 Vertraulich (Fusszeile ändern über «Einfügen > Kopf- und Fusszeile»)
14
Slide 15
Slide 15 text
How many Tests?
27. August 2024
Swiss Post – 100% Unit Test Coverage – Marc Sallin – C1 Public 15
Slide 16
Slide 16 text
Still missing...
Startup Flows
Contracts Test Quality
Slide 17
Slide 17 text
Startup – IoC Container Validation
27. August 2024
Swiss Post – 100% Unit Test Coverage – Marc Sallin – C1 Public 17
Slide 18
Slide 18 text
Flows – Integration Tests
27. August 2024
Swiss Post – 100% Unit Test Coverage – Marc Sallin – C1 Public 18
Slide 19
Slide 19 text
Test Quality - Mutation Testing
27. August 2024
Swiss Post – 100% Unit Test Coverage – Marc Sallin – C1 Public 19
Slide 20
Slide 20 text
Contracts – Contract Testing
27. August 2024
Swiss Post – 100% Unit Test Coverage – Marc Sallin – C1 Public 20
Slide 21
Slide 21 text
Exclude from Coverage!
27. August 2024
Swiss Post – 100% Unit Test Coverage – Marc Sallin – C1 Public 21
Slide 22
Slide 22 text
Conclusion
100% coverage is not the
end. It is not even the
beginning of the end. But it
is, perhaps, the end of the
beginning.
Slide 23
Slide 23 text
Thank You!
27. August 2024
Swiss Post – 100% Unit Test Coverage – Marc Sallin – C1 Public 23
Slide 24
Slide 24 text
Danke, Merci,
Grazie, Thank you
27. August 2024
Swiss Post – 100% Unit Test Coverage – Marc Sallin – C1 Public 24