A presentation by @stuherbert
for @GanbaroDigital
Go 100% Code
Coverage
Or Go Home
Slide 2
Slide 2 text
@GanbaroDigital
It Started
With A Tweet …
Slide 3
Slide 3 text
@GanbaroDigital
It Started With A Tweet
Slide 4
Slide 4 text
@GanbaroDigital
“With 100% code coverage,
you could still ship bugs.
Without 100% code coverage,
you’re definitely shipping bugs.
Slide 5
Slide 5 text
@GanbaroDigital
This factual statement
proved controversial.
Slide 6
Slide 6 text
@GanbaroDigital
“100% code coverage”
is a trigger for most commentators,
not a target.
Slide 7
Slide 7 text
@GanbaroDigital
https://flic.kr/p/6My3Ln
Makes Me Blue
Slide 8
Slide 8 text
@GanbaroDigital
??
??
Do you target
100% code coverage?
Slide 9
Slide 9 text
@GanbaroDigital
??
??
Why not?
Slide 10
Slide 10 text
@GanbaroDigital
In This Talk
1. What is Code Coverage?
2. What does it tell us?
3. What can’t it tell us?
4. How can we mitigate that?
5. Why does Code Coverage matter?
Slide 11
Slide 11 text
@GanbaroDigital
https://flic.kr/p/pGqBUu
I’m Here To Pick A Fight
Slide 12
Slide 12 text
@GanbaroDigital
What Is
Code Coverage?
Slide 13
Slide 13 text
@GanbaroDigital
Slide 14
Slide 14 text
@GanbaroDigital
Slide 15
Slide 15 text
@GanbaroDigital
Code coverage measures:
the lines of code
in your unit under test
that were executed
when you ran your tests
Slide 16
Slide 16 text
@GanbaroDigital
What Does
It Tell Us?
Slide 17
Slide 17 text
@GanbaroDigital
Code coverage measures:
the lines of code
in your unit under test
that were executed
when you ran your tests
Slide 18
Slide 18 text
@GanbaroDigital
Code coverage measures:
the lines of code
in your unit under test
that were executed
when you ran your tests
Slide 19
Slide 19 text
@GanbaroDigital
“
We can see exactly
what a test is proving.
Slide 20
Slide 20 text
@GanbaroDigital
Use code coverage annotations
to tell PHPUnit
what part(s) of your unit under test
you are actually testing.
Slide 21
Slide 21 text
@GanbaroDigital
Slide 22
Slide 22 text
@GanbaroDigital
Code coverage measures:
the lines of code
in your unit under test
that were executed
when you ran your tests
Slide 23
Slide 23 text
@GanbaroDigital
“
We can see which code
is currently untested.
Slide 24
Slide 24 text
@GanbaroDigital
Slide 25
Slide 25 text
@GanbaroDigital
Slide 26
Slide 26 text
@GanbaroDigital
Slide 27
Slide 27 text
@GanbaroDigital
“
Untested code
is potentially
unexecuted code.
Slide 28
Slide 28 text
@GanbaroDigital
Code coverage isn’t just
for the original author.
Slide 29
Slide 29 text
@GanbaroDigital
The next maintainer
doesn’t have to start
from a position of faith alone.
Slide 30
Slide 30 text
@GanbaroDigital
https://flic.kr/p/9gh9EA
Evidence
Slide 31
Slide 31 text
@GanbaroDigital
Just because we live in times
where science and engineering
are out of vogue,
doesn’t mean we have to
pander to the forces
of ignorance.
Slide 32
Slide 32 text
@GanbaroDigital
“
Guesswork is for amateurs.
Slide 33
Slide 33 text
@GanbaroDigital
What Can’t It
Tell Us?
Slide 34
Slide 34 text
@GanbaroDigital
Current code coverage tools
cannot prove
that every code path
has been tested.
Slide 35
Slide 35 text
@GanbaroDigital
Slide 36
Slide 36 text
@GanbaroDigital
Code coverage tools
cannot prove
that we’re testing
the right thing.
Slide 37
Slide 37 text
@GanbaroDigital
Code Coverage Does
• Prove that code executes
• Prove that code is tested
Slide 38
Slide 38 text
@GanbaroDigital
Code Coverage Does Not
• Prove that all possible paths have tests
• Prove that we’re testing the right things
Slide 39
Slide 39 text
@GanbaroDigital
How Can We
Mitigate That?
Slide 40
Slide 40 text
@GanbaroDigital
This isn’t a talk
about how to write tests.
Slide 41
Slide 41 text
@GanbaroDigital
But the arguments against
100% code coverage
are usually rooted
in poor test design.
Slide 42
Slide 42 text
@GanbaroDigital
(or poor code design)
Slide 43
Slide 43 text
@GanbaroDigital
Code Coverage Does Not
• Prove that all possible paths have tests
• Prove that we’re testing the right things
Slide 44
Slide 44 text
@GanbaroDigital
“With 100% code coverage,
you could still ship bugs.
Without 100% code coverage,
you’re definitely shipping bugs.
Slide 45
Slide 45 text
@GanbaroDigital
https://flic.kr/p/uz8YAe
Aim For The
True Target
Slide 46
Slide 46 text
@GanbaroDigital
“
100% code coverage
is the wrong target.
Slide 47
Slide 47 text
@GanbaroDigital
Don’t think in terms
of lines of code.
Think in terms of behaviours
and responsibilities.
Slide 48
Slide 48 text
@GanbaroDigital
Test your requirements
and your robustness.
Slide 49
Slide 49 text
@GanbaroDigital
“With 100% code coverage,
you could still ship bugs.
Without 100% code coverage,
you’re definitely shipping bugs.
Slide 50
Slide 50 text
@GanbaroDigital
“
With 100% behaviour coverage,
you’ll rarely ship bugs.
Without 100% behaviour coverage,
you’re definitely shipping bugs.
Slide 51
Slide 51 text
@GanbaroDigital
Slide 52
Slide 52 text
@GanbaroDigital
Code Coverage Does Not
• Prove that all possible paths have tests
• Prove that we’re testing the right things
Slide 53
Slide 53 text
@GanbaroDigital
One day, machine learning
will be able to solve that
for us.
Slide 54
Slide 54 text
@GanbaroDigital
Until then,
old-school code reviews
can help a lot.
Slide 55
Slide 55 text
@GanbaroDigital
Outcome-oriented testing,
driven by ISO 25000 SQuaRE,
can also help.
Slide 56
Slide 56 text
@GanbaroDigital
Even a bad test
still proves that
the code executes.
Slide 57
Slide 57 text
@GanbaroDigital
Summary
1. Code coverage proves code runs.
2. Test behaviours, not code.
3. Use humans to check what you’re testing.
4. Use standards to guide your tests.
Slide 58
Slide 58 text
@GanbaroDigital
Why Does
Code Coverage Matter?
Slide 59
Slide 59 text
@GanbaroDigital
https://flic.kr/p/6My3Ln
Makes Me Blue
Slide 60
Slide 60 text
@GanbaroDigital
https://flic.kr/p/pGqBUu
I’m Here To Pick A Fight
Slide 61
Slide 61 text
@GanbaroDigital
Let’s say your app
is 10,000 lines of code.
Your code coverage target
is 60%.
Slide 62
Slide 62 text
@GanbaroDigital
40%
60%
Slide 63
Slide 63 text
@GanbaroDigital
??
??
How many bugs
have your tests found
in the 60% of the code
that you have tests for?