Go 100% Code Coverage or Go Home

A presentation by @stuherbert  for @GanbaroDigital Go 100% Code Coverage
Or Go Home

@GanbaroDigital It Started With A Tweet …

@GanbaroDigital It Started With A Tweet

@GanbaroDigital “With 100% code coverage, you could still ship bugs.
Without 100% code coverage, you’re deﬁnitely shipping bugs.

@GanbaroDigital This factual statement proved controversial.

@GanbaroDigital “100% code coverage” is a trigger for most commentators,
not a target.

@GanbaroDigital https://ﬂic.kr/p/6My3Ln Makes Me Blue

@GanbaroDigital ?? ?? Do you target 100% code coverage?

@GanbaroDigital ?? ?? Why not?

@GanbaroDigital In This Talk 1. What is Code Coverage? 2.
What does it tell us? 3. What can’t it tell us? 4. How can we mitigate that? 5. Why does Code Coverage matter?

@GanbaroDigital https://ﬂic.kr/p/pGqBUu I’m Here To Pick A Fight

@GanbaroDigital What Is Code Coverage?

@GanbaroDigital

@GanbaroDigital Code coverage measures: the lines of code in your
unit under test that were executed when you ran your tests

@GanbaroDigital What Does It Tell Us?

@GanbaroDigital “ We can see exactly what a test is
proving.

@GanbaroDigital Use code coverage annotations to tell PHPUnit what part(s)
of your unit under test you are actually testing.

@GanbaroDigital

@GanbaroDigital “ We can see which code is currently untested.

@GanbaroDigital

@GanbaroDigital “ Untested code is potentially unexecuted code.

@GanbaroDigital Code coverage isn’t just for the original author.

@GanbaroDigital The next maintainer doesn’t have to start from a
position of faith alone.

@GanbaroDigital https://ﬂic.kr/p/9gh9EA Evidence

@GanbaroDigital Just because we live in times where science and
engineering are out of vogue, doesn’t mean we have to pander to the forces of ignorance.

@GanbaroDigital “ Guesswork is for amateurs.

@GanbaroDigital What Can’t It Tell Us?

@GanbaroDigital Current code coverage tools cannot prove that every code
path has been tested.

@GanbaroDigital

@GanbaroDigital Code coverage tools cannot prove that we’re testing the
right thing.

@GanbaroDigital Code Coverage Does • Prove that code executes •
Prove that code is tested

@GanbaroDigital Code Coverage Does Not • Prove that all possible
paths have tests • Prove that we’re testing the right things

@GanbaroDigital How Can We Mitigate That?

@GanbaroDigital This isn’t a talk about how to write tests.

@GanbaroDigital But the arguments against 100% code coverage are usually
rooted in poor test design.

@GanbaroDigital (or poor code design)

@GanbaroDigital https://ﬂic.kr/p/uz8YAe Aim For The True Target

@GanbaroDigital “ 100% code coverage is the wrong target.

@GanbaroDigital Don’t think in terms of lines of code. Think
in terms of behaviours and responsibilities.

@GanbaroDigital Test your requirements and your robustness.

@GanbaroDigital “ With 100% behaviour coverage, you’ll rarely ship bugs.
Without 100% behaviour coverage, you’re deﬁnitely shipping bugs.

@GanbaroDigital

@GanbaroDigital One day, machine learning will be able to solve
that for us.

@GanbaroDigital Until then, old-school code reviews can help a lot.

@GanbaroDigital Outcome-oriented testing, driven by ISO 25000 SQuaRE, can also
help.

@GanbaroDigital Even a bad test still proves that the code
executes.

@GanbaroDigital Summary 1. Code coverage proves code runs. 2. Test
behaviours, not code. 3. Use humans to check what you’re testing. 4. Use standards to guide your tests.

@GanbaroDigital Why Does Code Coverage Matter?

@GanbaroDigital https://ﬂic.kr/p/6My3Ln Makes Me Blue

@GanbaroDigital https://ﬂic.kr/p/pGqBUu I’m Here To Pick A Fight

@GanbaroDigital Let’s say your app is 10,000 lines of code.
Your code coverage target is 60%.

@GanbaroDigital 40% 60%

@GanbaroDigital ?? ?? How many bugs have your tests found
in the 60% of the code that you have tests for?

@GanbaroDigital Defects Found 0 30 60 90 120 Covered Code
Uncovered Code

@GanbaroDigital ?? ?? Why would the defect rate be any
different in the remaining 40% of uncovered code?

@GanbaroDigital Defects Found 0 30 60 90 120 Covered Code
Uncovered Code

@GanbaroDigital Defect rate alone doesn’t tell the whole story. Not
all defects cost the same when things go wrong.

@GanbaroDigital ?? ?? Have you heard of “happy path” programming?

@GanbaroDigital

@GanbaroDigital The “happy path” only deals with correctness under perfect
conditions.

@GanbaroDigital This is an app without robustness.

@GanbaroDigital “ Correctness bugs are logic ﬂaws that produce unacceptable
results.

@GanbaroDigital The happy path works when correctness bugs have been
found and resolved.

@GanbaroDigital “ Robustness code deals with all the things that
shouldn’t happen, but inevitably do.

@GanbaroDigital

@GanbaroDigital x

@GanbaroDigital Robustness turns working code into real-world code.

@GanbaroDigital Things People Don’t Test • “Trivial” methods • Robustness
code

@GanbaroDigital “ Even a trivial method can have a typo
in a variable name.

@GanbaroDigital Things People Don’t Test • “Trivial” methods • Robustness
code

@GanbaroDigital ?? ?? What happens when your untested robustness code
has bugs?

@GanbaroDigital “ Robustness code is the absolute worst place to
ship bugs.

@GanbaroDigital You’re already handling a failure situation.

@GanbaroDigital Badly-handled failures create the worst kinds of mess to
clean up.

@GanbaroDigital https://ﬂic.kr/p/83Z8Kp Angry Customers

@GanbaroDigital You hurt your organisation when you ship

@GanbaroDigital All failures are exponentially more expensive to put right.

@GanbaroDigital The rule of thumb is that it costs 10x
more to ﬁx a defect at the next stage.

@GanbaroDigital Design Dev Test Live

@GanbaroDigital Even if we assume only a 3x cost, the
numbers are still ugly.

@GanbaroDigital Design Dev Test Live

@GanbaroDigital ?? ?? Why does it cost so much?

@GanbaroDigital • Redesign footprint • Putting things right for the
customer • Opportunity costs

@GanbaroDigital ?? ?? Where does the money come from to
cover these costs?

@GanbaroDigital

@GanbaroDigital 10x programmers are real. They’re just folks who ship
quickly whilst consistently reduce and avoid costly rework.

@GanbaroDigital 0 75 150 225 300 Design Dev Test Live

@GanbaroDigital “ Excellence is a habit.

@GanbaroDigital “ Right ﬁrst time, not rewrite next time.

@GanbaroDigital “ Fit for purpose, not tick in a box.

@GanbaroDigital “ Work, not rework.

@GanbaroDigital https://ﬂic.kr/p/6vLhpM Meta-Ignorance

@GanbaroDigital https://ﬂic.kr/p/cwK7UN We’re Professionals, Damnit!

@GanbaroDigital “Go 100% code coverage, or go home and leave
the job to the professionals.

Thank You Any Questions? A presentation by @stuherbert  for @GanbaroDigital

Go 100% Code Coverage or Go Home

Go 100% Code Coverage or Go Home

More Decks by Stuart Herbert

Other Decks in Programming

Featured

Transcript