Slide 1

Slide 1 text

A presentation by @stuherbert
 for @GanbaroDigital Go 100% Code Coverage Or Go Home

Slide 2

Slide 2 text

@GanbaroDigital It Started With A Tweet …

Slide 3

Slide 3 text

@GanbaroDigital It Started With A Tweet

Slide 4

Slide 4 text

@GanbaroDigital “With 100% code coverage, you could still ship bugs. Without 100% code coverage, you’re definitely shipping bugs.

Slide 5

Slide 5 text

@GanbaroDigital This factual statement proved controversial.

Slide 6

Slide 6 text

@GanbaroDigital “100% code coverage” is a trigger for most commentators, not a target.

Slide 7

Slide 7 text

@GanbaroDigital https://flic.kr/p/6My3Ln Makes Me Blue

Slide 8

Slide 8 text

@GanbaroDigital ?? ?? Do you target 100% code coverage?

Slide 9

Slide 9 text

@GanbaroDigital ?? ?? Why not?

Slide 10

Slide 10 text

@GanbaroDigital In This Talk 1. What is Code Coverage? 2. What does it tell us? 3. What can’t it tell us? 4. How can we mitigate that? 5. Why does Code Coverage matter?

Slide 11

Slide 11 text

@GanbaroDigital https://flic.kr/p/pGqBUu I’m Here To Pick A Fight

Slide 12

Slide 12 text

@GanbaroDigital What Is Code Coverage?

Slide 13

Slide 13 text

@GanbaroDigital

Slide 14

Slide 14 text

@GanbaroDigital

Slide 15

Slide 15 text

@GanbaroDigital Code coverage measures: the lines of code in your unit under test that were executed when you ran your tests

Slide 16

Slide 16 text

@GanbaroDigital What Does It Tell Us?

Slide 17

Slide 17 text

@GanbaroDigital Code coverage measures: the lines of code in your unit under test that were executed when you ran your tests

Slide 18

Slide 18 text

@GanbaroDigital Code coverage measures: the lines of code in your unit under test that were executed when you ran your tests

Slide 19

Slide 19 text

@GanbaroDigital “ We can see exactly what a test is proving.

Slide 20

Slide 20 text

@GanbaroDigital Use code coverage annotations to tell PHPUnit what part(s) of your unit under test you are actually testing.

Slide 21

Slide 21 text

@GanbaroDigital

Slide 22

Slide 22 text

@GanbaroDigital Code coverage measures: the lines of code in your unit under test that were executed when you ran your tests

Slide 23

Slide 23 text

@GanbaroDigital “ We can see which code is currently untested.

Slide 24

Slide 24 text

@GanbaroDigital

Slide 25

Slide 25 text

@GanbaroDigital

Slide 26

Slide 26 text

@GanbaroDigital

Slide 27

Slide 27 text

@GanbaroDigital “ Untested code is potentially unexecuted code.

Slide 28

Slide 28 text

@GanbaroDigital Code coverage isn’t just for the original author.

Slide 29

Slide 29 text

@GanbaroDigital The next maintainer doesn’t have to start from a position of faith alone.

Slide 30

Slide 30 text

@GanbaroDigital https://flic.kr/p/9gh9EA Evidence

Slide 31

Slide 31 text

@GanbaroDigital Just because we live in times where science and engineering are out of vogue, doesn’t mean we have to pander to the forces of ignorance.

Slide 32

Slide 32 text

@GanbaroDigital “ Guesswork is for amateurs.

Slide 33

Slide 33 text

@GanbaroDigital What Can’t It Tell Us?

Slide 34

Slide 34 text

@GanbaroDigital Current code coverage tools cannot prove that every code path has been tested.

Slide 35

Slide 35 text

@GanbaroDigital

Slide 36

Slide 36 text

@GanbaroDigital Code coverage tools cannot prove that we’re testing the right thing.

Slide 37

Slide 37 text

@GanbaroDigital Code Coverage Does • Prove that code executes • Prove that code is tested

Slide 38

Slide 38 text

@GanbaroDigital Code Coverage Does Not • Prove that all possible paths have tests • Prove that we’re testing the right things

Slide 39

Slide 39 text

@GanbaroDigital How Can We Mitigate That?

Slide 40

Slide 40 text

@GanbaroDigital This isn’t a talk about how to write tests.

Slide 41

Slide 41 text

@GanbaroDigital But the arguments against 100% code coverage are usually rooted in poor test design.

Slide 42

Slide 42 text

@GanbaroDigital (or poor code design)

Slide 43

Slide 43 text

@GanbaroDigital Code Coverage Does Not • Prove that all possible paths have tests • Prove that we’re testing the right things

Slide 44

Slide 44 text

@GanbaroDigital “With 100% code coverage, you could still ship bugs. Without 100% code coverage, you’re definitely shipping bugs.

Slide 45

Slide 45 text

@GanbaroDigital https://flic.kr/p/uz8YAe Aim For The True Target

Slide 46

Slide 46 text

@GanbaroDigital “ 100% code coverage is the wrong target.

Slide 47

Slide 47 text

@GanbaroDigital Don’t think in terms of lines of code. Think in terms of behaviours and responsibilities.

Slide 48

Slide 48 text

@GanbaroDigital Test your requirements and your robustness.

Slide 49

Slide 49 text

@GanbaroDigital “With 100% code coverage, you could still ship bugs. Without 100% code coverage, you’re definitely shipping bugs.

Slide 50

Slide 50 text

@GanbaroDigital “ With 100% behaviour coverage, you’ll rarely ship bugs. Without 100% behaviour coverage, you’re definitely shipping bugs.

Slide 51

Slide 51 text

@GanbaroDigital

Slide 52

Slide 52 text

@GanbaroDigital Code Coverage Does Not • Prove that all possible paths have tests • Prove that we’re testing the right things

Slide 53

Slide 53 text

@GanbaroDigital One day, machine learning will be able to solve that for us.

Slide 54

Slide 54 text

@GanbaroDigital Until then, old-school code reviews can help a lot.

Slide 55

Slide 55 text

@GanbaroDigital Outcome-oriented testing, driven by ISO 25000 SQuaRE, can also help.

Slide 56

Slide 56 text

@GanbaroDigital Even a bad test still proves that the code executes.

Slide 57

Slide 57 text

@GanbaroDigital Summary 1. Code coverage proves code runs. 2. Test behaviours, not code. 3. Use humans to check what you’re testing. 4. Use standards to guide your tests.

Slide 58

Slide 58 text

@GanbaroDigital Why Does Code Coverage Matter?

Slide 59

Slide 59 text

@GanbaroDigital https://flic.kr/p/6My3Ln Makes Me Blue

Slide 60

Slide 60 text

@GanbaroDigital https://flic.kr/p/pGqBUu I’m Here To Pick A Fight

Slide 61

Slide 61 text

@GanbaroDigital Let’s say your app is 10,000 lines of code. Your code coverage target is 60%.

Slide 62

Slide 62 text

@GanbaroDigital 40% 60%

Slide 63

Slide 63 text

@GanbaroDigital ?? ?? How many bugs have your tests found in the 60% of the code that you have tests for?

Slide 64

Slide 64 text

@GanbaroDigital Defects Found 0 30 60 90 120 Covered Code Uncovered Code

Slide 65

Slide 65 text

@GanbaroDigital ?? ?? Why would the defect rate be any different in the remaining 40% of uncovered code?

Slide 66

Slide 66 text

@GanbaroDigital Defects Found 0 30 60 90 120 Covered Code Uncovered Code

Slide 67

Slide 67 text

@GanbaroDigital Defect rate alone doesn’t tell the whole story. Not all defects cost the same when things go wrong.

Slide 68

Slide 68 text

@GanbaroDigital ?? ?? Have you heard of “happy path” programming?

Slide 69

Slide 69 text

@GanbaroDigital

Slide 70

Slide 70 text

@GanbaroDigital

Slide 71

Slide 71 text

@GanbaroDigital

Slide 72

Slide 72 text

@GanbaroDigital The “happy path” only deals with correctness under perfect conditions.

Slide 73

Slide 73 text

@GanbaroDigital This is an app without robustness.

Slide 74

Slide 74 text

@GanbaroDigital “ Correctness bugs are logic flaws that produce unacceptable results.

Slide 75

Slide 75 text

@GanbaroDigital The happy path works when correctness bugs have been found and resolved.

Slide 76

Slide 76 text

@GanbaroDigital “ Robustness code deals with all the things that shouldn’t happen, but inevitably do.

Slide 77

Slide 77 text

@GanbaroDigital

Slide 78

Slide 78 text

@GanbaroDigital

Slide 79

Slide 79 text

@GanbaroDigital

Slide 80

Slide 80 text

@GanbaroDigital

Slide 81

Slide 81 text

@GanbaroDigital x

Slide 82

Slide 82 text

@GanbaroDigital Robustness turns working code into real-world code.

Slide 83

Slide 83 text

@GanbaroDigital Things People Don’t Test • “Trivial” methods • Robustness code

Slide 84

Slide 84 text

@GanbaroDigital “ Even a trivial method can have a typo in a variable name.

Slide 85

Slide 85 text

@GanbaroDigital Things People Don’t Test • “Trivial” methods • Robustness code

Slide 86

Slide 86 text

@GanbaroDigital ?? ?? What happens when your untested robustness code has bugs?

Slide 87

Slide 87 text

@GanbaroDigital “ Robustness code is the absolute worst place to ship bugs.

Slide 88

Slide 88 text

@GanbaroDigital You’re already handling a failure situation.

Slide 89

Slide 89 text

@GanbaroDigital Badly-handled failures create the worst kinds of mess to clean up.

Slide 90

Slide 90 text

@GanbaroDigital https://flic.kr/p/83Z8Kp Angry Customers

Slide 91

Slide 91 text

@GanbaroDigital You hurt your organisation when you ship

Slide 92

Slide 92 text

@GanbaroDigital All failures are exponentially more expensive to put right.

Slide 93

Slide 93 text

@GanbaroDigital The rule of thumb is that it costs 10x more to fix a defect at the next stage.

Slide 94

Slide 94 text

@GanbaroDigital Design Dev Test Live

Slide 95

Slide 95 text

@GanbaroDigital Even if we assume only a 3x cost, the numbers are still ugly.

Slide 96

Slide 96 text

@GanbaroDigital Design Dev Test Live

Slide 97

Slide 97 text

@GanbaroDigital Design Dev Test Live

Slide 98

Slide 98 text

@GanbaroDigital ?? ?? Why does it cost so much?

Slide 99

Slide 99 text

@GanbaroDigital • Redesign footprint • Putting things right for the customer • Opportunity costs

Slide 100

Slide 100 text

@GanbaroDigital • Redesign footprint • Putting things right for the customer • Opportunity costs

Slide 101

Slide 101 text

@GanbaroDigital • Redesign footprint • Putting things right for the customer • Opportunity costs

Slide 102

Slide 102 text

@GanbaroDigital ?? ?? Where does the money come from to cover these costs?

Slide 103

Slide 103 text

@GanbaroDigital

Slide 104

Slide 104 text

@GanbaroDigital 10x programmers are real. They’re just folks who ship quickly whilst consistently reduce and avoid costly rework.

Slide 105

Slide 105 text

@GanbaroDigital 0 75 150 225 300 Design Dev Test Live

Slide 106

Slide 106 text

@GanbaroDigital “ Excellence is a habit.

Slide 107

Slide 107 text

@GanbaroDigital “ Right first time, not rewrite next time.

Slide 108

Slide 108 text

@GanbaroDigital “ Fit for purpose, not tick in a box.

Slide 109

Slide 109 text

@GanbaroDigital “ Work, not rework.

Slide 110

Slide 110 text

@GanbaroDigital https://flic.kr/p/6vLhpM Meta-Ignorance

Slide 111

Slide 111 text

@GanbaroDigital https://flic.kr/p/cwK7UN We’re Professionals, Damnit!

Slide 112

Slide 112 text

@GanbaroDigital “Go 100% code coverage, or go home and leave the job to the professionals.

Slide 113

Slide 113 text

Thank You Any Questions? A presentation by @stuherbert
 for @GanbaroDigital