The Ultimate Feedback Loop: How We Decreased Customer Reported Defects by 80%

Slide 1

Slide 1 text

§ The Ultimate Feedback Loop How We Decreased Customer Reported Defects by 80% @EmanuilSlavov [email protected]

Slide 2

Slide 2 text

“If a tree falls in a forest and no one is around to hear it, does it make a sound?” If a defect goes unnoticed by customers, is it really a defect? @EmanuilSlavov

Slide 3

Slide 3 text

Quality is Deﬁned by The Customer @EmanuilSlavov

Slide 4

Slide 4 text

Tester Defect Found Defect Resolved Defects found after this line are way more expensive to ﬁx. Support Customer Developer All the defenses were bypassed The most expensive defects Reputation loss hard to measure Manager @EmanuilSlavov

Slide 5

Slide 5 text

We decided to analyze all customer reported issues for more than two years. @EmanuilSlavov

Slide 6

Slide 6 text

The Basic Stats

Slide 7

Slide 7 text

12%of all customer issues are actually defects @EmanuilSlavov

Slide 8

Slide 8 text

189 customer reported defects analyzed All data was collected manually Including their ﬁxes @EmanuilSlavov

Slide 9

Slide 9 text

10%of all ﬁxes were caused by 3rd party services @EmanuilSlavov

Slide 10

Slide 10 text

44%frontend defects 56% backend defects Java PHP JavaScript @EmanuilSlavov

Slide 11

Slide 11 text

38%of all the defects were regressions Any automated test would have alerted us before they reach production. @EmanuilSlavov

Slide 12

Slide 12 text

Not Just Any Type of Test…

Slide 13

Slide 13 text

7%of the methods containing defects already had unit tests @EmanuilSlavov

Slide 14

Slide 14 text

A Tale of Two Locations

Slide 15

Slide 15 text

Defect Location Defect Manifestation @EmanuilSlavov

Slide 16

Slide 16 text

Diﬀerent Kinds of Tests

Slide 17

Slide 17 text

Unit Tests

Slide 18

Slide 18 text

13%of the defects could have been caught if a simple unit test have been written @EmanuilSlavov

Slide 19

Slide 19 text

if(U.isEmpty(post) || post.isDark()) if(U.isEmpty(post) || post.isDark() && commentIdParts.length > 1) @EmanuilSlavov

Slide 20

Slide 20 text

With unit test defect detection “yield” so low, which methods should we test? 72%of the defects were in methods with complexity 3 and above 82%of the defects were in methods with more than 10 lines of code @EmanuilSlavov

Slide 21

Slide 21 text

API Tests

Slide 22

Slide 22 text

36%of the defects could have been realistically caught on API level (and not on unit tests level) @EmanuilSlavov

Slide 23

Slide 23 text

public function loadCursor($channel_id, $from, $to) { $collection = $this->getMongoCollection(); $this->applyChannelId($channel_id); $this->applyTimeRanges($from, $to); $this->cursor = $collection->find($this->filter); } public function loadCursor($channel_id, $from, $to) { $collection = $this->getMongoCollection(); $this->applyChannelId(urldecode($channel_id)); $this->applyTimeRanges($from, $to); $this->cursor = $collection->find($this->filter); } @EmanuilSlavov

Slide 24

Slide 24 text

UI Tests

Slide 25

Slide 25 text

21%of the defects could have been realistically caught on UI level @EmanuilSlavov

Slide 26

Slide 26 text

No content

Slide 27

Slide 27 text

reactions = self.facebook_reactions_model.first(); reactions_popover.find('.reactions_data').show(); reactions_popover.find('.reactions_like').text(reactions.LIKE); reactions = self.facebook_reactions_model.first(); reactions_popover.find('.reactions_data').show(); reactions_popover.find('.reactions_like').text(reactions.like); @EmanuilSlavov

Slide 28

Slide 28 text

SHOCKING: they don’t add up to 100%!

Slide 29

Slide 29 text

30%of the defects can never be realistically detected in-house, only in the “real world” Edge Cases Conﬁguration Issues Unintended Usage Incomplete Requirements Unit API UI @EmanuilSlavov

Slide 30

Slide 30 text

Forget those 30%. Instead, focus on early detection and fast recovery. @EmanuilSlavov

Slide 31

Slide 31 text

No content

Slide 32

Slide 32 text

31% of the defects are detected less than one month after they are introduced in the codebase @EmanuilSlavov

Slide 33

Slide 33 text

51% of the defects are detected less than two months after they are introduced in the codebase @EmanuilSlavov

Slide 34

Slide 34 text

48%of the defects are ﬁxed within one working day after detection @EmanuilSlavov

Slide 35

Slide 35 text

69%of the defects are ﬁxed within one working week after detection @EmanuilSlavov

Slide 36

Slide 36 text

More Stats

Slide 37

Slide 37 text

2.3%of the defects could be detected by custom static code analysis PHP: linter, HHVM, custom checks Java: The Sonar Suite, FindBugs JavaScript: ESLint @EmanuilSlavov

Slide 38

Slide 38 text

6%of the backend defects could be avoided by switching from PHP to Java. (but Java has hidden costs) @EmanuilSlavov

Slide 39

Slide 39 text

5% of the bugs were caused by me Developers should ﬁx their defects and learn from the experience. @EmanuilSlavov

Slide 40

Slide 40 text

Changes

Slide 41

Slide 41 text

At least one automated test written per method Manual sanity check even the smallest ﬁxes Mandatory code reviews Test with boundary values @EmanuilSlavov

Slide 42

Slide 42 text

Weird data generator for all automated tests Monitor exceptions after each automated tests run Fast interration tests - from 3 hours to 3 minutes Monitor and immediately ﬁx errors in production @EmanuilSlavov

Slide 43

Slide 43 text

10 20 30 40 Q3/14 Q4/14 Q1/15 Q2/15 Q3/15 Q4/15 Q1/16 Q2/16 Q3/16 Q4/16 Q1/17 Java Backend Fast Tests Monitor Java Code Reviews Monitor JS Weird Test Data Generator Check for exceptions after test Monitor PHP Lines of Code 35000 70000 105000 140000 Q3/14 Q4/14 Q1/15 Q2/15 Q3/15 Q4/15 Q1/16 Q2/16 Q3/16 Q4/16 Q1/17 Defects Count @EmanuilSlavov

Slide 44

Slide 44 text

How to Start

Slide 45

Slide 45 text

Allocate time Track customer reported defects Defect id in the commit message Investigate immediately Independent analysis Figure which metrics to track @EmanuilSlavov