The Ultimate Feedback Loop: Learning From Customer Reported Defects

Slide 1

Slide 1 text

§ The Ultimate Feedback Loop Learning From Customer Reported Defects @EmanuilSlavov [email protected]

Slide 2

Slide 2 text

W. Edwards Deming

Slide 3

Slide 3 text

CHECK DO ACT PLAN Most companies focus on plan - do We’re going to talk about check - act

Slide 4

Slide 4 text

Tester Defect Found Defect Resolved Defects found after this line are way more expensive to ﬁx. Support Customer Developer All the defenses were bypassed The most expensive defects Reputation loss hard to measure Manager

Slide 5

Slide 5 text

The Basic Stats

Slide 6

Slide 6 text

12%of all support calls turn up to be defects

Slide 7

Slide 7 text

189 customer reported defects analyzed Spanning two and a half years All data was collected manually Including all the ﬁxes

Slide 8

Slide 8 text

10%of all ﬁxes were caused by 3rd party services

Slide 9

Slide 9 text

44%frontend defects 56% backend defects Java PHP JavaScript

Slide 10

Slide 10 text

38%of all the defects were regressions An automated test would have alerted us before they reach production.

Slide 11

Slide 11 text

Not Just Any Type of Test…

Slide 12

Slide 12 text

7%of the methods containing defects already had unit tests

Slide 13

Slide 13 text

Diﬀerent Kinds of Tests

Slide 14

Slide 14 text

Unit Tests

Slide 15

Slide 15 text

13%of the defects could have been caught if a simple unit test have been written

Slide 16

Slide 16 text

if(U.isEmpty(post) || post.isDark() && commentIdParts.length > 1) if(U.isEmpty(post) || post.isDark())

Slide 17

Slide 17 text

With unit test defect detection “yield” so low, which methods should we test? 72%of the defects were in methods with complexity 3 and above 82%of the defects were in methods with more than 10 lines of code

Slide 18

Slide 18 text

API Tests

Slide 19

Slide 19 text

36%of the defects could have been realistically caught on API level (and not on unit tests level)

Slide 20

Slide 20 text

public function loadCursor($channel_id, $from, $to) { $collection = $this->getMongoCollection(); $this->applyChannelId($channel_id); $this->applyTimeRanges($from, $to); $this->cursor = $collection->find($this->filter); } public function loadCursor($channel_id, $from, $to) { $collection = $this->getMongoCollection(); $this->applyChannelId(urldecode($channel_id)); $this->applyTimeRanges($from, $to); $this->cursor = $collection->find($this->filter); }

Slide 21

Slide 21 text

UI Tests

Slide 22

Slide 22 text

21%of the defects could have been realistically caught on UI level

Slide 23

Slide 23 text

No content

Slide 24

Slide 24 text

reactions = self.facebook_reactions_model.first(); reactions_popover.find('.reactions_data').show(); reactions_popover.find('.reactions_like').text(reactions.LIKE); reactions = self.facebook_reactions_model.first(); reactions_popover.find('.reactions_data').show(); reactions_popover.find('.reactions_like').text(reactions.like);

Slide 25

Slide 25 text

SHOCKING: they don’t add up to 100%!

Slide 26

Slide 26 text

30%of the defects can never be realistically detected in-house, only in the “real world” Edge Cases Conﬁguration Issues Unintended Usage Incomplete Requirements Unit API UI

Slide 27

Slide 27 text

Forget those 30%. Instead, focus on early detection and fast recovery.

Slide 28

Slide 28 text

31%of the defects are detected less than one month after introduction

Slide 29

Slide 29 text

51%of the defects are detected less than two months after introduction

Slide 30

Slide 30 text

48%of the defects are ﬁxed within one working day after detection

Slide 31

Slide 31 text

69%of the defects are ﬁxed within one working week after detection

Slide 32

Slide 32 text

More Stats

Slide 33

Slide 33 text

2.3%of the defects could be detected by custom static code analysis PHP: linter, HHVM, custom checks Java: The Sonar Suite, FindBugs JavaScript: ESLint

Slide 34

Slide 34 text

6%of the backend defects could be avoided if switching from PHP to Java. (but Java has hidden costs)

Slide 35

Slide 35 text

5% of the bugs were caused by me Only developers should ﬁx defects and learn from them.

Slide 36

Slide 36 text

Changes

Slide 37

Slide 37 text

At least one automated test written per method Manual sanity check even the smallest ﬁxes Mandatory code reviews Test with boundary values

Slide 38

Slide 38 text

Weird data generator for the automated tests Monitor exceptions after each automated tests run Super fast API tests - from 3 hours to 3 minutes Monitor and immediately ﬁx errors in production

Slide 39

Slide 39 text

10 20 30 40 Q3/14 Q4/14 Q1/15 Q2/15 Q3/15 Q4/15 Q1/16 Q2/16 Q3/16 Q4/16 Q1/17 Java Backend Fast Tests Monitor Java Code Reviews Monitor JS Weird Test Data Generator Check for exceptions after test Monitor PHP Defects Count Lines of Code 35000 70000 105000 140000 Q3/14 Q4/14 Q1/15 Q2/15 Q3/15 Q4/15 Q1/16 Q2/16 Q3/16 Q4/16 Q1/17

Slide 40

Slide 40 text

How to Start

Slide 41

Slide 41 text

Allocate time Track customer reported defects Defect id in the commit message Investigate immediately Independent analysis Figure which metrics to track