The Ultimate Feedback Loop: How We Decreased Customer Reported Defects by 80%

1f686da361195e15bb4e478397a4fc8f?s=47 emanuil
November 17, 2017

The Ultimate Feedback Loop: How We Decreased Customer Reported Defects by 80%

“In God we trust; all others bring data.” W. Edwards Deming

A defect reported by your customers is the most expensive one. Beyond pure money loss, those defects are an embarrassment for any organization. They passed all the quality gates. But the greatest cost that cannot be measured easily is the loss of reputation. It comes as a great surprise then, that almost no company investigates the defects reported by its customers. The companies try to quickly patch the problem and move forward. It’s a shame as a great deal knowledge can be gained about the system that produced those defects.

We’ve analyzed more than two years of customer reported defects. Even though we thought that each defect is unique, some obvious patterns emerged quickly. We were able to debunk widely believed software dogmas that were not working for us. We figured out which of the techniques were helping us or not to lower the defects count: Following the software testing pyramid guidelines? Switching the backend from dynamically typed language to static one? Writing a simple unit test, where there was none? Writing a simple integration test, where there was none? Focusing test engineers to use specific techniques? Using static code analysis? Determining the typical profile of a method that’s likely to contain an error? The end result: we decreased customer reported defects by 80%.

I’ll share Komfo's experience in building a simple framework for analyzing defects as well as tips and tricks so that you can build a similar program in your organization.

1f686da361195e15bb4e478397a4fc8f?s=128

emanuil

November 17, 2017
Tweet

Transcript

  1. § The Ultimate Feedback Loop How We Decreased Customer Reported

    Defects by 80% @EmanuilSlavov emo@komfo.com
  2. “If a tree falls in a forest and no one

    is around to hear it, does it make a sound?” If a defect goes unnoticed by customers, is it really a defect? @EmanuilSlavov
  3. Quality is Defined by The Customer @EmanuilSlavov

  4. Tester Defect Found Defect Resolved Defects found after this line

    are way more expensive to fix. Support Customer Developer All the defenses were bypassed The most expensive defects Reputation loss hard to measure Manager @EmanuilSlavov
  5. We decided to analyze all customer reported issues for more

    than two years. @EmanuilSlavov
  6. The Basic Stats

  7. 12%of all customer issues are actually defects @EmanuilSlavov

  8. 189 customer reported defects analyzed All data was collected manually

    Including their fixes @EmanuilSlavov
  9. 10%of all fixes were caused by 3rd party services @EmanuilSlavov

  10. 44%frontend defects 56% backend defects Java PHP JavaScript @EmanuilSlavov

  11. 38%of all the defects were regressions Any automated test would

    have alerted us before they reach production. @EmanuilSlavov
  12. Not Just Any Type of Test…

  13. 7%of the methods containing defects already had unit tests @EmanuilSlavov

  14. A Tale of Two Locations

  15. Defect Location Defect Manifestation @EmanuilSlavov

  16. Different Kinds of Tests

  17. Unit Tests

  18. 13%of the defects could have been caught if a simple

    unit test have been written @EmanuilSlavov
  19. if(U.isEmpty(post) || post.isDark()) if(U.isEmpty(post) || post.isDark() && commentIdParts.length > 1)

    @EmanuilSlavov
  20. With unit test defect detection “yield” so low, which methods

    should we test? 72%of the defects were in methods with complexity 3 and above 82%of the defects were in methods with more than 10 lines of code @EmanuilSlavov
  21. API Tests

  22. 36%of the defects could have been realistically caught on API

    level (and not on unit tests level) @EmanuilSlavov
  23. public function loadCursor($channel_id, $from, $to) { $collection = $this->getMongoCollection(); $this->applyChannelId($channel_id);

    $this->applyTimeRanges($from, $to); $this->cursor = $collection->find($this->filter); } public function loadCursor($channel_id, $from, $to) { $collection = $this->getMongoCollection(); $this->applyChannelId(urldecode($channel_id)); $this->applyTimeRanges($from, $to); $this->cursor = $collection->find($this->filter); } @EmanuilSlavov
  24. UI Tests

  25. 21%of the defects could have been realistically caught on UI

    level @EmanuilSlavov
  26. None
  27. reactions = self.facebook_reactions_model.first(); reactions_popover.find('.reactions_data').show(); reactions_popover.find('.reactions_like').text(reactions.LIKE); reactions = self.facebook_reactions_model.first(); reactions_popover.find('.reactions_data').show(); reactions_popover.find('.reactions_like').text(reactions.like);

    @EmanuilSlavov
  28. SHOCKING: they don’t add up to 100%!

  29. 30%of the defects can never be realistically detected in-house, only

    in the “real world” Edge Cases Configuration Issues Unintended Usage Incomplete Requirements Unit API UI @EmanuilSlavov
  30. Forget those 30%. Instead, focus on early detection and fast

    recovery. @EmanuilSlavov
  31. None
  32. 31% of the defects are detected less than one month

    after they are introduced in the codebase @EmanuilSlavov
  33. 51% of the defects are detected less than two months

    after they are introduced in the codebase @EmanuilSlavov
  34. 48%of the defects are fixed within one working day after

    detection @EmanuilSlavov
  35. 69%of the defects are fixed within one working week after

    detection @EmanuilSlavov
  36. More Stats

  37. 2.3%of the defects could be detected by custom static code

    analysis PHP: linter, HHVM, custom checks Java: The Sonar Suite, FindBugs JavaScript: ESLint @EmanuilSlavov
  38. 6%of the backend defects could be avoided by switching from

    PHP to Java. (but Java has hidden costs) @EmanuilSlavov
  39. 5% of the bugs were caused by me Developers should

    fix their defects and learn from the experience. @EmanuilSlavov
  40. Changes

  41. At least one automated test written per method Manual sanity

    check even the smallest fixes Mandatory code reviews Test with boundary values @EmanuilSlavov
  42. Weird data generator for all automated tests Monitor exceptions after

    each automated tests run Fast interration tests - from 3 hours to 3 minutes Monitor and immediately fix errors in production @EmanuilSlavov
  43. 10 20 30 40 Q3/14 Q4/14 Q1/15 Q2/15 Q3/15 Q4/15

    Q1/16 Q2/16 Q3/16 Q4/16 Q1/17 Java Backend Fast Tests Monitor Java Code Reviews Monitor JS Weird Test Data Generator Check for exceptions after test Monitor PHP Lines of Code 35000 70000 105000 140000 Q3/14 Q4/14 Q1/15 Q2/15 Q3/15 Q4/15 Q1/16 Q2/16 Q3/16 Q4/16 Q1/17 Defects Count @EmanuilSlavov
  44. How to Start

  45. Allocate time Track customer reported defects Defect id in the

    commit message Investigate immediately Independent analysis Figure which metrics to track @EmanuilSlavov
  46. None
  47. goo.gl/3Gdsnm

  48. @EmanuilSlavov

  49. Thank You EmanuilSlavov.com @EmanuilSlavov