The Ultimate Feedback Loop: Learning From Customer Reported Defects

1f686da361195e15bb4e478397a4fc8f?s=47 emanuil
April 28, 2017

The Ultimate Feedback Loop: Learning From Customer Reported Defects

“In God we trust; all others bring data.” W. Edwards Deming

A defect reported by your customers is the most expensive one. Everyone in the developercustomer chain is getting paid for rework, and wastes time for something that should have beed done right in the first place. Beyond pure money loss, those defects are an embarrassment for any organization. They passed all the quality gates. But the greatest cost that cannot be measured easily is the loss of reputation.

It comes as a great surprise then, that almost no company investigates the defects reported by its customers. The companies try to quickly patch the problem and move forward. It’s a shame as a great deal knowledge can be gained about the system that produced the defect in the first place.

We’ve analyzed more than two years of customer reported defects data. Even though we thought that each defect is unique snowflake, some obvious patterns emerged quickly. We were able to debunk some widely believed software dogmas that were not working for us. We figured out which of the techniques listed were helping us or not to lower the defects count:

Following the software testing pyramid guidelines?
Switching the backend from PHP to Java?
Writing a simple unit test, where there was none?
Writing a simple integration test, where there was none?
Focusing test engineers to use specific techniques?
Using static code analysis?
Determining the typical profile of a method thats likely to contain an error?
What are MTBF, MTTD, MTTR and do they matter?

Each company is different, what works in one situation will not work in another. But we all need to learn from the most expensive kind of defects. This is very powerful feedback mechanism that should not be wasted. We’ll share our experience in building a simple framework for analyzing such defects and well as tips and trick so that you can build a similar program in your organization.

1f686da361195e15bb4e478397a4fc8f?s=128

emanuil

April 28, 2017
Tweet

Transcript

  1. § The Ultimate Feedback Loop Learning From Customer Reported Defects

    @EmanuilSlavov emo@komfo.com
  2. W. Edwards Deming

  3. CHECK DO ACT PLAN Most companies focus on plan -

    do We’re going to talk about check - act
  4. Tester Defect Found Defect Resolved Defects found after this line

    are way more expensive to fix. Support Customer Developer All the defenses were bypassed The most expensive defects Reputation loss hard to measure Manager
  5. The Basic Stats

  6. 12%of all support calls turn up to be defects

  7. 189 customer reported defects analyzed Spanning two and a half

    years All data was collected manually Including all the fixes
  8. 10%of all fixes were caused by 3rd party services

  9. 44%frontend defects 56% backend defects Java PHP JavaScript

  10. 38%of all the defects were regressions An automated test would

    have alerted us before they reach production.
  11. Not Just Any Type of Test…

  12. 7%of the methods containing defects already had unit tests

  13. Different Kinds of Tests

  14. Unit Tests

  15. 13%of the defects could have been caught if a simple

    unit test have been written
  16. if(U.isEmpty(post) || post.isDark() && commentIdParts.length > 1) if(U.isEmpty(post) || post.isDark())

  17. With unit test defect detection “yield” so low, which methods

    should we test? 72%of the defects were in methods with complexity 3 and above 82%of the defects were in methods with more than 10 lines of code
  18. API Tests

  19. 36%of the defects could have been realistically caught on API

    level (and not on unit tests level)
  20. public function loadCursor($channel_id, $from, $to) { $collection = $this->getMongoCollection(); $this->applyChannelId($channel_id);

    $this->applyTimeRanges($from, $to); $this->cursor = $collection->find($this->filter); } public function loadCursor($channel_id, $from, $to) { $collection = $this->getMongoCollection(); $this->applyChannelId(urldecode($channel_id)); $this->applyTimeRanges($from, $to); $this->cursor = $collection->find($this->filter); }
  21. UI Tests

  22. 21%of the defects could have been realistically caught on UI

    level
  23. None
  24. reactions = self.facebook_reactions_model.first(); reactions_popover.find('.reactions_data').show(); reactions_popover.find('.reactions_like').text(reactions.LIKE); reactions = self.facebook_reactions_model.first(); reactions_popover.find('.reactions_data').show(); reactions_popover.find('.reactions_like').text(reactions.like);

  25. SHOCKING: they don’t add up to 100%!

  26. 30%of the defects can never be realistically detected in-house, only

    in the “real world” Edge Cases Configuration Issues Unintended Usage Incomplete Requirements Unit API UI
  27. Forget those 30%. Instead, focus on early detection and fast

    recovery.
  28. 31%of the defects are detected less than one month after

    introduction
  29. 51%of the defects are detected less than two months after

    introduction
  30. 48%of the defects are fixed within one working day after

    detection
  31. 69%of the defects are fixed within one working week after

    detection
  32. More Stats

  33. 2.3%of the defects could be detected by custom static code

    analysis PHP: linter, HHVM, custom checks Java: The Sonar Suite, FindBugs JavaScript: ESLint
  34. 6%of the backend defects could be avoided if switching from

    PHP to Java. (but Java has hidden costs)
  35. 5% of the bugs were caused by me Only developers

    should fix defects and learn from them.
  36. Changes

  37. At least one automated test written per method Manual sanity

    check even the smallest fixes Mandatory code reviews Test with boundary values
  38. Weird data generator for the automated tests Monitor exceptions after

    each automated tests run Super fast API tests - from 3 hours to 3 minutes Monitor and immediately fix errors in production
  39. 10 20 30 40 Q3/14 Q4/14 Q1/15 Q2/15 Q3/15 Q4/15

    Q1/16 Q2/16 Q3/16 Q4/16 Q1/17 Java Backend Fast Tests Monitor Java Code Reviews Monitor JS Weird Test Data Generator Check for exceptions after test Monitor PHP Defects Count Lines of Code 35000 70000 105000 140000 Q3/14 Q4/14 Q1/15 Q2/15 Q3/15 Q4/15 Q1/16 Q2/16 Q3/16 Q4/16 Q1/17
  40. How to Start

  41. Allocate time Track customer reported defects Defect id in the

    commit message Investigate immediately Independent analysis Figure which metrics to track
  42. None
  43. goo.gl/3Gdsnm

  44. None
  45. Thank You EmanuilSlavov.com @EmanuilSlavov