The Ultimate Feedback Loop: Learning From Customer Reported Defects

“In God we trust; all others bring data.” W. Edwards Deming

A defect reported by your customers is the most expensive one. Everyone in the developer<->customer chain is getting paid for rework, and wastes time for something that should have beed done right in the first place. Beyond pure money loss, those defects are an embarrassment for any organization. They passed all the quality gates. But the greatest cost that cannot be measured easily is the loss of reputation.

It comes as a great surprise then, that almost no company investigates the defects reported by its customers. The companies try to quickly patch the problem and move forward. It’s a shame as a great deal knowledge can be gained about the system that produced the defect in the first place.

We’ve analyzed more than two years of customer reported defects data. Even though we thought that each defect is unique snowflake, some obvious patterns emerged quickly. We were able to debunk some widely believed software dogmas that were not working for us. We figured out which of the techniques listed were helping us or not to lower the defects count:

Following the software testing pyramid guidelines?
Switching the backend from PHP to Java?
Writing a simple unit test, where there was none?
Writing a simple integration test, where there was none?
Focusing test engineers to use specific techniques?
Using static code analysis?
Determining the typical profile of a method thats likely to contain an error?
What are MTBF, MTTD, MTTR and do they matter?

Each company is different, what works in one situation will not work in another. But we all need to learn from the most expensive kind of defects. This is very powerful feedback mechanism that should not be wasted. We’ll share our experience in building a simple framework for analyzing such defects and well as tips and trick so that you can build a similar program in your organization.

Transcript

1. §
   The Ultimate Feedback Loop
   Learning From Customer Reported Defects
   @EmanuilSlavov
   
   emo@komfo.com
   

   View Slide

2. W. Edwards
   Deming
   

   View Slide

3. CHECK
   DO
   ACT
   PLAN Most companies focus on plan - do
   We’re going to talk about check - act
   

   View Slide

4. Tester
   Defect Found
   Defect Resolved
   Defects found after this line
   are way more expensive to fix.
   Support Customer
   Developer
   All the defenses were bypassed
   The most expensive defects
   Reputation loss hard to measure
   Manager
   

   View Slide

5. The Basic Stats
   

   View Slide

6. 12%of all support calls
   turn up to be defects
   

   View Slide

7. 189 customer reported defects analyzed
   Spanning two and a half years
   All data was collected manually
   Including all the fixes
   

   View Slide

8. 10%of all fixes were caused
   by 3rd party services
   

   View Slide

9. 44%frontend defects
   56%
   backend defects
   Java
   PHP
   JavaScript
   

   View Slide

10. 38%of all the defects
    were regressions
    An automated test would have alerted
    us before they reach production.
    

    View Slide

11. Not Just Any
    Type of Test…
    

    View Slide

12. 7%of the methods containing
    defects already had unit tests
    

    View Slide

13. Different Kinds of Tests
    

    View Slide

14. Unit Tests
    

    View Slide

15. 13%of the defects could have
    been caught if a simple
    unit test have been written
    

    View Slide

16. if(U.isEmpty(post) || post.isDark() && commentIdParts.length > 1)
    if(U.isEmpty(post) || post.isDark())
    

    View Slide

17. With unit test defect detection “yield”
    so low, which methods should we test?
    72%of the defects were in methods
    with complexity 3 and above
    82%of the defects were in methods
    with more than 10 lines of code
    

    View Slide

18. API Tests
    

    View Slide

19. 36%of the defects could have
    been realistically caught on
    API level (and not on unit tests level)
    

    View Slide

20. public function loadCursor($channel_id, $from, $to)
    {
    $collection = $this->getMongoCollection();
    $this->applyChannelId($channel_id);
    $this->applyTimeRanges($from, $to);
    $this->cursor = $collection->find($this->filter);
    }
    public function loadCursor($channel_id, $from, $to)
    {
    $collection = $this->getMongoCollection();
    $this->applyChannelId(urldecode($channel_id));
    $this->applyTimeRanges($from, $to);
    $this->cursor = $collection->find($this->filter);
    }
    

    View Slide

21. UI Tests
    

    View Slide

22. 21%of the defects could have
    been realistically caught on
    UI level
    

    View Slide

23. View Slide

24. reactions = self.facebook_reactions_model.first();
    reactions_popover.find('.reactions_data').show();
    reactions_popover.find('.reactions_like').text(reactions.LIKE);
    reactions = self.facebook_reactions_model.first();
    reactions_popover.find('.reactions_data').show();
    reactions_popover.find('.reactions_like').text(reactions.like);
    

    View Slide

25. SHOCKING: they don’t
    add up to 100%!
    

    View Slide

26. 30%of the defects can never be
    realistically detected in-house,
    only in the “real world”
    Edge Cases
    Configuration Issues
    Unintended Usage
    Incomplete Requirements
    Unit API UI
    

    View Slide

27. Forget those 30%. Instead, focus
    
    on early detection and fast recovery.
    

    View Slide

28. 31%of the defects are detected less
    than one month after introduction
    

    View Slide

29. 51%of the defects are detected less
    than two months after introduction
    

    View Slide

30. 48%of the defects are fixed within
    one working day after detection
    

    View Slide

31. 69%of the defects are fixed within
    one working week after detection
    

    View Slide

32. More Stats
    

    View Slide

33. 2.3%of the defects could be detected
    by custom static code analysis
    PHP: linter, HHVM, custom checks
    
    Java: The Sonar Suite, FindBugs
    
    JavaScript: ESLint
    

    View Slide

34. 6%of the backend defects could be
    avoided if switching from PHP to Java.
    (but Java has hidden costs)
    

    View Slide

35. 5%
    of the bugs were caused by me
    Only developers should fix
    
    defects and learn from them.
    

    View Slide

36. Changes
    

    View Slide

37. At least one automated test written per method
    Manual sanity check even the smallest fixes
    Mandatory code reviews
    Test with boundary values
    

    View Slide

38. Weird data generator for the automated tests
    Monitor exceptions after each automated tests run
    Super fast API tests - from 3 hours to 3 minutes
    Monitor and immediately fix errors in production
    

    View Slide

39. 10
    20
    30
    40
    Q3/14 Q4/14 Q1/15 Q2/15 Q3/15 Q4/15 Q1/16 Q2/16 Q3/16 Q4/16 Q1/17
    Java Backend
    Fast Tests
    Monitor Java
    
    Code Reviews
    Monitor JS
    Weird Test
    
    Data Generator
    Check for
    exceptions
    
    after test
    Monitor PHP
    Defects Count
    Lines of Code
    35000
    70000
    105000
    140000
    Q3/14 Q4/14 Q1/15 Q2/15 Q3/15 Q4/15 Q1/16 Q2/16 Q3/16 Q4/16 Q1/17
    

    View Slide

40. How to Start
    

    View Slide

41. Allocate time
    Track customer reported defects
    Defect id in the commit message
    Investigate immediately
    Independent analysis
    Figure which metrics to track
    

    View Slide

42. View Slide

43. goo.gl/3Gdsnm
    

    View Slide

44. View Slide

45. Thank You
    EmanuilSlavov.com
    @EmanuilSlavov
    

    View Slide