Upgrade to Pro — share decks privately, control downloads, hide ads and more …

The Ultimate Feedback Loop: How We Decreased Customer Reported Defects by 80%

emanuil
November 17, 2017

The Ultimate Feedback Loop: How We Decreased Customer Reported Defects by 80%

“In God we trust; all others bring data.” W. Edwards Deming

A defect reported by your customers is the most expensive one. Beyond pure money loss, those defects are an embarrassment for any organization. They passed all the quality gates. But the greatest cost that cannot be measured easily is the loss of reputation. It comes as a great surprise then, that almost no company investigates the defects reported by its customers. The companies try to quickly patch the problem and move forward. It’s a shame as a great deal knowledge can be gained about the system that produced those defects.

We’ve analyzed more than two years of customer reported defects. Even though we thought that each defect is unique, some obvious patterns emerged quickly. We were able to debunk widely believed software dogmas that were not working for us. We figured out which of the techniques were helping us or not to lower the defects count: Following the software testing pyramid guidelines? Switching the backend from dynamically typed language to static one? Writing a simple unit test, where there was none? Writing a simple integration test, where there was none? Focusing test engineers to use specific techniques? Using static code analysis? Determining the typical profile of a method that’s likely to contain an error? The end result: we decreased customer reported defects by 80%.

I’ll share Komfo's experience in building a simple framework for analyzing defects as well as tips and tricks so that you can build a similar program in your organization.

emanuil

November 17, 2017
Tweet

More Decks by emanuil

Other Decks in Programming

Transcript

  1. §
    The Ultimate Feedback Loop
    How We Decreased Customer Reported Defects by 80%
    @EmanuilSlavov

    [email protected]

    View full-size slide

  2. “If a tree falls in a forest and no one is
    around to hear it, does it make a sound?”
    If a defect goes unnoticed by
    customers, is it really a defect?
    @EmanuilSlavov

    View full-size slide

  3. Quality is Defined by The Customer
    @EmanuilSlavov

    View full-size slide

  4. Tester
    Defect Found
    Defect Resolved
    Defects found after this line
    are way more expensive to fix.
    Support Customer
    Developer
    All the defenses were bypassed
    The most expensive defects
    Reputation loss hard to measure
    Manager
    @EmanuilSlavov

    View full-size slide

  5. We decided to analyze all customer
    reported issues for more than two
    years.
    @EmanuilSlavov

    View full-size slide

  6. The Basic Stats

    View full-size slide

  7. 12%of all customer issues
    are actually defects
    @EmanuilSlavov

    View full-size slide

  8. 189 customer reported defects analyzed
    All data was collected manually
    Including their fixes
    @EmanuilSlavov

    View full-size slide

  9. 10%of all fixes were caused
    by 3rd party services
    @EmanuilSlavov

    View full-size slide

  10. 44%frontend defects
    56%
    backend defects
    Java
    PHP
    JavaScript
    @EmanuilSlavov

    View full-size slide

  11. 38%of all the defects
    were regressions
    Any automated test would have alerted
    us before they reach production.
    @EmanuilSlavov

    View full-size slide

  12. Not Just Any
    Type of Test…

    View full-size slide

  13. 7%of the methods containing
    defects already had unit tests
    @EmanuilSlavov

    View full-size slide

  14. A Tale of Two
    Locations

    View full-size slide

  15. Defect Location
    Defect Manifestation
    @EmanuilSlavov

    View full-size slide

  16. Different Kinds of Tests

    View full-size slide

  17. 13%of the defects could have
    been caught if a simple
    unit test have been written
    @EmanuilSlavov

    View full-size slide

  18. if(U.isEmpty(post) || post.isDark())
    if(U.isEmpty(post) || post.isDark() && commentIdParts.length > 1)
    @EmanuilSlavov

    View full-size slide

  19. With unit test defect detection “yield”
    so low, which methods should we test?
    72%of the defects were in methods
    with complexity 3 and above
    82%of the defects were in methods
    with more than 10 lines of code
    @EmanuilSlavov

    View full-size slide

  20. 36%of the defects could have
    been realistically caught on
    API level (and not on unit tests level)
    @EmanuilSlavov

    View full-size slide

  21. public function loadCursor($channel_id, $from, $to)
    {
    $collection = $this->getMongoCollection();
    $this->applyChannelId($channel_id);
    $this->applyTimeRanges($from, $to);
    $this->cursor = $collection->find($this->filter);
    }
    public function loadCursor($channel_id, $from, $to)
    {
    $collection = $this->getMongoCollection();
    $this->applyChannelId(urldecode($channel_id));
    $this->applyTimeRanges($from, $to);
    $this->cursor = $collection->find($this->filter);
    }
    @EmanuilSlavov

    View full-size slide

  22. 21%of the defects could have
    been realistically caught on
    UI level
    @EmanuilSlavov

    View full-size slide

  23. reactions = self.facebook_reactions_model.first();
    reactions_popover.find('.reactions_data').show();
    reactions_popover.find('.reactions_like').text(reactions.LIKE);
    reactions = self.facebook_reactions_model.first();
    reactions_popover.find('.reactions_data').show();
    reactions_popover.find('.reactions_like').text(reactions.like);
    @EmanuilSlavov

    View full-size slide

  24. SHOCKING: they don’t
    add up to 100%!

    View full-size slide

  25. 30%of the defects can never be
    realistically detected in-house,
    only in the “real world”
    Edge Cases
    Configuration Issues
    Unintended Usage
    Incomplete Requirements
    Unit API UI
    @EmanuilSlavov

    View full-size slide

  26. Forget those 30%. Instead, focus

    on early detection and fast recovery.
    @EmanuilSlavov

    View full-size slide

  27. 31% of the defects are detected less
    than one month after they are
    introduced in the codebase
    @EmanuilSlavov

    View full-size slide

  28. 51% of the defects are detected less
    than two months after they are
    introduced in the codebase
    @EmanuilSlavov

    View full-size slide

  29. 48%of the defects are fixed within
    one working day after detection
    @EmanuilSlavov

    View full-size slide

  30. 69%of the defects are fixed within
    one working week after detection
    @EmanuilSlavov

    View full-size slide

  31. 2.3%of the defects could be detected
    by custom static code analysis
    PHP: linter, HHVM, custom checks

    Java: The Sonar Suite, FindBugs

    JavaScript: ESLint
    @EmanuilSlavov

    View full-size slide

  32. 6%of the backend defects could be
    avoided by switching from PHP to Java.
    (but Java has hidden costs)
    @EmanuilSlavov

    View full-size slide

  33. 5%
    of the bugs were caused by me
    Developers should fix their defects

    and learn from the experience.
    @EmanuilSlavov

    View full-size slide

  34. At least one automated test written per method
    Manual sanity check even the smallest fixes
    Mandatory code reviews
    Test with boundary values
    @EmanuilSlavov

    View full-size slide

  35. Weird data generator for all automated tests
    Monitor exceptions after each automated tests run
    Fast interration tests - from 3 hours to 3 minutes
    Monitor and immediately fix errors in production
    @EmanuilSlavov

    View full-size slide

  36. 10
    20
    30
    40
    Q3/14 Q4/14 Q1/15 Q2/15 Q3/15 Q4/15 Q1/16 Q2/16 Q3/16 Q4/16 Q1/17
    Java Backend
    Fast Tests
    Monitor Java

    Code Reviews
    Monitor JS
    Weird Test

    Data Generator
    Check for
    exceptions

    after test
    Monitor PHP
    Lines of Code
    35000
    70000
    105000
    140000
    Q3/14 Q4/14 Q1/15 Q2/15 Q3/15 Q4/15 Q1/16 Q2/16 Q3/16 Q4/16 Q1/17
    Defects Count
    @EmanuilSlavov

    View full-size slide

  37. How to Start

    View full-size slide

  38. Allocate time
    Track customer reported defects
    Defect id in the commit message
    Investigate immediately
    Independent analysis
    Figure which metrics to track
    @EmanuilSlavov

    View full-size slide

  39. goo.gl/3Gdsnm

    View full-size slide

  40. @EmanuilSlavov

    View full-size slide

  41. Thank You
    EmanuilSlavov.com
    @EmanuilSlavov

    View full-size slide