“In God we trust; all others bring data.” W. Edwards Deming
A defect reported by your customers is the most expensive one. Beyond pure money loss, those defects are an embarrassment for any organization. They passed all the quality gates. But the greatest cost that cannot be measured easily is the loss of reputation. It comes as a great surprise then, that almost no company investigates the defects reported by its customers. The companies try to quickly patch the problem and move forward. It’s a shame as a great deal knowledge can be gained about the system that produced those defects.
We’ve analyzed more than two years of customer reported defects. Even though we thought that each defect is unique, some obvious patterns emerged quickly. We were able to debunk widely believed software dogmas that were not working for us. We figured out which of the techniques were helping us or not to lower the defects count: Following the software testing pyramid guidelines? Switching the backend from dynamically typed language to static one? Writing a simple unit test, where there was none? Writing a simple integration test, where there was none? Focusing test engineers to use specific techniques? Using static code analysis? Determining the typical profile of a method that’s likely to contain an error? The end result: we decreased customer reported defects by 80%.
I’ll share Komfo's experience in building a simple framework for analyzing defects as well as tips and tricks so that you can build a similar program in your organization.