Improvement of Root cause analysis with iterative process

Root Cause Analysis method using the iterative process in order to improve speed and quality of the anaysis of defects in software systems.
This was presented at 6th World Congress of Software Quality in London.

Transcript

1. London, 1st - 3rd July 2014 Improvement of Root cause analysis with iterative process Implementation of Agile Root Cause Analysis Atsushi Nagata Sony Corporation Nobuhiro Hosokawa IBM
2. London, 1st - 3rd July 2014 Introduction Sony Corporation Quality Engineering Manager Professional Division Software Quality Assurance Test Process Improvement JSTQB Advanced Level Test Manager copyright © A.Nagata 2
3. London, 1st - 3rd July 2014 Acknowledgement I would like to acknowledge Tom Gilb and Kai Gilb. They teach me true agile and improvement process through Agile Inspection. copyright © A.Nagata 3
4. London, 1st - 3rd July 2014 Root Cause Analysis • Root Cause Analysis(RCA) is useful tool to improve software quality. • Using for not only defect prevention but also it shall contribute important information for test approach and test design. • We were using Five Whys, one of the popular method of RCA copyright © A.Nagata 4
5. London, 1st - 3rd July 2014 5 Whys analysis template Why ① Why 2 Why 3 Why 4 Why 5 References Symptoms ＩＤ Issue Cause of defect installation Root Cuase . . copyright © A.Nagata 5
6. London, 1st - 3rd July 2014 Problem of RCA for software development • Time consuming • Stressful approach to involve human factor. • Slow improvement in skill of analysis copyright © A.Nagata 6
7. London, 1st - 3rd July 2014 Time consuming • Preparation • Document of explanation • Analysis • RCA review meeting with 5Why template . • Efficiency • We could not finish only one time. copyright © A.Nagata 7 More than 2 hours 2 hours More than once More than 8 hours http://info.legalzoom.com/ documents-required- preparation-4187.html
8. London, 1st - 3rd July 2014 Stressful approach to involve human factor • Why is the strongest question that might attack the people who made mistakes. copyright © A.Nagata 8 The Art and Architecture of Powerful Questions, Eric E Vogt, Juanita Brown and David Isaacs, 2003
9. London, 1st - 3rd July 2014 Stressful approach to involve human factor • Why is the strongest question that might attack the people who made mistakes. copyright © A.Nagata 9 The Art and Architecture of Powerful Questions, Eric E Vogt, Juanita Brown and David Isaacs, 2003
10. London, 1st - 3rd July 2014 Stressful approach to involve human factor • Why is the strongest question that might attack the people who made mistakes. Why did you do this? • How do you feel ? copyright © A.Nagata 10
11. London, 1st - 3rd July 2014 copyright © A.Nagata 11 https://www.flickr.com/photos/ruthhb/2917888819/in/photostream/
12. London, 1st - 3rd July 2014 5 Whys : Strong Questions copyright © A.Nagata 12 Original : in Manufacturing Objects : physical materials Software Development Defects : physical problems Objects : Engineers Defects : Bugs
13. London, 1st - 3rd July 2014 Stressful approach to involve human factor • Why is the strongest question that might attack the people who made mistakes. • Reaction of blaming, [Defensive mode] • Condition {Poor facilitation & Critical Thinking} copyright © A.Nagata 13 People try to justify their answer rather than proceed in a spirit of inquiry. Eric E. Vogt said
14. London, 1st - 3rd July 2014 copyright © A.Nagata 14 https://www.flickr.com/photos/ruthhb/2917888819/in/photostream/
15. London, 1st - 3rd July 2014 Not root cause Answer Answer Defense mode in 5 Whys Answer Incident Root Cause Why Why Effective Prevention copyright © A.Nagata 15
16. London, 1st - 3rd July 2014 Appropriate Questions Strong Questions Appropriate Questions What are Appropriate Questions? copyright © A.Nagata 16
17. London, 1st - 3rd July 2014 Five specify questions 1st Question 2nd Question 3rd Question 4th Question 5th Question Question Answers ID Question Answers ID Question Answers ID Question Answers ID Question Answers ID N/A ID Question Answers ID N/A ID Question Answers ID N/A ID copyright © A.Nagata 17
18. London, 1st - 3rd July 2014 Slow improvement in analysis skill • Time consuming We could not RCA many times. • It is difficult to consider Appropriate Questions We could not reach the root cause. We need to practice of analysis and facilitation to improve RCA skill. copyright © A.Nagata 18
19. London, 1st - 3rd July 2014 Iterative RCA • Light weight time box • 15 minutes through 30 minutes per each iteration • Make a habit and a rhythm • Doing every day as possible • Same time every day • After the stand up meeting • 15 minutes to lunch time. • Review the answer and make next question • The reviewer of RCA could discuss with people who made answer in the time box. • The reviewer could make appropriate questions. copyright © A.Nagata 19
20. London, 1st - 3rd July 2014 Iterative RCA 1st Iteration 2nd Iteration 3rd Iteration 4th Question Question Answers ID Question Answers ID Question Answers ID Question Answers ID Question Answers ID Question Answers ID Question Answers ID copyright © A.Nagata 20
21. London, 1st - 3rd July 2014 Effective loop of the Iterative RCA Good question Right direction Good answer Discuss with answer copyright © A.Nagata 21
22. London, 1st - 3rd July 2014 Answer Iterations make loops of feedback Incident Root Cause Effective Prevention Answer Q&A Discuss Q&A Discuss Q&A Discuss Q&A Discuss Q&A Discuss Learning Awareness copyright © A.Nagata 22
23. London, 1st - 3rd July 2014 Problems of Iterative RCA • Sometimes it is difficult to find the next question. • The mechanism of problem is complicated. • A defect of software would be caused by many factors • Each factor is related and influenced with each other. • The model of the defect should be network. • The property of each factor should be classified and described in the model. • The RCA template is not network but tree list of question and answer. • Necessity for network model of Defects copyright © A.Nagata 23
24. London, 1st - 3rd July 2014 RCA template : Tree structure 1st Question 2nd Question 3rd Question 4th Question 5th Question Question Answers ID Question Answers ID Question Answers ID Question Answers ID Question Answers ID N/A ID Question Answers ID N/A ID Question Answers ID N/A ID copyright © A.Nagata 24
25. London, 1st - 3rd July 2014 Necessity for the defect model • Factors of defect model • Incident • Defect • Amplifier : accelerate chain reaction to defects • Negligence Factor : Mistakes at human activities • Induction Trigger : Trigger to produce defect Nobuhiro Hosokawa, Yasuharu Nishi, Aya Ureshino, Makoto Nonaka, Yukiko Hara, JaSST 2013 Tokyo – Project Fabre, 2013 copyright © A.Nagata 25
26. London, 1st - 3rd July 2014 Example of Defect Model Nobuhiro Hosokawa, Yasuharu Nishi, Aya Ureshino, Makoto Nonaka, Yukiko Hara, JaSST 2013 Tokyo – Project Fabre, 2013 <Induction Trigger> Manner to use “Double” data style < Negligence Factor > Select style of data without understanding <Defects> Did not select “Currency” but “Double” as data style <Incident> Rounding error in currency data <Amplifier> Checking system is week <Amplifier> No standard rule of style copyright © A.Nagata 26
27. London, 1st - 3rd July 2014 Answer Iterations make loops of feedback Incident Root Cause Effective Prevention Answer Q&A Discuss Q&A Discuss Q&A Discuss Q&A Discuss Q&A Discuss copyright © A.Nagata 27
28. London, 1st - 3rd July 2014 Answer RCA Iterations with Modeling Incident Root Cause Effective Prevention Answer Q&A Discuss Q&A Discuss Q&A Discuss Q&A Discuss Q&A Discuss Modeling Modeling Modeling Modeling copyright © A.Nagata Q&A Discuss : Divergence : concrete Modeling : Convergence : abstract 28
29. London, 1st - 3rd July 2014 Process of Agile Root Cause Analysis (ARCA) copyright © A.Nagata 29
30. London, 1st - 3rd July 2014 Three loop of ARCA • Incident Analysis loop • To Identify the problem and clarify the mechanism to cause the problem. • Exploratory Loop • Facilitator and reviewer make some questions to explore the defect and other factors with indication and conjecture. • It should be finished in the time box. • Agile RCA Loop • After the time box, facilitator shall analyze it and make modeling, identify root cause and/or defects. Also work out the next questions with strategy. copyright © A.Nagata 30
31. London, 1st - 3rd July 2014 ARCA process : analysis team • Moderator • The Moderator should be existed who should understand the principles and process of ARCA. • Participants ( Person concerned) • Reviewers copyright © A.Nagata 31
32. London, 1st - 3rd July 2014 Iteration Day1 : Incident Analysis The moderator shall make questions to the participants to make clear and share the incident, what was happened. The answers should be analyzed to understand the facts of the problem as possible. Paraphrasing is work very well. copyright © A.Nagata 32
33. London, 1st - 3rd July 2014 Iteration Day 2 : Questions from Incident Agile RCA meeting with participants and reviewers in 15min through 30 min. The moderator and reviewers ask exploratory questions. copyright © A.Nagata 33
34. London, 1st - 3rd July 2014 After the exploratory loop : Modeling Moderator makes defect model and consider the next question with the strategy. copyright © A.Nagata 34
35. London, 1st - 3rd July 2014 Iteration Day 3 : Questions by Defect Model 1. Make agreement of modeling with participants and reviewers 2. The moderator and reviewers ask exploratory questions. Agile RCA meeting with participants and reviewers in 15min through 30 min. copyright © A.Nagata 35
36. London, 1st - 3rd July 2014 After the exploratory loop : Modeling Moderator would prune the model with abstraction. copyright © A.Nagata 36
37. London, 1st - 3rd July 2014 Process of Agile RCA with Prevention copyright © A.Nagata 37 Divergence Convergence
38. London, 1st - 3rd July 2014 ARCA Case: Day 1 copyright © A.Nagata 38
39. London, 1st - 3rd July 2014 ARCA Case: Day 2 copyright © A.Nagata 39
40. London, 1st - 3rd July 2014 ARCA Case: Day 3 copyright © A.Nagata 40
41. London, 1st - 3rd July 2014 Tacit Knowledge 41 Tacit Knowledge
42. London, 1st - 3rd July 2014 Communication using Tacit Knowledge 42 Tacit Knowledge
43. London, 1st - 3rd July 2014 Unexpected Missing Tacit Knowledge 43 Tacit Knowledge
44. London, 1st - 3rd July 2014 Supporting Tacit Knowledge 44 Tacit Knowledge Domain Knowledge Context
45. London, 1st - 3rd July 2014 Policy Supports Tacit Knowledge 45 Tacit Knowledge Domain Knowledge Context
46. London, 1st - 3rd July 2014 ARCA Case: last Day copyright © A.Nagata 46
47. London, 1st - 3rd July 2014 Case 2 copyright © A.Nagata 47
48. London, 1st - 3rd July 2014 Case 2 copyright © A.Nagata 48 Condition in Personal Condition in Process
49. London, 1st - 3rd July 2014 Case 3 copyright © A.Nagata 49
50. London, 1st - 3rd July 2014 Boiling Frog In Agile development the code is small and working at first. copyright © A.Nagata 50 • Modification incrementally • Increase complexity • Unexpected influences • Implant modules from other programs • Unexpected side effects and reactions • The changing is small, we could not have a motivation to react for it.
51. London, 1st - 3rd July 2014 Boiled Frog • Complicated code • Become Legacy code • Degrade • Unteachable code copyright © A.Nagata 51
52. London, 1st - 3rd July 2014 Effective of ARCA 1 • Continuity • Before • Due to too slow and heavy work, almost teams could not keep RCA. • After • Two teams continue ARCA more than six month. • The members of the team have become habituated to doing Agile RCA everyday. copyright © A.Nagata 52
53. London, 1st - 3rd July 2014 Effective of ARCA 2 • Improve Analysis skill • Speed of analysis • More than four times faster • Before Total 8 hours more • After Total 2 hours • Reach to root cause • Before Some times we could not reach to root cause. • After About one and half hour, we could reach. copyright © A.Nagata 53
54. London, 1st - 3rd July 2014 Effective of ARCA 3 • Improve skill of the facilitation and analysis of RCA because of the continuity long time. • Making the trust relationship with teams. • Defect model becomes object of RCA. • Respect • Understand the difficulty to do correctly right from the beginning • Improve skill of identify problems : 4 times and more • defects and related factors. • Improve the motivation for quality improvement. copyright © A.Nagata 54
55. London, 1st - 3rd July 2014 Cool-headed Analysis Before Objects : Engineers After Defects : Bugs Objects : Defect Model Defects : Bugs copyright © A.Nagata 55
56. London, 1st - 3rd July 2014 Results of Agile RCA copyright © A.Nagata 56 Number of Issues : The number of issues (problems to analysis) in six month Number of iteration : Average of number of iterations in issue. Number of Defects : Average of number of defects that were found in issue. Number of Factors : Average of number of factors that were found in issue. Total Times : Average of total time in issue. Number of Issues Number of Iterations Number of Defects Number of Factors Total time (minutes) Team A 6 4 2 15 103 Team B 7 4 4 21 133
57. London, 1st - 3rd July 2014 The reason of agile • Improvement loop • The outputs from Agile RCA, Defect model is progressing every iteration. • Evaluate Defect model to get feedback • Agree Defect model • Consider and discuss on Defect model • Revise Defect mode to the next stage. . • Agile RCA will contribute contiguous improvement of root cause analysis activities using iterative multiple loops. copyright © A.Nagata 57
58. London, 1st - 3rd July 2014 Answer Agile RCA process Incident Root Cause Effective Prevention Answer Q&A Discuss Q&A Discuss Q&A Discuss Q&A Discuss Q&A Discuss Modeling Modeling Modeling Modeling copyright © A.Nagata Q&A Discuss : Divergence : concrete Modeling : Convergence : abstract 58
59. London, 1st - 3rd July 2014 Process of Agile RCA with Prevention copyright © A.Nagata 59 Divergence Convergence
60. London, 1st - 3rd July 2014 Issues of ARCA • Improve the defect modeling • Increase the factors • Speed up the modeling • Consider the pattern of modeling • Making the database of modeling • Improve the strategy to create the question. • Speed up the ARCA to real time solution. • Using not only defect prevention but also fixing the problem at development and debugging. copyright © A.Nagata 60
61. London, 1st - 3rd July 2014 Thank you very much for your attention copyright © A.Nagata Agile RCA drives the actions for improvement of software quality. 61
62. London, 1st - 3rd July 2014 Reference • Agile Specification Quality Control: Shifting emphasis from cleanup to sampling defects, Tom Gilb, 2005 • Nobuhiro Hosokawa, Yasuharu Nishi, Aya Ureshino, Makoto Nonaka, Yukiko Hara, JaSST 2013 Tokyo – Project Fabre, 2013 • THE ART OF POWERFUL QUESTIONS, Eric E.Vogt, Juanita Brown, and David Isaacs, 2003 • Software Inspection, Tom Gilb, Dorothy Graham, 1993 • The Practical Guide to Defect Prevention, Marc McDonald, Robert Musson, Ross Smith, 2007, Microsoft Press • ODC - a 10x for Root Cause Analysis, Ram Chillarege, 2006 • Naze Naze Bunseki, Hitoshi Ogura, Nikkei BP, 2010 copyright © A.Nagata 62