Share
Embed
Improvement of Root cause analysis with iterative process
Root Cause Analysis method using the iterative process in order to improve speed and quality of the anaysis of defects in software systems.
This was presented at 6th World Congress of Software Quality in London.
Atsushi Nagata
May 27, 2022
Transcript
62 slides
-
London, 1st - 3rd July 2014 Improvement of Root cause
analysis with iterative process Implementation of Agile Root Cause Analysis Atsushi Nagata Sony Corporation Nobuhiro Hosokawa IBM -
London, 1st - 3rd July 2014 Introduction Sony Corporation Quality
Engineering Manager Professional Division Software Quality Assurance Test Process Improvement JSTQB Advanced Level Test Manager copyright © A.Nagata 2 -
London, 1st - 3rd July 2014 Acknowledgement I would like
to acknowledge Tom Gilb and Kai Gilb. They teach me true agile and improvement process through Agile Inspection. copyright © A.Nagata 3 -
London, 1st - 3rd July 2014 Root Cause Analysis •
Root Cause Analysis(RCA) is useful tool to improve software quality. • Using for not only defect prevention but also it shall contribute important information for test approach and test design. • We were using Five Whys, one of the popular method of RCA copyright © A.Nagata 4 -
London, 1st - 3rd July 2014 5 Whys analysis template
Why ① Why 2 Why 3 Why 4 Why 5 References Symptoms ID Issue Cause of defect installation Root Cuase . . copyright © A.Nagata 5 -
London, 1st - 3rd July 2014 Problem of RCA for
software development • Time consuming • Stressful approach to involve human factor. • Slow improvement in skill of analysis copyright © A.Nagata 6 -
London, 1st - 3rd July 2014 Time consuming • Preparation
• Document of explanation • Analysis • RCA review meeting with 5Why template . • Efficiency • We could not finish only one time. copyright © A.Nagata 7 More than 2 hours 2 hours More than once More than 8 hours http://info.legalzoom.com/ documents-required- preparation-4187.html -
London, 1st - 3rd July 2014 Stressful approach to involve
human factor • Why is the strongest question that might attack the people who made mistakes. copyright © A.Nagata 8 The Art and Architecture of Powerful Questions, Eric E Vogt, Juanita Brown and David Isaacs, 2003 -
London, 1st - 3rd July 2014 Stressful approach to involve
human factor • Why is the strongest question that might attack the people who made mistakes. copyright © A.Nagata 9 The Art and Architecture of Powerful Questions, Eric E Vogt, Juanita Brown and David Isaacs, 2003 -
London, 1st - 3rd July 2014 Stressful approach to involve
human factor • Why is the strongest question that might attack the people who made mistakes. Why did you do this? • How do you feel ? copyright © A.Nagata 10 -
London, 1st - 3rd July 2014 copyright © A.Nagata 11
https://www.flickr.com/photos/ruthhb/2917888819/in/photostream/ -
London, 1st - 3rd July 2014 5 Whys : Strong
Questions copyright © A.Nagata 12 Original : in Manufacturing Objects : physical materials Software Development Defects : physical problems Objects : Engineers Defects : Bugs -
London, 1st - 3rd July 2014 Stressful approach to involve
human factor • Why is the strongest question that might attack the people who made mistakes. • Reaction of blaming, [Defensive mode] • Condition {Poor facilitation & Critical Thinking} copyright © A.Nagata 13 People try to justify their answer rather than proceed in a spirit of inquiry. Eric E. Vogt said -
London, 1st - 3rd July 2014 copyright © A.Nagata 14
https://www.flickr.com/photos/ruthhb/2917888819/in/photostream/ -
London, 1st - 3rd July 2014 Not root cause Answer
Answer Defense mode in 5 Whys Answer Incident Root Cause Why Why Effective Prevention copyright © A.Nagata 15 -
London, 1st - 3rd July 2014 Appropriate Questions Strong Questions
Appropriate Questions What are Appropriate Questions? copyright © A.Nagata 16 -
London, 1st - 3rd July 2014 Five specify questions 1st
Question 2nd Question 3rd Question 4th Question 5th Question Question Answers ID Question Answers ID Question Answers ID Question Answers ID Question Answers ID N/A ID Question Answers ID N/A ID Question Answers ID N/A ID copyright © A.Nagata 17 -
London, 1st - 3rd July 2014 Slow improvement in analysis
skill • Time consuming We could not RCA many times. • It is difficult to consider Appropriate Questions We could not reach the root cause. We need to practice of analysis and facilitation to improve RCA skill. copyright © A.Nagata 18 -
London, 1st - 3rd July 2014 Iterative RCA • Light
weight time box • 15 minutes through 30 minutes per each iteration • Make a habit and a rhythm • Doing every day as possible • Same time every day • After the stand up meeting • 15 minutes to lunch time. • Review the answer and make next question • The reviewer of RCA could discuss with people who made answer in the time box. • The reviewer could make appropriate questions. copyright © A.Nagata 19 -
London, 1st - 3rd July 2014 Iterative RCA 1st Iteration
2nd Iteration 3rd Iteration 4th Question Question Answers ID Question Answers ID Question Answers ID Question Answers ID Question Answers ID Question Answers ID Question Answers ID copyright © A.Nagata 20 -
London, 1st - 3rd July 2014 Effective loop of the
Iterative RCA Good question Right direction Good answer Discuss with answer copyright © A.Nagata 21 -
London, 1st - 3rd July 2014 Answer Iterations make loops
of feedback Incident Root Cause Effective Prevention Answer Q&A Discuss Q&A Discuss Q&A Discuss Q&A Discuss Q&A Discuss Learning Awareness copyright © A.Nagata 22 -
London, 1st - 3rd July 2014 Problems of Iterative RCA
• Sometimes it is difficult to find the next question. • The mechanism of problem is complicated. • A defect of software would be caused by many factors • Each factor is related and influenced with each other. • The model of the defect should be network. • The property of each factor should be classified and described in the model. • The RCA template is not network but tree list of question and answer. • Necessity for network model of Defects copyright © A.Nagata 23 -
London, 1st - 3rd July 2014 RCA template : Tree
structure 1st Question 2nd Question 3rd Question 4th Question 5th Question Question Answers ID Question Answers ID Question Answers ID Question Answers ID Question Answers ID N/A ID Question Answers ID N/A ID Question Answers ID N/A ID copyright © A.Nagata 24 -
London, 1st - 3rd July 2014 Necessity for the defect
model • Factors of defect model • Incident • Defect • Amplifier : accelerate chain reaction to defects • Negligence Factor : Mistakes at human activities • Induction Trigger : Trigger to produce defect Nobuhiro Hosokawa, Yasuharu Nishi, Aya Ureshino, Makoto Nonaka, Yukiko Hara, JaSST 2013 Tokyo – Project Fabre, 2013 copyright © A.Nagata 25 -
London, 1st - 3rd July 2014 Example of Defect Model
Nobuhiro Hosokawa, Yasuharu Nishi, Aya Ureshino, Makoto Nonaka, Yukiko Hara, JaSST 2013 Tokyo – Project Fabre, 2013 <Induction Trigger> Manner to use “Double” data style < Negligence Factor > Select style of data without understanding <Defects> Did not select “Currency” but “Double” as data style <Incident> Rounding error in currency data <Amplifier> Checking system is week <Amplifier> No standard rule of style copyright © A.Nagata 26 -
London, 1st - 3rd July 2014 Answer Iterations make loops
of feedback Incident Root Cause Effective Prevention Answer Q&A Discuss Q&A Discuss Q&A Discuss Q&A Discuss Q&A Discuss copyright © A.Nagata 27 -
London, 1st - 3rd July 2014 Answer RCA Iterations with
Modeling Incident Root Cause Effective Prevention Answer Q&A Discuss Q&A Discuss Q&A Discuss Q&A Discuss Q&A Discuss Modeling Modeling Modeling Modeling copyright © A.Nagata Q&A Discuss : Divergence : concrete Modeling : Convergence : abstract 28 -
London, 1st - 3rd July 2014 Process of Agile Root
Cause Analysis (ARCA) copyright © A.Nagata 29 -
London, 1st - 3rd July 2014 Three loop of ARCA
• Incident Analysis loop • To Identify the problem and clarify the mechanism to cause the problem. • Exploratory Loop • Facilitator and reviewer make some questions to explore the defect and other factors with indication and conjecture. • It should be finished in the time box. • Agile RCA Loop • After the time box, facilitator shall analyze it and make modeling, identify root cause and/or defects. Also work out the next questions with strategy. copyright © A.Nagata 30 -
London, 1st - 3rd July 2014 ARCA process : analysis
team • Moderator • The Moderator should be existed who should understand the principles and process of ARCA. • Participants ( Person concerned) • Reviewers copyright © A.Nagata 31 -
London, 1st - 3rd July 2014 Iteration Day1 : Incident
Analysis The moderator shall make questions to the participants to make clear and share the incident, what was happened. The answers should be analyzed to understand the facts of the problem as possible. Paraphrasing is work very well. copyright © A.Nagata 32 -
London, 1st - 3rd July 2014 Iteration Day 2 :
Questions from Incident Agile RCA meeting with participants and reviewers in 15min through 30 min. The moderator and reviewers ask exploratory questions. copyright © A.Nagata 33 -
London, 1st - 3rd July 2014 After the exploratory loop
: Modeling Moderator makes defect model and consider the next question with the strategy. copyright © A.Nagata 34 -
London, 1st - 3rd July 2014 Iteration Day 3 :
Questions by Defect Model 1. Make agreement of modeling with participants and reviewers 2. The moderator and reviewers ask exploratory questions. Agile RCA meeting with participants and reviewers in 15min through 30 min. copyright © A.Nagata 35 -
London, 1st - 3rd July 2014 After the exploratory loop
: Modeling Moderator would prune the model with abstraction. copyright © A.Nagata 36 -
London, 1st - 3rd July 2014 Process of Agile RCA
with Prevention copyright © A.Nagata 37 Divergence Convergence -
London, 1st - 3rd July 2014 ARCA Case: Day 1
copyright © A.Nagata 38 -
London, 1st - 3rd July 2014 ARCA Case: Day 2
copyright © A.Nagata 39 -
London, 1st - 3rd July 2014 ARCA Case: Day 3
copyright © A.Nagata 40 -
London, 1st - 3rd July 2014 Communication using Tacit Knowledge
42 Tacit Knowledge -
London, 1st - 3rd July 2014 Unexpected Missing Tacit Knowledge
43 Tacit Knowledge -
London, 1st - 3rd July 2014 Supporting Tacit Knowledge 44
Tacit Knowledge Domain Knowledge Context -
London, 1st - 3rd July 2014 Policy Supports Tacit Knowledge
45 Tacit Knowledge Domain Knowledge Context -
London, 1st - 3rd July 2014 ARCA Case: last Day
copyright © A.Nagata 46 -
London, 1st - 3rd July 2014 Case 2 copyright ©
A.Nagata 47 -
London, 1st - 3rd July 2014 Case 2 copyright ©
A.Nagata 48 Condition in Personal Condition in Process -
London, 1st - 3rd July 2014 Case 3 copyright ©
A.Nagata 49 -
London, 1st - 3rd July 2014 Boiling Frog In Agile
development the code is small and working at first. copyright © A.Nagata 50 • Modification incrementally • Increase complexity • Unexpected influences • Implant modules from other programs • Unexpected side effects and reactions • The changing is small, we could not have a motivation to react for it. -
London, 1st - 3rd July 2014 Boiled Frog • Complicated
code • Become Legacy code • Degrade • Unteachable code copyright © A.Nagata 51 -
London, 1st - 3rd July 2014 Effective of ARCA 1
• Continuity • Before • Due to too slow and heavy work, almost teams could not keep RCA. • After • Two teams continue ARCA more than six month. • The members of the team have become habituated to doing Agile RCA everyday. copyright © A.Nagata 52 -
London, 1st - 3rd July 2014 Effective of ARCA 2
• Improve Analysis skill • Speed of analysis • More than four times faster • Before Total 8 hours more • After Total 2 hours • Reach to root cause • Before Some times we could not reach to root cause. • After About one and half hour, we could reach. copyright © A.Nagata 53 -
London, 1st - 3rd July 2014 Effective of ARCA 3
• Improve skill of the facilitation and analysis of RCA because of the continuity long time. • Making the trust relationship with teams. • Defect model becomes object of RCA. • Respect • Understand the difficulty to do correctly right from the beginning • Improve skill of identify problems : 4 times and more • defects and related factors. • Improve the motivation for quality improvement. copyright © A.Nagata 54 -
London, 1st - 3rd July 2014 Cool-headed Analysis Before Objects
: Engineers After Defects : Bugs Objects : Defect Model Defects : Bugs copyright © A.Nagata 55 -
London, 1st - 3rd July 2014 Results of Agile RCA
copyright © A.Nagata 56 Number of Issues : The number of issues (problems to analysis) in six month Number of iteration : Average of number of iterations in issue. Number of Defects : Average of number of defects that were found in issue. Number of Factors : Average of number of factors that were found in issue. Total Times : Average of total time in issue. Number of Issues Number of Iterations Number of Defects Number of Factors Total time (minutes) Team A 6 4 2 15 103 Team B 7 4 4 21 133 -
London, 1st - 3rd July 2014 The reason of agile
• Improvement loop • The outputs from Agile RCA, Defect model is progressing every iteration. • Evaluate Defect model to get feedback • Agree Defect model • Consider and discuss on Defect model • Revise Defect mode to the next stage. . • Agile RCA will contribute contiguous improvement of root cause analysis activities using iterative multiple loops. copyright © A.Nagata 57 -
London, 1st - 3rd July 2014 Answer Agile RCA process
Incident Root Cause Effective Prevention Answer Q&A Discuss Q&A Discuss Q&A Discuss Q&A Discuss Q&A Discuss Modeling Modeling Modeling Modeling copyright © A.Nagata Q&A Discuss : Divergence : concrete Modeling : Convergence : abstract 58 -
London, 1st - 3rd July 2014 Process of Agile RCA
with Prevention copyright © A.Nagata 59 Divergence Convergence -
London, 1st - 3rd July 2014 Issues of ARCA •
Improve the defect modeling • Increase the factors • Speed up the modeling • Consider the pattern of modeling • Making the database of modeling • Improve the strategy to create the question. • Speed up the ARCA to real time solution. • Using not only defect prevention but also fixing the problem at development and debugging. copyright © A.Nagata 60 -
London, 1st - 3rd July 2014 Thank you very much
for your attention copyright © A.Nagata Agile RCA drives the actions for improvement of software quality. 61 -
London, 1st - 3rd July 2014 Reference • Agile Specification
Quality Control: Shifting emphasis from cleanup to sampling defects, Tom Gilb, 2005 • Nobuhiro Hosokawa, Yasuharu Nishi, Aya Ureshino, Makoto Nonaka, Yukiko Hara, JaSST 2013 Tokyo – Project Fabre, 2013 • THE ART OF POWERFUL QUESTIONS, Eric E.Vogt, Juanita Brown, and David Isaacs, 2003 • Software Inspection, Tom Gilb, Dorothy Graham, 1993 • The Practical Guide to Defect Prevention, Marc McDonald, Robert Musson, Ross Smith, 2007, Microsoft Press • ODC - a 10x for Root Cause Analysis, Ram Chillarege, 2006 • Naze Naze Bunseki, Hitoshi Ogura, Nikkei BP, 2010 copyright © A.Nagata 62