Improvement of Root cause analysis with iterative process

Root Cause Analysis method using the iterative process in order to improve speed and quality of the anaysis of defects in software systems.
This was presented at 6th World Congress of Software Quality in London.

Transcript

1. London, 1st - 3rd July 2014
   Improvement of Root cause analysis
   with iterative process
   Implementation of Agile Root Cause Analysis
   Atsushi Nagata
   Sony Corporation
   Nobuhiro Hosokawa
   IBM
   

   View Slide

2. London, 1st - 3rd July 2014
   Introduction
   Sony Corporation
   Quality Engineering Manager
   Professional Division
   Software Quality Assurance
   Test Process Improvement
   JSTQB Advanced Level Test Manager
   copyright © A.Nagata 2
   

   View Slide

3. London, 1st - 3rd July 2014
   Acknowledgement
   I would like to acknowledge Tom Gilb and Kai Gilb.
   They teach me true agile and improvement
   process through Agile Inspection.
   copyright © A.Nagata 3
   

   View Slide

4. London, 1st - 3rd July 2014
   Root Cause Analysis
   • Root Cause Analysis(RCA) is useful tool to
   improve software quality.
   • Using for not only defect prevention but also it
   shall contribute important information for test
   approach and test design.
   • We were using Five Whys, one of the popular
   method of RCA
   copyright © A.Nagata 4
   

   View Slide

5. London, 1st - 3rd July 2014
   5 Whys analysis template
   Why ① Why 2 Why 3 Why 4 Why 5
   References
   Symptoms
   ＩＤ Issue
   Cause of defect installation
   Root Cuase
   . .
   copyright © A.Nagata 5
   

   View Slide

6. London, 1st - 3rd July 2014
   Problem of RCA for software development
   • Time consuming
   • Stressful approach to involve human
   factor.
   • Slow improvement in skill of analysis
   copyright © A.Nagata 6
   

   View Slide

7. London, 1st - 3rd July 2014
   Time consuming
   • Preparation
   • Document of explanation
   • Analysis
   • RCA review meeting with 5Why template
   .
   • Efficiency
   • We could not finish only one time.
   copyright © A.Nagata 7
   More than 2 hours
   2 hours
   More than once
   More than 8 hours
   http://info.legalzoom.com/
   documents-required-
   preparation-4187.html
   

   View Slide

8. London, 1st - 3rd July 2014
   Stressful approach to involve human factor
   • Why is the strongest question that might
   attack the people who made mistakes.
   copyright © A.Nagata 8
   The Art and Architecture of Powerful Questions, Eric E Vogt, Juanita Brown and David Isaacs, 2003
   

   View Slide

9. London, 1st - 3rd July 2014
   Stressful approach to involve human factor
   • Why is the strongest question that might
   attack the people who made mistakes.
   copyright © A.Nagata 9
   The Art and Architecture of Powerful Questions, Eric E Vogt, Juanita Brown and David Isaacs, 2003
   

   View Slide

10. London, 1st - 3rd July 2014
    Stressful approach to involve human factor
    • Why is the strongest question that might
    attack the people who made mistakes.
    Why did you do this?
    • How do you feel ?
    copyright © A.Nagata 10
    

    View Slide

11. London, 1st - 3rd July 2014 copyright © A.Nagata 11
    https://www.flickr.com/photos/ruthhb/2917888819/in/photostream/
    

    View Slide

12. London, 1st - 3rd July 2014
    5 Whys : Strong Questions
    copyright © A.Nagata 12
    Original : in Manufacturing
    Objects : physical materials
    Software Development
    Defects : physical problems
    Objects : Engineers
    Defects : Bugs
    

    View Slide

13. London, 1st - 3rd July 2014
    Stressful approach to involve human factor
    • Why is the strongest question that might
    attack the people who made mistakes.
    • Reaction of blaming,
    [Defensive mode]
    • Condition {Poor facilitation &
    Critical Thinking}
    copyright © A.Nagata 13
    People try to justify their answer
    rather than proceed in a spirit of
    inquiry.
    Eric E. Vogt said
    

    View Slide

14. London, 1st - 3rd July 2014 copyright © A.Nagata 14
    https://www.flickr.com/photos/ruthhb/2917888819/in/photostream/
    

    View Slide

15. London, 1st - 3rd July 2014
    Not root
    cause
    Answer
    Answer
    Defense mode in 5 Whys
    Answer
    Incident
    Root
    Cause
    Why
    Why
    Effective
    Prevention
    copyright © A.Nagata 15
    

    View Slide

16. London, 1st - 3rd July 2014
    Appropriate Questions
    Strong Questions
    Appropriate Questions
    What are Appropriate Questions?
    copyright © A.Nagata 16
    

    View Slide

17. London, 1st - 3rd July 2014
    Five specify questions
    1st Question 2nd Question 3rd Question 4th Question 5th Question
    Question
    Answers
    ID
    Question
    Answers
    ID
    Question
    Answers
    ID
    Question
    Answers
    ID
    Question
    Answers
    ID
    N/A
    ID
    Question
    Answers
    ID
    N/A
    ID
    Question
    Answers
    ID
    N/A
    ID
    copyright © A.Nagata 17
    

    View Slide

18. London, 1st - 3rd July 2014
    Slow improvement in analysis skill
    • Time consuming
    We could not RCA many times.
    • It is difficult to consider Appropriate Questions
    We could not reach the root cause.
    We need to practice of analysis and facilitation to
    improve RCA skill.
    copyright © A.Nagata 18
    

    View Slide

19. London, 1st - 3rd July 2014
    Iterative RCA
    • Light weight time box
    • 15 minutes through 30 minutes per each iteration
    • Make a habit and a rhythm
    • Doing every day as possible
    • Same time every day
    • After the stand up meeting
    • 15 minutes to lunch time.
    • Review the answer and make next question
    • The reviewer of RCA could discuss with people who
    made answer in the time box.
    • The reviewer could make appropriate questions.
    copyright © A.Nagata 19
    

    View Slide

20. London, 1st - 3rd July 2014
    Iterative RCA
    1st Iteration 2nd Iteration 3rd Iteration 4th Question
    Question
    Answers
    ID
    Question
    Answers
    ID
    Question
    Answers
    ID
    Question
    Answers
    ID
    Question
    Answers
    ID
    Question
    Answers
    ID
    Question
    Answers
    ID
    copyright © A.Nagata 20
    

    View Slide

21. London, 1st - 3rd July 2014
    Effective loop of the Iterative RCA
    Good
    question
    Right
    direction
    Good
    answer
    Discuss
    with answer
    copyright © A.Nagata 21
    

    View Slide

22. London, 1st - 3rd July 2014
    Answer
    Iterations make loops of feedback
    Incident
    Root
    Cause
    Effective
    Prevention
    Answer
    Q&A
    Discuss
    Q&A
    Discuss
    Q&A
    Discuss
    Q&A
    Discuss
    Q&A
    Discuss
    Learning
    Awareness
    copyright © A.Nagata 22
    

    View Slide

23. London, 1st - 3rd July 2014
    Problems of Iterative RCA
    • Sometimes it is difficult to find the next question.
    • The mechanism of problem is complicated.
    • A defect of software would be caused by many factors
    • Each factor is related and influenced with each other.
    • The model of the defect should be network.
    • The property of each factor should be classified and
    described in the model.
    • The RCA template is not network but tree list of
    question and answer.
    • Necessity for network model of Defects
    copyright © A.Nagata 23
    

    View Slide

24. London, 1st - 3rd July 2014
    RCA template : Tree structure
    1st Question 2nd Question 3rd Question 4th Question 5th Question
    Question
    Answers
    ID
    Question
    Answers
    ID
    Question
    Answers
    ID
    Question
    Answers
    ID
    Question
    Answers
    ID
    N/A
    ID
    Question
    Answers
    ID
    N/A
    ID
    Question
    Answers
    ID
    N/A
    ID
    copyright © A.Nagata 24
    

    View Slide

25. London, 1st - 3rd July 2014
    Necessity for the defect model
    • Factors of defect model
    • Incident
    • Defect
    • Amplifier : accelerate chain reaction to
    defects
    • Negligence Factor : Mistakes at human activities
    • Induction Trigger : Trigger to produce defect
    Nobuhiro Hosokawa, Yasuharu Nishi, Aya Ureshino, Makoto Nonaka, Yukiko Hara, JaSST
    2013 Tokyo – Project Fabre, 2013
    copyright © A.Nagata 25
    

    View Slide

26. London, 1st - 3rd July 2014
    Example of Defect Model
    Nobuhiro Hosokawa, Yasuharu Nishi, Aya Ureshino, Makoto Nonaka, Yukiko Hara, JaSST
    2013 Tokyo – Project Fabre, 2013
    
    Manner to use
    “Double” data style
    < Negligence Factor >
    Select style of data
    without understanding
    
    Did not select
    “Currency”
    but “Double”
    as data style
    
    Rounding error in
    currency data
    
    Checking
    system is
    week
    
    No
    standard
    rule of style
    copyright © A.Nagata 26
    

    View Slide

27. London, 1st - 3rd July 2014
    Answer
    Iterations make loops of feedback
    Incident
    Root
    Cause
    Effective
    Prevention
    Answer
    Q&A
    Discuss
    Q&A
    Discuss
    Q&A
    Discuss
    Q&A
    Discuss
    Q&A
    Discuss
    copyright © A.Nagata 27
    

    View Slide

28. London, 1st - 3rd July 2014
    Answer
    RCA Iterations with Modeling
    Incident
    Root
    Cause
    Effective
    Prevention
    Answer
    Q&A
    Discuss
    Q&A
    Discuss
    Q&A
    Discuss
    Q&A
    Discuss
    Q&A
    Discuss
    Modeling
    Modeling
    Modeling
    Modeling
    copyright © A.Nagata
    Q&A Discuss : Divergence : concrete
    Modeling : Convergence : abstract
    28
    

    View Slide

29. London, 1st - 3rd July 2014
    Process of Agile Root Cause Analysis
    (ARCA)
    copyright © A.Nagata 29
    

    View Slide

30. London, 1st - 3rd July 2014
    Three loop of ARCA
    • Incident Analysis loop
    • To Identify the problem and clarify the mechanism to
    cause the problem.
    • Exploratory Loop
    • Facilitator and reviewer make some questions to
    explore the defect and other factors with indication
    and conjecture.
    • It should be finished in the time box.
    • Agile RCA Loop
    • After the time box, facilitator shall analyze it and make
    modeling, identify root cause and/or defects. Also
    work out the next questions with strategy.
    copyright © A.Nagata 30
    

    View Slide

31. London, 1st - 3rd July 2014
    ARCA process : analysis team
    • Moderator
    • The Moderator should be existed who should
    understand the principles and process of ARCA.
    • Participants ( Person concerned)
    • Reviewers
    copyright © A.Nagata 31
    

    View Slide

32. London, 1st - 3rd July 2014
    Iteration Day1 : Incident Analysis
    The moderator shall make questions to the participants to make
    clear and share the incident, what was happened. The answers
    should be analyzed to understand the facts of the problem as possible.
    Paraphrasing is work very well.
    copyright © A.Nagata 32
    

    View Slide

33. London, 1st - 3rd July 2014
    Iteration Day 2 : Questions from Incident
    Agile RCA meeting with participants and reviewers in 15min through 30 min.
    The moderator and reviewers ask exploratory questions.
    copyright © A.Nagata 33
    

    View Slide

34. London, 1st - 3rd July 2014
    After the exploratory loop : Modeling
    Moderator makes defect model and consider the next question with the strategy.
    copyright © A.Nagata 34
    

    View Slide

35. London, 1st - 3rd July 2014
    Iteration Day 3 : Questions by
    Defect Model
    1. Make agreement of modeling with participants and reviewers
    2. The moderator and reviewers ask exploratory questions.
    Agile RCA meeting with participants and reviewers in 15min through 30 min.
    copyright © A.Nagata 35
    

    View Slide

36. London, 1st - 3rd July 2014
    After the exploratory loop : Modeling
    Moderator would prune the model with abstraction.
    copyright © A.Nagata 36
    

    View Slide

37. London, 1st - 3rd July 2014
    Process of Agile RCA with Prevention
    copyright © A.Nagata 37
    Divergence
    Convergence
    

    View Slide

38. London, 1st - 3rd July 2014
    ARCA Case: Day 1
    copyright © A.Nagata 38
    

    View Slide

39. London, 1st - 3rd July 2014
    ARCA Case: Day 2
    copyright © A.Nagata 39
    

    View Slide

40. London, 1st - 3rd July 2014
    ARCA Case: Day 3
    copyright © A.Nagata 40
    

    View Slide

41. London, 1st - 3rd July 2014
    Tacit Knowledge
    41
    Tacit
    Knowledge
    

    View Slide

42. London, 1st - 3rd July 2014
    Communication using Tacit Knowledge
    42
    Tacit
    Knowledge
    

    View Slide

43. London, 1st - 3rd July 2014
    Unexpected Missing Tacit Knowledge
    43
    Tacit
    Knowledge
    

    View Slide

44. London, 1st - 3rd July 2014
    Supporting Tacit Knowledge
    44
    Tacit
    Knowledge
    Domain
    Knowledge
    Context
    

    View Slide

45. London, 1st - 3rd July 2014
    Policy Supports Tacit Knowledge
    45
    Tacit
    Knowledge
    Domain
    Knowledge
    Context
    

    View Slide

46. London, 1st - 3rd July 2014
    ARCA Case: last Day
    copyright © A.Nagata 46
    

    View Slide

47. London, 1st - 3rd July 2014
    Case 2
    copyright © A.Nagata 47
    

    View Slide

48. London, 1st - 3rd July 2014
    Case 2
    copyright © A.Nagata 48
    Condition in Personal
    Condition in Process
    

    View Slide

49. London, 1st - 3rd July 2014
    Case 3
    copyright © A.Nagata 49
    

    View Slide

50. London, 1st - 3rd July 2014
    Boiling Frog
    In Agile development the code is small and
    working at first.
    copyright © A.Nagata 50
    • Modification incrementally
    • Increase complexity
    • Unexpected influences
    • Implant modules from other programs
    • Unexpected side effects and reactions
    • The changing is small, we could not have a
    motivation to react for it.
    

    View Slide

51. London, 1st - 3rd July 2014
    Boiled Frog
    • Complicated code
    • Become Legacy code
    • Degrade
    • Unteachable code
    copyright © A.Nagata 51
    

    View Slide

52. London, 1st - 3rd July 2014
    Effective of ARCA 1
    • Continuity
    • Before
    • Due to too slow and heavy work, almost teams could not
    keep RCA.
    • After
    • Two teams continue ARCA more than six month.
    • The members of the team have become habituated to doing
    Agile RCA everyday.
    copyright © A.Nagata 52
    

    View Slide

53. London, 1st - 3rd July 2014
    Effective of ARCA 2
    • Improve Analysis skill
    • Speed of analysis
    • More than four times faster
    • Before Total 8 hours more
    • After Total 2 hours
    • Reach to root cause
    • Before Some times we could not reach to root cause.
    • After About one and half hour, we could reach.
    copyright © A.Nagata 53
    

    View Slide

54. London, 1st - 3rd July 2014
    Effective of ARCA 3
    • Improve skill of the facilitation and analysis of
    RCA because of the continuity long time.
    • Making the trust relationship with teams.
    • Defect model becomes object of RCA.
    • Respect
    • Understand the difficulty to do correctly right from the
    beginning
    • Improve skill of identify problems : 4 times and
    more
    • defects and related factors.
    • Improve the motivation for quality improvement.
    copyright © A.Nagata 54
    

    View Slide

55. London, 1st - 3rd July 2014
    Cool-headed Analysis
    Before
    Objects : Engineers
    After
    Defects : Bugs
    Objects : Defect Model
    Defects : Bugs
    copyright © A.Nagata 55
    

    View Slide

56. London, 1st - 3rd July 2014
    Results of Agile RCA
    copyright © A.Nagata 56
    Number of Issues : The number of issues (problems to analysis) in six month
    Number of iteration : Average of number of iterations in issue.
    Number of Defects : Average of number of defects that were found in issue.
    Number of Factors : Average of number of factors that were found in issue.
    Total Times : Average of total time in issue.
    Number of
    Issues
    Number of
    Iterations
    Number of
    Defects
    Number of
    Factors
    Total time
    (minutes)
    Team A 6 4 2 15 103
    Team B 7 4 4 21 133
    

    View Slide

57. London, 1st - 3rd July 2014
    The reason of agile
    • Improvement loop
    • The outputs from Agile RCA, Defect model is
    progressing every iteration.
    • Evaluate Defect model to get feedback
    • Agree Defect model
    • Consider and discuss on Defect model
    • Revise Defect mode to the next stage.
    .
    • Agile RCA will contribute contiguous
    improvement of root cause analysis activities
    using iterative multiple loops.
    copyright © A.Nagata 57
    

    View Slide

58. London, 1st - 3rd July 2014
    Answer
    Agile RCA process
    Incident
    Root
    Cause
    Effective
    Prevention
    Answer
    Q&A
    Discuss
    Q&A
    Discuss
    Q&A
    Discuss
    Q&A
    Discuss
    Q&A
    Discuss
    Modeling
    Modeling
    Modeling
    Modeling
    copyright © A.Nagata
    Q&A Discuss : Divergence : concrete
    Modeling : Convergence : abstract
    58
    

    View Slide

59. London, 1st - 3rd July 2014
    Process of Agile RCA with Prevention
    copyright © A.Nagata 59
    Divergence
    Convergence
    

    View Slide

60. London, 1st - 3rd July 2014
    Issues of ARCA
    • Improve the defect modeling
    • Increase the factors
    • Speed up the modeling
    • Consider the pattern of modeling
    • Making the database of modeling
    • Improve the strategy to create the question.
    • Speed up the ARCA to real time solution.
    • Using not only defect prevention but also fixing the
    problem at development and debugging.
    copyright © A.Nagata 60
    

    View Slide

61. London, 1st - 3rd July 2014
    Thank you very much for your attention
    copyright © A.Nagata
    Agile RCA drives the actions for
    improvement of software quality.
    61
    

    View Slide

62. London, 1st - 3rd July 2014
    Reference
    • Agile Specification Quality Control: Shifting emphasis from cleanup to
    sampling defects, Tom Gilb, 2005
    • Nobuhiro Hosokawa, Yasuharu Nishi, Aya Ureshino, Makoto Nonaka,
    Yukiko Hara, JaSST 2013 Tokyo – Project Fabre, 2013
    • THE ART OF POWERFUL QUESTIONS, Eric E.Vogt, Juanita Brown, and
    David Isaacs, 2003
    • Software Inspection, Tom Gilb, Dorothy Graham, 1993
    • The Practical Guide to Defect Prevention, Marc McDonald, Robert
    Musson, Ross Smith, 2007, Microsoft Press
    • ODC - a 10x for Root Cause Analysis, Ram Chillarege, 2006
    • Naze Naze Bunseki, Hitoshi Ogura, Nikkei BP, 2010
    copyright © A.Nagata 62
    

    View Slide