$30 off During Our Annual Pro Sale. View Details »

Improvement of Root cause analysis with iterative process

Improvement of Root cause analysis with iterative process

Root Cause Analysis method using the iterative process in order to improve speed and quality of the anaysis of defects in software systems.
This was presented at 6th World Congress of Software Quality in London.

Atsushi Nagata

May 27, 2022
Tweet

More Decks by Atsushi Nagata

Other Decks in Technology

Transcript

  1. London, 1st - 3rd July 2014
    Improvement of Root cause analysis
    with iterative process
    Implementation of Agile Root Cause Analysis
    Atsushi Nagata
    Sony Corporation
    Nobuhiro Hosokawa
    IBM

    View Slide

  2. London, 1st - 3rd July 2014
    Introduction
    Sony Corporation
    Quality Engineering Manager
    Professional Division
    Software Quality Assurance
    Test Process Improvement
    JSTQB Advanced Level Test Manager
    copyright © A.Nagata 2

    View Slide

  3. London, 1st - 3rd July 2014
    Acknowledgement
    I would like to acknowledge Tom Gilb and Kai Gilb.
    They teach me true agile and improvement
    process through Agile Inspection.
    copyright © A.Nagata 3

    View Slide

  4. London, 1st - 3rd July 2014
    Root Cause Analysis
    • Root Cause Analysis(RCA) is useful tool to
    improve software quality.
    • Using for not only defect prevention but also it
    shall contribute important information for test
    approach and test design.
    • We were using Five Whys, one of the popular
    method of RCA
    copyright © A.Nagata 4

    View Slide

  5. London, 1st - 3rd July 2014
    5 Whys analysis template
    Why ① Why 2 Why 3 Why 4 Why 5
    References
    Symptoms
    ID Issue
    Cause of defect installation
    Root Cuase
    . .
    copyright © A.Nagata 5

    View Slide

  6. London, 1st - 3rd July 2014
    Problem of RCA for software development
    • Time consuming
    • Stressful approach to involve human
    factor.
    • Slow improvement in skill of analysis
    copyright © A.Nagata 6

    View Slide

  7. London, 1st - 3rd July 2014
    Time consuming
    • Preparation
    • Document of explanation
    • Analysis
    • RCA review meeting with 5Why template
    .
    • Efficiency
    • We could not finish only one time.
    copyright © A.Nagata 7
    More than 2 hours
    2 hours
    More than once
    More than 8 hours
    http://info.legalzoom.com/
    documents-required-
    preparation-4187.html

    View Slide

  8. London, 1st - 3rd July 2014
    Stressful approach to involve human factor
    • Why is the strongest question that might
    attack the people who made mistakes.
    copyright © A.Nagata 8
    The Art and Architecture of Powerful Questions, Eric E Vogt, Juanita Brown and David Isaacs, 2003

    View Slide

  9. London, 1st - 3rd July 2014
    Stressful approach to involve human factor
    • Why is the strongest question that might
    attack the people who made mistakes.
    copyright © A.Nagata 9
    The Art and Architecture of Powerful Questions, Eric E Vogt, Juanita Brown and David Isaacs, 2003

    View Slide

  10. London, 1st - 3rd July 2014
    Stressful approach to involve human factor
    • Why is the strongest question that might
    attack the people who made mistakes.
    Why did you do this?
    • How do you feel ?
    copyright © A.Nagata 10

    View Slide

  11. London, 1st - 3rd July 2014 copyright © A.Nagata 11
    https://www.flickr.com/photos/ruthhb/2917888819/in/photostream/

    View Slide

  12. London, 1st - 3rd July 2014
    5 Whys : Strong Questions
    copyright © A.Nagata 12
    Original : in Manufacturing
    Objects : physical materials
    Software Development
    Defects : physical problems
    Objects : Engineers
    Defects : Bugs

    View Slide

  13. London, 1st - 3rd July 2014
    Stressful approach to involve human factor
    • Why is the strongest question that might
    attack the people who made mistakes.
    • Reaction of blaming,
    [Defensive mode]
    • Condition {Poor facilitation &
    Critical Thinking}
    copyright © A.Nagata 13
    People try to justify their answer
    rather than proceed in a spirit of
    inquiry.
    Eric E. Vogt said

    View Slide

  14. London, 1st - 3rd July 2014 copyright © A.Nagata 14
    https://www.flickr.com/photos/ruthhb/2917888819/in/photostream/

    View Slide

  15. London, 1st - 3rd July 2014
    Not root
    cause
    Answer
    Answer
    Defense mode in 5 Whys
    Answer
    Incident
    Root
    Cause
    Why
    Why
    Effective
    Prevention
    copyright © A.Nagata 15

    View Slide

  16. London, 1st - 3rd July 2014
    Appropriate Questions
    Strong Questions
    Appropriate Questions
    What are Appropriate Questions?
    copyright © A.Nagata 16

    View Slide

  17. London, 1st - 3rd July 2014
    Five specify questions
    1st Question 2nd Question 3rd Question 4th Question 5th Question
    Question
    Answers
    ID
    Question
    Answers
    ID
    Question
    Answers
    ID
    Question
    Answers
    ID
    Question
    Answers
    ID
    N/A
    ID
    Question
    Answers
    ID
    N/A
    ID
    Question
    Answers
    ID
    N/A
    ID
    copyright © A.Nagata 17

    View Slide

  18. London, 1st - 3rd July 2014
    Slow improvement in analysis skill
    • Time consuming
    We could not RCA many times.
    • It is difficult to consider Appropriate Questions
    We could not reach the root cause.
    We need to practice of analysis and facilitation to
    improve RCA skill.
    copyright © A.Nagata 18

    View Slide

  19. London, 1st - 3rd July 2014
    Iterative RCA
    • Light weight time box
    • 15 minutes through 30 minutes per each iteration
    • Make a habit and a rhythm
    • Doing every day as possible
    • Same time every day
    • After the stand up meeting
    • 15 minutes to lunch time.
    • Review the answer and make next question
    • The reviewer of RCA could discuss with people who
    made answer in the time box.
    • The reviewer could make appropriate questions.
    copyright © A.Nagata 19

    View Slide

  20. London, 1st - 3rd July 2014
    Iterative RCA
    1st Iteration 2nd Iteration 3rd Iteration 4th Question
    Question
    Answers
    ID
    Question
    Answers
    ID
    Question
    Answers
    ID
    Question
    Answers
    ID
    Question
    Answers
    ID
    Question
    Answers
    ID
    Question
    Answers
    ID
    copyright © A.Nagata 20

    View Slide

  21. London, 1st - 3rd July 2014
    Effective loop of the Iterative RCA
    Good
    question
    Right
    direction
    Good
    answer
    Discuss
    with answer
    copyright © A.Nagata 21

    View Slide

  22. London, 1st - 3rd July 2014
    Answer
    Iterations make loops of feedback
    Incident
    Root
    Cause
    Effective
    Prevention
    Answer
    Q&A
    Discuss
    Q&A
    Discuss
    Q&A
    Discuss
    Q&A
    Discuss
    Q&A
    Discuss
    Learning
    Awareness
    copyright © A.Nagata 22

    View Slide

  23. London, 1st - 3rd July 2014
    Problems of Iterative RCA
    • Sometimes it is difficult to find the next question.
    • The mechanism of problem is complicated.
    • A defect of software would be caused by many factors
    • Each factor is related and influenced with each other.
    • The model of the defect should be network.
    • The property of each factor should be classified and
    described in the model.
    • The RCA template is not network but tree list of
    question and answer.
    • Necessity for network model of Defects
    copyright © A.Nagata 23

    View Slide

  24. London, 1st - 3rd July 2014
    RCA template : Tree structure
    1st Question 2nd Question 3rd Question 4th Question 5th Question
    Question
    Answers
    ID
    Question
    Answers
    ID
    Question
    Answers
    ID
    Question
    Answers
    ID
    Question
    Answers
    ID
    N/A
    ID
    Question
    Answers
    ID
    N/A
    ID
    Question
    Answers
    ID
    N/A
    ID
    copyright © A.Nagata 24

    View Slide

  25. London, 1st - 3rd July 2014
    Necessity for the defect model
    • Factors of defect model
    • Incident
    • Defect
    • Amplifier : accelerate chain reaction to
    defects
    • Negligence Factor : Mistakes at human activities
    • Induction Trigger : Trigger to produce defect
    Nobuhiro Hosokawa, Yasuharu Nishi, Aya Ureshino, Makoto Nonaka, Yukiko Hara, JaSST
    2013 Tokyo – Project Fabre, 2013
    copyright © A.Nagata 25

    View Slide

  26. London, 1st - 3rd July 2014
    Example of Defect Model
    Nobuhiro Hosokawa, Yasuharu Nishi, Aya Ureshino, Makoto Nonaka, Yukiko Hara, JaSST
    2013 Tokyo – Project Fabre, 2013

    Manner to use
    “Double” data style
    < Negligence Factor >
    Select style of data
    without understanding

    Did not select
    “Currency”
    but “Double”
    as data style

    Rounding error in
    currency data

    Checking
    system is
    week

    No
    standard
    rule of style
    copyright © A.Nagata 26

    View Slide

  27. London, 1st - 3rd July 2014
    Answer
    Iterations make loops of feedback
    Incident
    Root
    Cause
    Effective
    Prevention
    Answer
    Q&A
    Discuss
    Q&A
    Discuss
    Q&A
    Discuss
    Q&A
    Discuss
    Q&A
    Discuss
    copyright © A.Nagata 27

    View Slide

  28. London, 1st - 3rd July 2014
    Answer
    RCA Iterations with Modeling
    Incident
    Root
    Cause
    Effective
    Prevention
    Answer
    Q&A
    Discuss
    Q&A
    Discuss
    Q&A
    Discuss
    Q&A
    Discuss
    Q&A
    Discuss
    Modeling
    Modeling
    Modeling
    Modeling
    copyright © A.Nagata
    Q&A Discuss : Divergence : concrete
    Modeling : Convergence : abstract
    28

    View Slide

  29. London, 1st - 3rd July 2014
    Process of Agile Root Cause Analysis
    (ARCA)
    copyright © A.Nagata 29

    View Slide

  30. London, 1st - 3rd July 2014
    Three loop of ARCA
    • Incident Analysis loop
    • To Identify the problem and clarify the mechanism to
    cause the problem.
    • Exploratory Loop
    • Facilitator and reviewer make some questions to
    explore the defect and other factors with indication
    and conjecture.
    • It should be finished in the time box.
    • Agile RCA Loop
    • After the time box, facilitator shall analyze it and make
    modeling, identify root cause and/or defects. Also
    work out the next questions with strategy.
    copyright © A.Nagata 30

    View Slide

  31. London, 1st - 3rd July 2014
    ARCA process : analysis team
    • Moderator
    • The Moderator should be existed who should
    understand the principles and process of ARCA.
    • Participants ( Person concerned)
    • Reviewers
    copyright © A.Nagata 31

    View Slide

  32. London, 1st - 3rd July 2014
    Iteration Day1 : Incident Analysis
    The moderator shall make questions to the participants to make
    clear and share the incident, what was happened. The answers
    should be analyzed to understand the facts of the problem as possible.
    Paraphrasing is work very well.
    copyright © A.Nagata 32

    View Slide

  33. London, 1st - 3rd July 2014
    Iteration Day 2 : Questions from Incident
    Agile RCA meeting with participants and reviewers in 15min through 30 min.
    The moderator and reviewers ask exploratory questions.
    copyright © A.Nagata 33

    View Slide

  34. London, 1st - 3rd July 2014
    After the exploratory loop : Modeling
    Moderator makes defect model and consider the next question with the strategy.
    copyright © A.Nagata 34

    View Slide

  35. London, 1st - 3rd July 2014
    Iteration Day 3 : Questions by
    Defect Model
    1. Make agreement of modeling with participants and reviewers
    2. The moderator and reviewers ask exploratory questions.
    Agile RCA meeting with participants and reviewers in 15min through 30 min.
    copyright © A.Nagata 35

    View Slide

  36. London, 1st - 3rd July 2014
    After the exploratory loop : Modeling
    Moderator would prune the model with abstraction.
    copyright © A.Nagata 36

    View Slide

  37. London, 1st - 3rd July 2014
    Process of Agile RCA with Prevention
    copyright © A.Nagata 37
    Divergence
    Convergence

    View Slide

  38. London, 1st - 3rd July 2014
    ARCA Case: Day 1
    copyright © A.Nagata 38

    View Slide

  39. London, 1st - 3rd July 2014
    ARCA Case: Day 2
    copyright © A.Nagata 39

    View Slide

  40. London, 1st - 3rd July 2014
    ARCA Case: Day 3
    copyright © A.Nagata 40

    View Slide

  41. London, 1st - 3rd July 2014
    Tacit Knowledge
    41
    Tacit
    Knowledge

    View Slide

  42. London, 1st - 3rd July 2014
    Communication using Tacit Knowledge
    42
    Tacit
    Knowledge

    View Slide

  43. London, 1st - 3rd July 2014
    Unexpected Missing Tacit Knowledge
    43
    Tacit
    Knowledge

    View Slide

  44. London, 1st - 3rd July 2014
    Supporting Tacit Knowledge
    44
    Tacit
    Knowledge
    Domain
    Knowledge
    Context

    View Slide

  45. London, 1st - 3rd July 2014
    Policy Supports Tacit Knowledge
    45
    Tacit
    Knowledge
    Domain
    Knowledge
    Context

    View Slide

  46. London, 1st - 3rd July 2014
    ARCA Case: last Day
    copyright © A.Nagata 46

    View Slide

  47. London, 1st - 3rd July 2014
    Case 2
    copyright © A.Nagata 47

    View Slide

  48. London, 1st - 3rd July 2014
    Case 2
    copyright © A.Nagata 48
    Condition in Personal
    Condition in Process

    View Slide

  49. London, 1st - 3rd July 2014
    Case 3
    copyright © A.Nagata 49

    View Slide

  50. London, 1st - 3rd July 2014
    Boiling Frog
    In Agile development the code is small and
    working at first.
    copyright © A.Nagata 50
    • Modification incrementally
    • Increase complexity
    • Unexpected influences
    • Implant modules from other programs
    • Unexpected side effects and reactions
    • The changing is small, we could not have a
    motivation to react for it.

    View Slide

  51. London, 1st - 3rd July 2014
    Boiled Frog
    • Complicated code
    • Become Legacy code
    • Degrade
    • Unteachable code
    copyright © A.Nagata 51

    View Slide

  52. London, 1st - 3rd July 2014
    Effective of ARCA 1
    • Continuity
    • Before
    • Due to too slow and heavy work, almost teams could not
    keep RCA.
    • After
    • Two teams continue ARCA more than six month.
    • The members of the team have become habituated to doing
    Agile RCA everyday.
    copyright © A.Nagata 52

    View Slide

  53. London, 1st - 3rd July 2014
    Effective of ARCA 2
    • Improve Analysis skill
    • Speed of analysis
    • More than four times faster
    • Before Total 8 hours more
    • After Total 2 hours
    • Reach to root cause
    • Before Some times we could not reach to root cause.
    • After About one and half hour, we could reach.
    copyright © A.Nagata 53

    View Slide

  54. London, 1st - 3rd July 2014
    Effective of ARCA 3
    • Improve skill of the facilitation and analysis of
    RCA because of the continuity long time.
    • Making the trust relationship with teams.
    • Defect model becomes object of RCA.
    • Respect
    • Understand the difficulty to do correctly right from the
    beginning
    • Improve skill of identify problems : 4 times and
    more
    • defects and related factors.
    • Improve the motivation for quality improvement.
    copyright © A.Nagata 54

    View Slide

  55. London, 1st - 3rd July 2014
    Cool-headed Analysis
    Before
    Objects : Engineers
    After
    Defects : Bugs
    Objects : Defect Model
    Defects : Bugs
    copyright © A.Nagata 55

    View Slide

  56. London, 1st - 3rd July 2014
    Results of Agile RCA
    copyright © A.Nagata 56
    Number of Issues : The number of issues (problems to analysis) in six month
    Number of iteration : Average of number of iterations in issue.
    Number of Defects : Average of number of defects that were found in issue.
    Number of Factors : Average of number of factors that were found in issue.
    Total Times : Average of total time in issue.
    Number of
    Issues
    Number of
    Iterations
    Number of
    Defects
    Number of
    Factors
    Total time
    (minutes)
    Team A 6 4 2 15 103
    Team B 7 4 4 21 133

    View Slide

  57. London, 1st - 3rd July 2014
    The reason of agile
    • Improvement loop
    • The outputs from Agile RCA, Defect model is
    progressing every iteration.
    • Evaluate Defect model to get feedback
    • Agree Defect model
    • Consider and discuss on Defect model
    • Revise Defect mode to the next stage.
    .
    • Agile RCA will contribute contiguous
    improvement of root cause analysis activities
    using iterative multiple loops.
    copyright © A.Nagata 57

    View Slide

  58. London, 1st - 3rd July 2014
    Answer
    Agile RCA process
    Incident
    Root
    Cause
    Effective
    Prevention
    Answer
    Q&A
    Discuss
    Q&A
    Discuss
    Q&A
    Discuss
    Q&A
    Discuss
    Q&A
    Discuss
    Modeling
    Modeling
    Modeling
    Modeling
    copyright © A.Nagata
    Q&A Discuss : Divergence : concrete
    Modeling : Convergence : abstract
    58

    View Slide

  59. London, 1st - 3rd July 2014
    Process of Agile RCA with Prevention
    copyright © A.Nagata 59
    Divergence
    Convergence

    View Slide

  60. London, 1st - 3rd July 2014
    Issues of ARCA
    • Improve the defect modeling
    • Increase the factors
    • Speed up the modeling
    • Consider the pattern of modeling
    • Making the database of modeling
    • Improve the strategy to create the question.
    • Speed up the ARCA to real time solution.
    • Using not only defect prevention but also fixing the
    problem at development and debugging.
    copyright © A.Nagata 60

    View Slide

  61. London, 1st - 3rd July 2014
    Thank you very much for your attention
    copyright © A.Nagata
    Agile RCA drives the actions for
    improvement of software quality.
    61

    View Slide

  62. London, 1st - 3rd July 2014
    Reference
    • Agile Specification Quality Control: Shifting emphasis from cleanup to
    sampling defects, Tom Gilb, 2005
    • Nobuhiro Hosokawa, Yasuharu Nishi, Aya Ureshino, Makoto Nonaka,
    Yukiko Hara, JaSST 2013 Tokyo – Project Fabre, 2013
    • THE ART OF POWERFUL QUESTIONS, Eric E.Vogt, Juanita Brown, and
    David Isaacs, 2003
    • Software Inspection, Tom Gilb, Dorothy Graham, 1993
    • The Practical Guide to Defect Prevention, Marc McDonald, Robert
    Musson, Ross Smith, 2007, Microsoft Press
    • ODC - a 10x for Root Cause Analysis, Ram Chillarege, 2006
    • Naze Naze Bunseki, Hitoshi Ogura, Nikkei BP, 2010
    copyright © A.Nagata 62

    View Slide