Improvement of Root cause analysis with iterative process

London, 1st - 3rd July 2014
Improvement of Root cause analysis
with iterative process
Implementation of Agile Root Cause Analysis
Atsushi Nagata
Sony Corporation
Nobuhiro Hosokawa
IBM

View Slide

Introduction
Sony Corporation
Quality Engineering Manager
Professional Division
Software Quality Assurance
Test Process Improvement
JSTQB Advanced Level Test Manager
copyright © A.Nagata 2

View Slide

Acknowledgement
I would like to acknowledge Tom Gilb and Kai Gilb.
They teach me true agile and improvement
process through Agile Inspection.

View Slide

Root Cause Analysis
• Root Cause Analysis(RCA) is useful tool to
improve software quality.
• Using for not only defect prevention but also it
shall contribute important information for test
approach and test design.
• We were using Five Whys, one of the popular
method of RCA

View Slide

5 Whys analysis template
Why ① Why 2 Why 3 Why 4 Why 5
References
Symptoms
ＩＤ Issue
Cause of defect installation
Root Cuase
. .

View Slide

Problem of RCA for software development
• Time consuming
• Stressful approach to involve human
factor.
• Slow improvement in skill of analysis

View Slide

Time consuming
• Preparation
• Document of explanation
• Analysis
• RCA review meeting with 5Why template
.
• Efficiency
• We could not finish only one time.
More than 2 hours
2 hours
More than once
More than 8 hours
http://info.legalzoom.com/
documents-required-
preparation-4187.html

View Slide

Stressful approach to involve human factor
• Why is the strongest question that might
attack the people who made mistakes.
The Art and Architecture of Powerful Questions, Eric E Vogt, Juanita Brown and David Isaacs, 2003

View Slide

The Art and Architecture of Powerful Questions, Eric E Vogt, Juanita Brown and David Isaacs, 2003

View Slide

Why did you do this?
• How do you feel ?

View Slide

London, 1st - 3rd July 2014 copyright © A.Nagata 11
https://www.flickr.com/photos/ruthhb/2917888819/in/photostream/

View Slide

5 Whys : Strong Questions
Original : in Manufacturing
Objects : physical materials
Software Development
Defects : physical problems
Objects : Engineers
Defects : Bugs

View Slide

• Reaction of blaming,
[Defensive mode]
• Condition {Poor facilitation &
Critical Thinking}
People try to justify their answer
rather than proceed in a spirit of
inquiry.
Eric E. Vogt said

View Slide

London, 1st - 3rd July 2014 copyright © A.Nagata 14
https://www.flickr.com/photos/ruthhb/2917888819/in/photostream/

View Slide

Not root
cause
Answer
Answer
Defense mode in 5 Whys
Answer
Incident
Root
Cause
Why
Why
Effective
Prevention

View Slide

Appropriate Questions
Strong Questions
Appropriate Questions
What are Appropriate Questions?

View Slide

Five specify questions
1st Question 2nd Question 3rd Question 4th Question 5th Question
Question
Answers
ID
Question
Answers
ID
Question
Answers
ID
Question
Answers
ID
Question
Answers
ID
N/A
ID
Question
Answers
ID
N/A
ID
Question
Answers
ID
N/A
ID

View Slide

Slow improvement in analysis skill
• Time consuming
We could not RCA many times.
• It is difficult to consider Appropriate Questions
We could not reach the root cause.
We need to practice of analysis and facilitation to
improve RCA skill.

View Slide

Iterative RCA
• Light weight time box
• 15 minutes through 30 minutes per each iteration
• Make a habit and a rhythm
• Doing every day as possible
• Same time every day
• After the stand up meeting
• 15 minutes to lunch time.
• Review the answer and make next question
• The reviewer of RCA could discuss with people who
made answer in the time box.
• The reviewer could make appropriate questions.

View Slide

Iterative RCA
1st Iteration 2nd Iteration 3rd Iteration 4th Question
Question
Answers
ID
Question
Answers
ID
Question
Answers
ID
Question
Answers
ID
Question
Answers
ID
Question
Answers
ID
Question
Answers
ID

View Slide

Effective loop of the Iterative RCA
Good
question
Right
direction
Good
answer
Discuss
with answer

View Slide

Answer
Iterations make loops of feedback
Incident
Root
Cause
Effective
Prevention
Answer
Q&A
Discuss
Q&A
Discuss
Q&A
Discuss
Q&A
Discuss
Q&A
Discuss
Learning
Awareness

View Slide

Problems of Iterative RCA
• Sometimes it is difficult to find the next question.
• The mechanism of problem is complicated.
• A defect of software would be caused by many factors
• Each factor is related and influenced with each other.
• The model of the defect should be network.
• The property of each factor should be classified and
described in the model.
• The RCA template is not network but tree list of
question and answer.
• Necessity for network model of Defects

View Slide

RCA template : Tree structure
1st Question 2nd Question 3rd Question 4th Question 5th Question
Question
Answers
ID
Question
Answers
ID
Question
Answers
ID
Question
Answers
ID
Question
Answers
ID
N/A
ID
Question
Answers
ID
N/A
ID
Question
Answers
ID
N/A
ID

View Slide

Necessity for the defect model
• Factors of defect model
• Incident
• Defect
• Amplifier : accelerate chain reaction to
defects
• Negligence Factor : Mistakes at human activities
• Induction Trigger : Trigger to produce defect
Nobuhiro Hosokawa, Yasuharu Nishi, Aya Ureshino, Makoto Nonaka, Yukiko Hara, JaSST
2013 Tokyo – Project Fabre, 2013

View Slide

Example of Defect Model
Nobuhiro Hosokawa, Yasuharu Nishi, Aya Ureshino, Makoto Nonaka, Yukiko Hara, JaSST
2013 Tokyo – Project Fabre, 2013

Manner to use
“Double” data style
< Negligence Factor >
Select style of data
without understanding

Did not select
“Currency”
but “Double”
as data style

Rounding error in
currency data

Checking
system is
week

No
standard
rule of style

View Slide

Answer
Iterations make loops of feedback
Incident
Root
Cause
Effective
Prevention
Answer
Q&A
Discuss
Q&A
Discuss
Q&A
Discuss
Q&A
Discuss
Q&A
Discuss

View Slide

Answer
RCA Iterations with Modeling
Incident
Root
Cause
Effective
Prevention
Answer
Q&A
Discuss
Q&A
Discuss
Q&A
Discuss
Q&A
Discuss
Q&A
Discuss
Modeling
Modeling
Modeling
Modeling
copyright © A.Nagata
Q&A Discuss : Divergence : concrete
Modeling : Convergence : abstract
28

View Slide

Process of Agile Root Cause Analysis
(ARCA)

View Slide

Three loop of ARCA
• Incident Analysis loop
• To Identify the problem and clarify the mechanism to
cause the problem.
• Exploratory Loop
• Facilitator and reviewer make some questions to
explore the defect and other factors with indication
and conjecture.
• It should be finished in the time box.
• Agile RCA Loop
• After the time box, facilitator shall analyze it and make
modeling, identify root cause and/or defects. Also
work out the next questions with strategy.

View Slide

ARCA process : analysis team
• Moderator
• The Moderator should be existed who should
understand the principles and process of ARCA.
• Participants ( Person concerned)
• Reviewers

View Slide

Iteration Day1 : Incident Analysis
The moderator shall make questions to the participants to make
clear and share the incident, what was happened. The answers
should be analyzed to understand the facts of the problem as possible.
Paraphrasing is work very well.

View Slide

Iteration Day 2 : Questions from Incident
Agile RCA meeting with participants and reviewers in 15min through 30 min.
The moderator and reviewers ask exploratory questions.

View Slide

After the exploratory loop : Modeling
Moderator makes defect model and consider the next question with the strategy.

View Slide

Iteration Day 3 : Questions by
Defect Model
1. Make agreement of modeling with participants and reviewers
2. The moderator and reviewers ask exploratory questions.
Agile RCA meeting with participants and reviewers in 15min through 30 min.

View Slide

After the exploratory loop : Modeling
Moderator would prune the model with abstraction.

View Slide

Process of Agile RCA with Prevention
Divergence
Convergence

View Slide

ARCA Case: Day 1

View Slide

ARCA Case: Day 2

View Slide

ARCA Case: Day 3

View Slide

Tacit Knowledge
41
Tacit
Knowledge

View Slide

Communication using Tacit Knowledge
42
Tacit
Knowledge

View Slide

Unexpected Missing Tacit Knowledge
43
Tacit
Knowledge

View Slide

Supporting Tacit Knowledge
44
Tacit
Knowledge
Domain
Knowledge
Context

View Slide

Policy Supports Tacit Knowledge
45
Tacit
Knowledge
Domain
Knowledge
Context

View Slide

ARCA Case: last Day

View Slide

Case 2

View Slide

Case 2
Condition in Personal
Condition in Process

View Slide

Case 3

View Slide

Boiling Frog
In Agile development the code is small and
working at first.
• Modification incrementally
• Increase complexity
• Unexpected influences
• Implant modules from other programs
• Unexpected side effects and reactions
• The changing is small, we could not have a
motivation to react for it.

View Slide

Boiled Frog
• Complicated code
• Become Legacy code
• Degrade
• Unteachable code

View Slide

Effective of ARCA 1
• Continuity
• Before
• Due to too slow and heavy work, almost teams could not
keep RCA.
• After
• Two teams continue ARCA more than six month.
• The members of the team have become habituated to doing
Agile RCA everyday.

View Slide

Effective of ARCA 2
• Improve Analysis skill
• Speed of analysis
• More than four times faster
• Before Total 8 hours more
• After Total 2 hours
• Reach to root cause
• Before Some times we could not reach to root cause.
• After About one and half hour, we could reach.

View Slide

Effective of ARCA 3
• Improve skill of the facilitation and analysis of
RCA because of the continuity long time.
• Making the trust relationship with teams.
• Defect model becomes object of RCA.
• Respect
• Understand the difficulty to do correctly right from the
beginning
• Improve skill of identify problems : 4 times and
more
• defects and related factors.
• Improve the motivation for quality improvement.

View Slide

Cool-headed Analysis
Before
Objects : Engineers
After
Defects : Bugs
Objects : Defect Model
Defects : Bugs

View Slide

Results of Agile RCA
Number of Issues : The number of issues (problems to analysis) in six month
Number of iteration : Average of number of iterations in issue.
Number of Defects : Average of number of defects that were found in issue.
Number of Factors : Average of number of factors that were found in issue.
Total Times : Average of total time in issue.
Number of
Issues
Number of
Iterations
Number of
Defects
Number of
Factors
Total time
(minutes)
Team A 6 4 2 15 103
Team B 7 4 4 21 133

View Slide

The reason of agile
• Improvement loop
• The outputs from Agile RCA, Defect model is
progressing every iteration.
• Evaluate Defect model to get feedback
• Agree Defect model
• Consider and discuss on Defect model
• Revise Defect mode to the next stage.
.
• Agile RCA will contribute contiguous
improvement of root cause analysis activities
using iterative multiple loops.

View Slide

Answer
Agile RCA process
Incident
Root
Cause
Effective
Prevention
Answer
Q&A
Discuss
Q&A
Discuss
Q&A
Discuss
Q&A
Discuss
Q&A
Discuss
Modeling
Modeling
Modeling
Modeling
Q&A Discuss : Divergence : concrete
Modeling : Convergence : abstract
58

View Slide

Process of Agile RCA with Prevention
Divergence
Convergence

View Slide

Issues of ARCA
• Improve the defect modeling
• Increase the factors
• Speed up the modeling
• Consider the pattern of modeling
• Making the database of modeling
• Improve the strategy to create the question.
• Speed up the ARCA to real time solution.
• Using not only defect prevention but also fixing the
problem at development and debugging.

View Slide

Thank you very much for your attention
Agile RCA drives the actions for
improvement of software quality.
61

View Slide

Reference
• Agile Specification Quality Control: Shifting emphasis from cleanup to
sampling defects, Tom Gilb, 2005
• Nobuhiro Hosokawa, Yasuharu Nishi, Aya Ureshino, Makoto Nonaka,
Yukiko Hara, JaSST 2013 Tokyo – Project Fabre, 2013
• THE ART OF POWERFUL QUESTIONS, Eric E.Vogt, Juanita Brown, and
David Isaacs, 2003
• Software Inspection, Tom Gilb, Dorothy Graham, 1993
• The Practical Guide to Defect Prevention, Marc McDonald, Robert
Musson, Ross Smith, 2007, Microsoft Press
• ODC - a 10x for Root Cause Analysis, Ram Chillarege, 2006
• Naze Naze Bunseki, Hitoshi Ogura, Nikkei BP, 2010

View Slide

Improvement of Root cause analysis with iterative process

Improvement of Root cause analysis with iterative process

Atsushi Nagata

More Decks by Atsushi Nagata

Other Decks in Technology

Featured

Transcript