Using Simulation to Manage Software Delivery Risk

Effective Modeling and Simulating Kanban and Scrum Projects using Monte Carlo Techniques presented at the Lean Software and Systems Conference. For more information see FocusedObjective.com

Transcript

1. Using Simulation to Manage Software Delivery Risk Effective Modeling and Simulating Kanban and Scrum Projects using Monte Carlo Techniques Troy Magennis Troy.magennis@focusedobjective.com @AgileSimulation
2. Are we meeting our commitments? Schedule Risk
3. What team skillset additions (or losses) have the biggest impact? Staff Risk
4. What are the top three risks jeopardizing delivery? Manageable Risk
5. Arm my teams (and yours) with the tools and techniques to answer these questions My Mission And manage risk more effectively
6. • Currently: Founder and CTO Focused Objective • Previously: Vice President of Technology (Arch) • Travelocity and Lastminute.com • Director Architecture, Corbis • Various: Automotive, Banking Contact: @AgileSimulation and @t_magennis FocusedObjective.com Troy.Magennis@focusedobjective.com
8. DEFINITIONS, HISTORY AND USE What, when, who, why
9. A model is a tool used to mimic a real world process A tool for low-cost experimentation Definition: Model
10. A technique of using a model to determine a result given a set of input conditions Definition: Simulation
11. Performing a simulation of a model multiple times using random input conditions and recording the frequency of each result occurrence Monte Carlo Simulation
12. DEMO: VISUAL MODEL SIMULATION DEMO: MONTE-CARLO SIMULATION Simple to more complex model and simulation of a software project In case of demo disaster, press here…
13. History Stan Ulam Holding the FERMIAC Credits: Wikipedia
14. When there is no answer (knowable in advance) or when the time/effort taken to compute an answer is beyond realistic When to use Monte Carlo Simulation
15. When a range of input conditions can MASSIVELY alter the final outcome When to use Monte Carlo Simulation
17. High risk industries Natural resource exploration, insurance, finance, banking, pharmaceutical… Who Uses Monte Carlo Simulation Software Development == High Risk! Just look at our reputation, and on-time, on-budget success rate…
18. APPLYING MONTE CARLO METHODS TO SOFTWARE DEV
19. Why? To Answer Tough Questions… Date and cost forecasts Impact of staff hire/loss Cost of defects Cost of blocking events … And my three 1:1 questions each week!
20. But doesn’t it require estimates? Yes, but very few… MUST: Estimate major risks SHOULD: Column cycle-times and story counts
21. We need to estimate risk events We spend all our time estimating here 1 2 3 **Major risk events have the predominate role in deciding where deliver actually occurs **
22. Is it Accurate? 1. Gin still equals Gout 2. Doesn’t suffer from the “Flaw of Averages”
23. Flaw of Averages 50% Possible Outcomes 50% Possible Outcomes
24. Frequency of Result 1 5 10 15 20 25 Major issue: Race condition, third party component failure… The average release date!!! Major Risk Event Shifts Delivery Shape Right Developer Estimates
25. We need to estimate risk events See model example… We spend all our time estimating here 1 2 3 **Major risk events have the predominate role in deciding where deliver actually occurs **
26. Risk likelihood changes constantly 1 2 3 95th Confidence Interval
27. Risk likelihood changes constantly 1 2 3 95th Confidence Interval
28. Risk likelihood changes constantly 1 2 3 95th Confidence Interval
29. Risk likelihood changes constantly 1 2 3 95th Confidence Interval
30. DEMO: FORECASTING (DATES & COST) DEMO: SENSITIVITY (COST OF DEFECTS) DEMO: STAFF IMPACT (STAFF RISK) In case of demo disaster or no internet, press here…
31. BEST PRACTICES AND TIPS
32. Model (a little) Visually Test Monte- Carlo Test Sensitivity Test The Model Creation Cycle
33. Baseline Make Single Change Compare Results Make Informed Decision(s) The Experiment Cycle
34. Best Practice 1 Start simple and add ONE input condition at a time. Visually / Monte-carlo test each input to verify it works
35. Best Practice 2 Find the likelihood of major events and estimate delay E.g. vendor dependencies, performance/memory issues, third party component failures.
36. Best Practice 3 Only obtain and add detailed estimates and opinion to a model if Sensitivity Analysis says that input is material
37. Best Practice 4 Use a uniform random input distribution UNTIL sensitivity analysis says that input is influencing the output
38. Best Practice 5 Educate your managers’ about risk. They will still want a “single” date for planning, but let them decide 75th or 95th confidence level (average is NEVER an option)
39. Q1. Are we meeting our commitments? Q2. What are the top three risks jeopardizing on-time delivery? Q3. What skillsets do your next three hires need to have? Is the likelihood of the models forecast date increasing or decreasing? Top three items in the Sensitivity report Skills applicable to the top three WIP limit increases that cause the biggest reduction in forecast
40. Call to action • Read these books • Download the software FocusedObjective.com • Follow @AgileSimulation • Learn: http://strategicdecisions.stanford.edu/
41. Questions? My Contact Details and to get these slides, the software or the book used in this session - FocusedObjective.com Me: Troy.magennis@FocusedObjective.com Follow: @AgileSimulation and @t_magennis
43. BASICS OF MODELING AND SIMULATION Return to main presentation…
44. Manual Kanban Model & Simulation Backlog Design Develop Test Deployed 2 1 – 2 days 1 – 2 days 1 – 2 days 1 2 3 4 PLUS: For this manual example, at least 1 defect, blocking event and scope-creep item. 5
45. Day 1 Backlog Design Develop Test Deployed 2 1 – 2 days 1 – 5 days 1 – 2 days 1 Day picked at random for this columns cycle- time range
46. Day 2 Backlog Design Develop Test Deployed 2 1 day 1 – 2 days 1 – 5 days 1 – 2 days
47. Day 3 Backlog Design Develop Test Deployed 2 1 day 1 – 2 days 1 – 5 days 1 – 2 days
48. Day 4 Backlog Design Develop Test Deployed 2 1 day 1 – 2 days 1 – 5 days 1 – 2 days Added Scope
49. Day 5 Backlog Design Develop Test Deployed 2 1 day 1 – 2 days 1 – 5 days 1 – 2 days Added Scope
50. Day 6 Backlog Design Develop Test Deployed 2 1 day 1 – 2 days 1 – 5 days 1 – 2 days Added Scope
51. Day 7 Backlog Design Develop Test Deployed 2 1 day 1 – 2 days 1 – 5 days 1 – 2 days Added Scope
52. Day 8 Backlog Design Develop Test Deployed 2 1 day 1 – 2 days 1 – 5 days 1 – 2 days Added Scope
53. Day 9 Backlog Design Develop Test Deployed 2 1 – 2 days 1 – 5 days 1 – 2 days Added Scope
54. Day 10 Backlog Design Develop Test Deployed 2 1 – 2 days 1 – 5 days 1 – 2 days Added Scope
55. Day 11 Backlog Design Develop Test Deployed 2 1 – 2 days 1 – 5 days 1 – 2 days Added Scope
56. Result versus Frequency (50 runs) More Often Less Often Result Values – For example, Days 15 10 20 Frequency of Result 1 5 10 15 20 25
57. Result versus Frequency (250 runs) More Often Less Often Result Values – For example, Days 15 10 20 Frequency of Result 1 5 10 15 20 25
58. Result versus Frequency (1000+ runs) More Often Less Often Result Values – For example, Days 15 10 60 Frequency of Result 1 5 10 15 20 25
59. Central Limit Theorum Return to main presentation…
60. Flaw of Averages 50% Possible Outcomes 50% Possible Outcomes Return to main presentation…
61. Software Development Model Model Columns & WIP Work Defects Blocking Events Added Work Staff Vacations … 1 2 3 4 5 6 7 Return to main presentation…
62. Return to main presentation…
63. SIMULATION EXAMPLES Return to main presentation…
64. unlikely certain Forecasts Return to main presentation…
65. unlikely certain Forecasts 50% Possible Outcomes 50% Possible Outcomes Return to main presentation…
66. Actively Manage Ignore for the moment Sensitivity Report Return to main presentation…
67. Staff Skill Impact Report Explore what staff changes have the greatest impact Return to main presentation…
68. Return to main presentation…
69. MULTI-MODAL RESULT MODEL
70. Return to main presentation…
71. <setup> <backlog type="custom" > <deliverable name=“work"> <custom count="10">Build website</custom> </deliverable> <deliverable name="performance issues, add caching" skipPercentage="50"> <custom count="10" >Rework: Performance Issues</custom> </deliverable> </backlog> <columns> <column id="1" estimateLowBound="1" estimateHighBound="5" wipLimit="1">Develop</column> <column id="2" estimateLowBound="1" estimateHighBound="5" wipLimit="1">Test</column> </columns> </setup> Return to main presentation…
72. Return to main presentation…