Coaching in a data-driven world (#agilecitylon)

Slides from the talk I gave at Agile In The City:London on using metrics as a coaching aid for teams, to model curiosity and ask better questions.

Transcript

1. Coaching in a data-driven world Nicolas Brown @nbrown02 #agilecitylon 1
2. TLDR; Stop using metrics as targets. Use metrics as a coaching aid to inform, to have safe conversations, open questions and model curiosity.
3. Google Ngram
4. The growth in metric obsession
5. The growth in metric obsession
6. Which of the following best describes decision-making in your organisation? 8% Rarely data-driven Somewhat data-driven Highly data-driven 53% 39% PwC's Global Data and Analytics Survey: Big Decisions TM. Base: 2,106 senior executives 6
7. “When a measure becomes a target, it ceases to be a good measure” - Charles Goodhart
10. Commissioner Ervin Burrell Deputy Commissioner Bill Rawls
11. Police Majors & Lieutenants
13. No more murders. You will reduce the felony rate by 5% or more or…. let no man come back alive
14. Anyone of you who can’t bring in the numbers we need will be replaced by someone who can
15. Major Bunny Colvin
17. “Hamsterdam”
18. We’ve had a 14% drop in felony rate over the last 4 weeks
19. Ah yes, now I am happy. You won’t mind if I check your numbers will you?
24. The tyranny of metrics • Penalizing hospitals based on the percentage of patients who fail to survive for thirty days beyond surgery Patients sometimes kept alive for thirty-one days, so that their mortality is not reflect in the hospitals metrics • The National Health Service (NHS) decided that a major problem was that people were having to wait too long to be admitted to emergency wards, they declared that hospitals would be evaluated based on to what extent patients were admitted within four hours Some hospitals responded by keeping incoming patients in queues of ambulances outside the hospital doors, until staff were confident that the patient could be seen within the allotted four hours of being admitted • No Child Left Behind (NCLB) legislation measured students grade 3-8 each year in math, reading and science. Aiming to bring all students to academic proficiency whilst penalties and sanctions for schools when students did not progress Teachers diverted class time to teaching the subjects tested, neglecting other subjects as well as reclassifying weaker students as disabled (thus removing from the assessment pool), altering results, reducing test difficulty or lowering the grades needed to pass them
25. The tyranny of velocity?
26. The tyranny of velocity?
27. The tyranny of velocity?
28. The tyranny of velocity?
29. Police Majors & Lieutenants Agile Project Managers/ Scrum Masters
30. We’ve got to maximise our points!
31. You asked to improve velocity and I have delivered. We are now agile.
32. We are so friggin agile...
34. So long as I have increased velocity, none of this will matter Hamsterdam Technical Debt
36. Two-thirds of the population are visual-spatial learners
37. Agile experience in relation to hair Date 2017 2018 2016 2015 2014 ? Volume (hair) ? 2013
38. Our Agile roots... Transparency Inspection Adaptation Scrum
39. Data-based coaching uses data visualisation as a basis for safe conversations and open questions, as well as facilitating more informed, transparent decision making whilst staying true to our Agile roots...
40. Coaching in a data-driven world Descriptive reporting
41. WIP Items Per Week What happened? What is happening?
42. Scatter Plot What happened?
43. Scatter Plot What is happening?
44. Data-based retrospectives Let’s take anything above 10 days - what happened?
45. Scatter Plot Retrospectives Let’s take anything above 10 days - what happened? Data not available (improve DoR check) - Better BDD needed - Work as a team when it’s not working - Too long for PO feedback (20 days!) - Better DoR check needed as was not met PO Feedback Data not available (improve DoR check) Rework due to tech debt - Better DoR check needed - Flowed quickly once sat with the SME Delayed due to lack of clarity on deployment
46. Scatter Plot Retrospectives What is happening? Process Policies Process Policies Process Policies Continuous attention to technical excellence Process Policies Process Policies Feedback loops Feedback loops Feedback loops Feedback loops Process Policies
47. Net Flow per Week Do we have flow? Stop starting, start finishing?
48. Coaching in a data-driven world Descriptive reporting Questions What happened? What is happening? Visuals Diagnostic discover & explore - WIP Items per Week - Scatter Plot - Net flow per week - Throughput History Trend
49. Stuck Work Where is the problem? What are the trends? Why is it happening?
50. Estimate Correlation What happened? #NoEstimates? Our sprints are 21 days - why could this be happening?
51. The best I’ve seen so far!
52. Coaching in a data-driven world Descriptive reporting Questions Where is the problem? Why is it happening? What are the trends? What happened? What is happening? Visuals Diagnostic discover & explore Predictive forecast - WIP Items per Week - Scatter Plot - Net flow per week - Throughput History Trend - Stuck Work - Flow Efficiency - Estimate Correlation - Cumulative Flow (CFD)
53. Landing Zone (Burn-up) What is likely to happen? What options are there?
54. Potentially Deliverable Scope (PDS) What options are there? What is likely to happen?
55. Throughput Forecaster
56. Throughput Forecaster 16 weeks later... 1 story out!
57. Coaching in a data-driven world Descriptive reporting Questions Where is the problem? Why is it happening? What are the trends? What happened? What is happening? Visuals Diagnostic discover & explore Predictive forecast Prescriptive anticipative What is likely to happen? What options are there? - WIP Items per Week - Scatter Plot - Net flow per week - Throughput History Trend - Stuck Work - Flow Efficiency - Estimate Correlation - Cumulative Flow (CFD) - PDS - Landing Zone (Burn-up) - Throughput Forecaster - Monte Carlo: How Many?
58. WIP by Person What should I do?
59. Stale Work What is the next best action?
61. “When a measure becomes a target, it ceases to be a good measure” - Charles Goodhart
62. Goodhart’s law... “I’m working on that card but didn’t want to move it forward on the board because it would mean we go over our WIP limit”
63. “The ostrich problem includes situations in which people receive relevant information but intentionally fail to evaluate the implications for their goal progress - in other words, they reject the information.” - Dr Thomas Webb
64. The Ostrich Problem... “Hmm….let’s see how we get on for the next 4 weeks before we show the data of what we can deliver to the business”
65. Coaching in a data-driven world Descriptive reporting Questions Where is the problem? Why is it happening? What are the trends? What happened? What is happening? Visuals What should I do? What is the next best action? Diagnostic discover & explore Predictive forecast Prescriptive anticipative - WIP Items per Week - Scatter Plot - Net flow per week - Throughput History Trend - Stuck Work - Flow Efficiency - Estimate Correlation - Cumulative Flow (CFD) - Stale Work - WIP by Person - Aging Work In Progress - Requirements Readiness - PDS - Landing Zone (Burn-up) - Throughput Forecaster - Monte Carlo: How Many? What is likely to happen? What options are there?
66. Coaching in a data-driven world Descriptive reporting Questions What happened? What is happening? Visuals Diagnostic discover & explore Predictive forecast Prescriptive anticipative - WIP Items per Week - Scatter Plot - Net flow per week - Throughput History Trend Where can you get it? - Focused Objective
67. Coaching in a data-driven world Descriptive reporting Questions What happened? What is happening? Visuals Diagnostic discover & explore Predictive forecast Prescriptive anticipative - WIP Items per Week - Scatter Plot - Net flow per week - Throughput History Trend Where can you get it? - SenseAdapt
68. Coaching in a data-driven world Descriptive reporting Questions What happened? What is happening? Visuals Diagnostic discover & explore Predictive forecast Prescriptive anticipative - WIP Items per Week - Scatter Plot - Net flow per week - Throughput History Trend Where can you get it? - Focused Objective
69. Coaching in a data-driven world Descriptive reporting Questions What happened? What is happening? Visuals Diagnostic discover & explore Predictive forecast Prescriptive anticipative - WIP Items per Week - Scatter Plot - Net flow per week - Throughput History Trend Where can you get it? - Focused Objective
70. Coaching in a data-driven world Questions Where is the problem? Why is it happening? What are the trends? Visuals Diagnostic discover & explore Predictive forecast Prescriptive anticipative - Stuck Work - Flow Efficiency - Estimate Correlation - Cumulative Flow (CFD) Descriptive reporting Where can you get it? - SenseAdapt
71. Coaching in a data-driven world Questions Where is the problem? Why is it happening? What are the trends? Visuals Diagnostic discover & explore Predictive forecast Prescriptive anticipative - Stuck Work - Flow Efficiency - Estimate Correlation - Cumulative Flow (CFD) Descriptive reporting Where can you get it? - Actionable Agile
72. Coaching in a data-driven world Questions Where is the problem? Why is it happening? What are the trends? Visuals Diagnostic discover & explore Predictive forecast Prescriptive anticipative - Stuck Work - Flow Efficiency - Estimate Correlation - Cumulative Flow (CFD) Descriptive reporting Where can you get it? - SenseAdapt
73. Coaching in a data-driven world Questions Where is the problem? Why is it happening? What are the trends? Visuals Diagnostic discover & explore Predictive forecast Prescriptive anticipative - Stuck Work - Flow Efficiency - Estimate Correlation - Cumulative Flow (CFD) Descriptive reporting Where can you get it? - Actionable Agile
74. Coaching in a data-driven world Questions Visuals Predictive forecast Prescriptive anticipative Descriptive reporting Diagnostic discover & explore - PDS - Landing Zone (Burn-up) - Throughput Forecaster - Monte Carlo: How Many? What is likely to happen? What options are there? Where can you get it? - SenseAdapt
75. Coaching in a data-driven world Questions Visuals Predictive forecast Prescriptive anticipative Descriptive reporting Diagnostic discover & explore - PDS - Landing Zone (Burn-up) - Throughput Forecaster - Monte Carlo: How Many? What is likely to happen? What options are there? Where can you get it? - SenseAdapt
76. Coaching in a data-driven world Questions Visuals Predictive forecast Prescriptive anticipative Descriptive reporting Diagnostic discover & explore - PDS - Landing Zone (Burn-up) - Throughput Forecaster - Monte Carlo: How Many? What is likely to happen? What options are there? Where can you get it? - Focused Objective
77. Coaching in a data-driven world Questions Visuals Predictive forecast Prescriptive anticipative Descriptive reporting Diagnostic discover & explore - PDS - Landing Zone (Burn-up) - Throughput Forecaster - Monte Carlo: How Many? What is likely to happen? What options are there? Where can you get it? - Actionable Agile
78. Coaching in a data-driven world Descriptive reporting Questions Visuals Diagnostic discover & explore Predictive forecast Prescriptive anticipative What should I do? What is the next best action? - Stale Work - WIP by Person - Aging Work In Progress - Requirements Readiness Where can you get it? - SenseAdapt
79. Coaching in a data-driven world Descriptive reporting Questions Visuals Diagnostic discover & explore Predictive forecast Prescriptive anticipative What should I do? What is the next best action? - Stale Work - WIP by Person - Aging Work In Progress - Requirements Readiness Where can you get it? - SenseAdapt
80. Coaching in a data-driven world Descriptive reporting Questions Visuals Diagnostic discover & explore Predictive forecast Prescriptive anticipative What should I do? What is the next best action? - Stale Work - WIP by Person - Aging Work In Progress - Requirements Readiness Where can you get it? - Actionable Agile
81. Coaching in a data-driven world Descriptive reporting Questions Visuals Diagnostic discover & explore Predictive forecast Prescriptive anticipative What should I do? What is the next best action? - Stale Work - WIP by Person - Aging Work In Progress - Requirements Readiness Where can you get it? - SenseAdapt
82. Coaching in a data-driven world Descriptive reporting Questions Where is the problem? Why is it happening? What are the trends? What happened? What is happening? Visuals What should I do? What is the next best action? Diagnostic discover & explore Predictive forecast Prescriptive anticipative - WIP Items per Week - Scatter Plot - Net flow per week - Throughput History Trend - Stuck Work - Flow Efficiency - Estimate Correlation - Cumulative Flow (CFD) - Stale Work - WIP by Person - Aging Work In Progress - Requirements Readiness - PDS - Landing Zone (Burn-up) - Throughput Forecaster - Monte Carlo: How Many? What is likely to happen? What options are there?
83. Further reading • Troy Magennis - Focused Objective - bit.ly/SimResources • Dan Vacanti - Actionable Agile Metrics for Predictability - tiny.cc/danvac • Dominica Degrandis - Making Work Visible - tiny.cc/workvisible • Bazil Arden - Try data-based coaching to counteract cognitive biases - tiny.cc/bazil1 • Bazil Arden - How data-based coaching enhances psychological safety - tiny.cc/bazil2 • Jerry Z. Muller - The Tyranny of Metrics - tiny.cc/tyranny • Goodhart, Charles (1981). "Problems of Monetary Management: The U.K. Experience" • Webb, T., Chang, B. and Benn, Y. (2013). ‘The Ostrich Problem’: Motivated Avoidance or Rejection of Information About Goal Progress.
84. © 2017 PwC. All rights reserved. PwC refers to the PwC network and/or one or more of its member firms, each of which is a separate legal entity. Please see www.pwc.com/structure for further details. Nicolas Brown, Agile Lead nicolas.brown@pwc.com @nbrown02 Thank you! Any questions?