Finding Metrics that Matter and Using them Safely

Finding Metrics that Matter and Using them Safely

Data-driven decision-making is a goal of most leaders today. Unfortunately, between the overwhelming amounts of data available to us and the ease of falling prey to misleading vanity metrics, how to begin this journey can feel extremely elusive.

How do we know what’s important enough to measure? How do we get a holistic picture of organizational health? How do we ensure our metrics help us more than they hurt us? There are many important questions to consider before doubling down on your inventory of metrics.

This interactive presentation will help you ensure you:

* can map your metrics back to your desired goals.
* have a balanced set of metrics to provide a holistic picture of health.
* have calculated the gaming potential of your metrics to ensure the value outweighs the risk.

With these new tools in your arsenal and a better understanding of metrics safety, you can set the stage for smart, data-driven decision-making in your organization.

1100c19690e8eb006da7a35a6b17be97?s=128

Julia Wester

May 29, 2018
Tweet

Transcript

  1. 3.

    @everydaykanban | @TeamLagom #devopscon18 Too many metrics? When we don’t

    know what to measure, we take what’s easy / available…
  2. 6.

    What you measure shows people what you value, so take

    care when choosing your metrics @everydaykanban | @TeamLagom #devopscon18
  3. 10.

    Walk the path to discover your metrics @everydaykanban | @TeamLagom

    Deliver value more frequently #devopscon18
  4. 11.

    Walk the path to discover your metrics @everydaykanban | @TeamLagom

    Deliver value more frequently Start more or help finish? #devopscon18
  5. 12.

    Walk the path to discover your metrics @everydaykanban | @TeamLagom

    Deliver value more frequently Start more or help finish? do WiP levels correlates to delivery speed? #devopscon18
  6. 13.

    Walk the path to discover your metrics @everydaykanban | @TeamLagom

    Deliver value more frequently Start more or help finish? do WiP levels correlates to delivery speed? Team’s current WiP levels AND duration (cycle time) #devopscon18
  7. 14.

    Know what each metric really tells you… @everydaykanban | @TeamLagom

    #devopscon18 Fooling yourself with vanity metrics everydaykanban I sure do get a lot done! I add so much value!
  8. 15.

    Know what each metric really tells you… @everydaykanban | @TeamLagom

    #devopscon18 Fooling yourself with vanity metrics everydaykanban I sure do get a lot done! I add so much value! • lines of code written • # of issues closed • # of deploys per day • # of points finished
  9. 16.

    Know what each metric really tells you… @everydaykanban | @TeamLagom

    #devopscon18 Fooling yourself with vanity metrics everydaykanban I sure do get a lot done! I add so much value! • lines of code written • # of issues closed • # of deploys per day • # of points finished • revenue generated • revenue protected • cost avoided • cost saved
  10. 18.

    Put each metric to the “So What?” test #devopscon18 @everydaykanban

    | @TeamLagom Does it: 1. matter to my customer? 2. help me make a decision or take action?
  11. 20.

    @everydaykanban | @TeamLagom #devopscon18 Quality How good? Responsiveness How fast?

    Sustainability How repeatable? Productivity How much?
  12. 21.

    Quality: Issues by Application / Service over time @everydaykanban |

    @TeamLagom #devopscon18 0 25 50 75 100 Sys A Sys B Sys C Sys D Q4 2017 Q1 2018 Q2 2018 Find your problem apps and focus improvement efforts there…
  13. 22.

    Quality: Customer Satisfaction @everydaykanban | @TeamLagom #devopscon18 0 25 50

    75 100 Jan Feb Mar Apr May Phone Website Measure it however you can, even if its qualitative…
  14. 23.
  15. 29.

    @everydaykanban | @TeamLagom #devopscon18 0 25 50 75 100 Sys

    A Sys B Sys C Sys D Q4 2017 Q1 2018 Q2 2018 Incidents by System MTTR Throughput Open vs Closed
  16. 30.

    Improvement: Flow Efficiency @everydaykanban | @TeamLagom #devopscon18 The best metric

    you’re not using! Lead Time Wait Wait Wait Work Work Work http://www.everydaykanban.com/2016/09/25/flow-efficiency/
  17. 31.

    @everydaykanban | @TeamLagom #devopscon18 It’s all 3 ways of DevOps

    rolled up into one metric! Improvement: Flow Efficiency
  18. 33.

    @everydaykanban | @TeamLagom #devopscon18 Better and with company trend Oops.

    Still good, but trending adversely Creeping up…
  19. 34.

    You don’t need perfect data, you just need more information

    than you have right now. If you have none, even just a little information is a big improvement. @everydaykanban | @TeamLagom #devopscon18 Remind yourself…
  20. 40.

    @everydaykanban | @TeamLagom #devopscon18 Created by Julia Wester 2018 More

    Info on https://lagom.solutions Lagom Discovery Canvas Key Decision Hypothesis Plan / Do Study Adjust Lagom (Just Enough) Too Much (Extreme) Too Little (Extreme) What decision are you trying to make? What steps do you think will get you closer to lagom? What consequences occur when you operate at this extreme… positive and negative! How will you test your hypothesis? How will you know if you are closer to lagom? When will you review? What do your success metrics tell you? Are you closer to lagom, farther away or the same distance? Do you need to tweak the plan & continue or make a new hypothesis? What consequences occur when you operate at this extreme… positive and negative! What are the ideal outcomes when you do this just right? Natural Tendency Check the box in the section below that resembles your normal operating condition
  21. 41.

    @everydaykanban | @TeamLagom #devopscon18 Created by Julia Wester 2018 More

    Info on https://lagom.solutions Lagom Discovery Canvas Key Decision Hypothesis Plan / Do Study Adjust Lagom (Just Enough) Too Much (Extreme) Too Little (Extreme) What decision are you trying to make? What steps do you think will get you closer to lagom? What consequences occur when you operate at this extreme… positive and negative! How will you test your hypothesis? How will you know if you are closer to lagom? When will you review? What do your success metrics tell you? Are you closer to lagom, farther away or the same distance? Do you need to tweak the plan & continue or make a new hypothesis? What consequences occur when you operate at this extreme… positive and negative! What are the ideal outcomes when you do this just right? Natural Tendency Check the box in the section below that resembles your normal operating condition
  22. 42.

    @everydaykanban | @TeamLagom #devopscon18 Created by Julia Wester 2018 More

    Info on https://lagom.solutions Lagom Discovery Canvas Key Decision Hypothesis Plan / Do Study Adjust Lagom (Just Enough) Too Much (Extreme) Too Little (Extreme) What decision are you trying to make? What steps do you think will get you closer to lagom? What consequences occur when you operate at this extreme… positive and negative! How will you test your hypothesis? How will you know if you are closer to lagom? When will you review? What do your success metrics tell you? Are you closer to lagom, farther away or the same distance? Do you need to tweak the plan & continue or make a new hypothesis? What consequences occur when you operate at this extreme… positive and negative! What are the ideal outcomes when you do this just right? Natural Tendency Check the box in the section below that resembles your normal operating condition
  23. 43.

    @everydaykanban | @TeamLagom #devopscon18 • Fast delivery • Keeping pace

    • Team challenged, not overwhelmed • Low # of bugs / High Customer Satisfaction scores • Fast delivery ✔ • Keeping pace ✔ • Bored People • Mediocre quality • Seen as overstaffed • Slow delivery • Not Keeping Pace • Team Burnout • Poor quality • No time for improvement How much work should we have in process at once? ❌ Just measuring these gives a false + measure a balanced set of metrics to see negative consequences
  24. 47.

    “Fear invites wrong figures. Bearers of bad news fare badly.

    To keep his job, anyone may present to his boss only good news.” –W. Edwards Deming #devopscon18 @everydaykanban | @TeamLagom
  25. 49.

    –W. Edwards Deming “If you give a manager a numerical

    target, he’ll make it, even if he has to destroy the organization in the process.” @everydaykanban | @TeamLagom #devopscon18
  26. 50.

    To get ahead of these issues, intentionally explore the dark

    side of metrics @everydaykanban | @TeamLagom #devopscon18 I’m just misunderstood!
  27. 52.

    –Eli Godratt “Tell me how you’ll measure me and I’ll

    tell you how I’ll behave. If you measure me in an illogical way, do not complain about illogical behavior.” @everydaykanban | @TeamLagom #devopscon18
  28. 54.

    #devopscon18 How can we make this safer? What is the

    impact? How can you game it? “How Can I Game It?” Ask your team to play and get a wider perspective @everydaykanban | @TeamLagom
  29. 57.

    Do you have any metric tips to share? Any questions?

    @everydaykanban | @TeamLagom #devopscon18