Finding Metrics that Matter and Using them Safely

Finding Metrics that Matter and Using them Safely

Data-driven decision-making is a goal of most leaders today. Unfortunately, between the overwhelming amounts of data available to us and the ease of falling prey to misleading vanity metrics, how to begin this journey can feel extremely elusive.

How do we know what’s important enough to measure? How do we get a holistic picture of organizational health? How do we ensure our metrics help us more than they hurt us? There are many important questions to consider before doubling down on your inventory of metrics.

This interactive presentation will help you ensure you:

* can map your metrics back to your desired goals.
* have a balanced set of metrics to provide a holistic picture of health.
* have calculated the gaming potential of your metrics to ensure the value outweighs the risk.

With these new tools in your arsenal and a better understanding of metrics safety, you can set the stage for smart, data-driven decision-making in your organization.

1100c19690e8eb006da7a35a6b17be97?s=128

Julia Wester

May 29, 2018
Tweet

Transcript

  1. Finding Metrics that Matter and Using them Safely @everydaykanban |

    @TeamLagom #devopscon18
  2. @everydaykanban | @TeamLagom #devopscon18 Too many metrics?

  3. @everydaykanban | @TeamLagom #devopscon18 Too many metrics? When we don’t

    know what to measure, we take what’s easy / available…
  4. There is no one-size-fits-all perfect set of metrics (sorry!) @everydaykanban

    | @TeamLagom #devopscon18
  5. @everydaykanban | @TeamLagom #devopscon18 …though there may be some common,

    good baseline metrics
  6. What you measure shows people what you value, so take

    care when choosing your metrics @everydaykanban | @TeamLagom #devopscon18
  7. Doing Metrics the “right way” @everydaykanban | @TeamLagom #devopscon18 #winning

    safe ^
  8. @everydaykanban | @TeamLagom #devopscon18 Start with a clear outcome in

    mind
  9. Walk the path to discover your metrics @everydaykanban | @TeamLagom

    #devopscon18
  10. Walk the path to discover your metrics @everydaykanban | @TeamLagom

    Deliver value more frequently #devopscon18
  11. Walk the path to discover your metrics @everydaykanban | @TeamLagom

    Deliver value more frequently Start more or help finish? #devopscon18
  12. Walk the path to discover your metrics @everydaykanban | @TeamLagom

    Deliver value more frequently Start more or help finish? do WiP levels correlates to delivery speed? #devopscon18
  13. Walk the path to discover your metrics @everydaykanban | @TeamLagom

    Deliver value more frequently Start more or help finish? do WiP levels correlates to delivery speed? Team’s current WiP levels AND duration (cycle time) #devopscon18
  14. Know what each metric really tells you… @everydaykanban | @TeamLagom

    #devopscon18 Fooling yourself with vanity metrics everydaykanban I sure do get a lot done! I add so much value!
  15. Know what each metric really tells you… @everydaykanban | @TeamLagom

    #devopscon18 Fooling yourself with vanity metrics everydaykanban I sure do get a lot done! I add so much value! • lines of code written • # of issues closed • # of deploys per day • # of points finished
  16. Know what each metric really tells you… @everydaykanban | @TeamLagom

    #devopscon18 Fooling yourself with vanity metrics everydaykanban I sure do get a lot done! I add so much value! • lines of code written • # of issues closed • # of deploys per day • # of points finished • revenue generated • revenue protected • cost avoided • cost saved
  17. Put each metric to the “So What?” test #devopscon18 @everydaykanban

    | @TeamLagom
  18. Put each metric to the “So What?” test #devopscon18 @everydaykanban

    | @TeamLagom Does it: 1. matter to my customer? 2. help me make a decision or take action?
  19. Choose metrics that compete with each other @everydaykanban | @TeamLagom

    #devopscon18
  20. @everydaykanban | @TeamLagom #devopscon18 Quality How good? Responsiveness How fast?

    Sustainability How repeatable? Productivity How much?
  21. Quality: Issues by Application / Service over time @everydaykanban |

    @TeamLagom #devopscon18 0 25 50 75 100 Sys A Sys B Sys C Sys D Q4 2017 Q1 2018 Q2 2018 Find your problem apps and focus improvement efforts there…
  22. Quality: Customer Satisfaction @everydaykanban | @TeamLagom #devopscon18 0 25 50

    75 100 Jan Feb Mar Apr May Phone Website Measure it however you can, even if its qualitative…
  23. Responsiveness: Duration (Cycle Time) @everydaykanban | @TeamLagom #devopscon18 Can you

    see different patterns for different types of work?
  24. Responsiveness: Mean Time to Recovery (MTTR) @everydaykanban | @TeamLagom #devopscon18

    Not all outages are created equal
  25. Productivity: Throughput @everydaykanban | @TeamLagom #devopscon18 Gold star if you

    show how much value the items generate!
  26. Productivity: Deployment Frequency @everydaykanban | @TeamLagom #devopscon18 Gold star if

    you show how much value the deploys generate!
  27. Sustainability: Open vs Closed @everydaykanban | @TeamLagom #devopscon18 Are we

    keeping pace with demand?
  28. Sustainability: Team Satisfaction @everydaykanban | @TeamLagom #devopscon18 Would you recommend

    people to work here?
  29. @everydaykanban | @TeamLagom #devopscon18 0 25 50 75 100 Sys

    A Sys B Sys C Sys D Q4 2017 Q1 2018 Q2 2018 Incidents by System MTTR Throughput Open vs Closed
  30. Improvement: Flow Efficiency @everydaykanban | @TeamLagom #devopscon18 The best metric

    you’re not using! Lead Time Wait Wait Wait Work Work Work http://www.everydaykanban.com/2016/09/25/flow-efficiency/
  31. @everydaykanban | @TeamLagom #devopscon18 It’s all 3 ways of DevOps

    rolled up into one metric! Improvement: Flow Efficiency
  32. @everydaykanban | @TeamLagom #devopscon18

  33. @everydaykanban | @TeamLagom #devopscon18 Better and with company trend Oops.

    Still good, but trending adversely Creeping up…
  34. You don’t need perfect data, you just need more information

    than you have right now. If you have none, even just a little information is a big improvement. @everydaykanban | @TeamLagom #devopscon18 Remind yourself…
  35. @everydaykanban | @TeamLagom #devopscon18 Avoid these metric missteps

  36. #devopscon18 Over-optimizing for a single metric @everydaykanban | @TeamLagom What

    happens when you take it to extremes?
  37. @everydaykanban | @TeamLagom #devopscon18

  38. @everydaykanban | @TeamLagom #devopscon18

  39. @everydaykanban | @TeamLagom #devopscon18

  40. @everydaykanban | @TeamLagom #devopscon18 Created by Julia Wester 2018 More

    Info on https://lagom.solutions Lagom Discovery Canvas Key Decision Hypothesis Plan / Do Study Adjust Lagom (Just Enough) Too Much (Extreme) Too Little (Extreme) What decision are you trying to make? What steps do you think will get you closer to lagom? What consequences occur when you operate at this extreme… positive and negative! How will you test your hypothesis? How will you know if you are closer to lagom? When will you review? What do your success metrics tell you? Are you closer to lagom, farther away or the same distance? Do you need to tweak the plan & continue or make a new hypothesis? What consequences occur when you operate at this extreme… positive and negative! What are the ideal outcomes when you do this just right? Natural Tendency Check the box in the section below that resembles your normal operating condition
  41. @everydaykanban | @TeamLagom #devopscon18 Created by Julia Wester 2018 More

    Info on https://lagom.solutions Lagom Discovery Canvas Key Decision Hypothesis Plan / Do Study Adjust Lagom (Just Enough) Too Much (Extreme) Too Little (Extreme) What decision are you trying to make? What steps do you think will get you closer to lagom? What consequences occur when you operate at this extreme… positive and negative! How will you test your hypothesis? How will you know if you are closer to lagom? When will you review? What do your success metrics tell you? Are you closer to lagom, farther away or the same distance? Do you need to tweak the plan & continue or make a new hypothesis? What consequences occur when you operate at this extreme… positive and negative! What are the ideal outcomes when you do this just right? Natural Tendency Check the box in the section below that resembles your normal operating condition
  42. @everydaykanban | @TeamLagom #devopscon18 Created by Julia Wester 2018 More

    Info on https://lagom.solutions Lagom Discovery Canvas Key Decision Hypothesis Plan / Do Study Adjust Lagom (Just Enough) Too Much (Extreme) Too Little (Extreme) What decision are you trying to make? What steps do you think will get you closer to lagom? What consequences occur when you operate at this extreme… positive and negative! How will you test your hypothesis? How will you know if you are closer to lagom? When will you review? What do your success metrics tell you? Are you closer to lagom, farther away or the same distance? Do you need to tweak the plan & continue or make a new hypothesis? What consequences occur when you operate at this extreme… positive and negative! What are the ideal outcomes when you do this just right? Natural Tendency Check the box in the section below that resembles your normal operating condition
  43. @everydaykanban | @TeamLagom #devopscon18 • Fast delivery • Keeping pace

    • Team challenged, not overwhelmed • Low # of bugs / High Customer Satisfaction scores • Fast delivery ✔ • Keeping pace ✔ • Bored People • Mediocre quality • Seen as overstaffed • Slow delivery • Not Keeping Pace • Team Burnout • Poor quality • No time for improvement How much work should we have in process at once? ❌ Just measuring these gives a false + measure a balanced set of metrics to see negative consequences
  44. Measuring Individuals when you value Team outcomes @everydaykanban | @TeamLagom

    #devopscon18
  45. #devopscon18 Superstars aren’t always the best team members @everydaykanban |

    @TeamLagom
  46. Using shame as an incentive @everydaykanban | @TeamLagom #devopscon18 Courtesy

    of Troy Magennis
  47. “Fear invites wrong figures. Bearers of bad news fare badly.

    To keep his job, anyone may present to his boss only good news.” –W. Edwards Deming #devopscon18 @everydaykanban | @TeamLagom
  48. Blindly pursuing targets @everydaykanban | @TeamLagom #devopscon18 95%

  49. –W. Edwards Deming “If you give a manager a numerical

    target, he’ll make it, even if he has to destroy the organization in the process.” @everydaykanban | @TeamLagom #devopscon18
  50. To get ahead of these issues, intentionally explore the dark

    side of metrics @everydaykanban | @TeamLagom #devopscon18 I’m just misunderstood!
  51. @everydaykanban | @TeamLagom #devopscon18 Remember, what you measure shows what

    you value…
  52. –Eli Godratt “Tell me how you’ll measure me and I’ll

    tell you how I’ll behave. If you measure me in an illogical way, do not complain about illogical behavior.” @everydaykanban | @TeamLagom #devopscon18
  53. Know when to discard a metric @everydaykanban | @TeamLagom #devopscon18

  54. #devopscon18 How can we make this safer? What is the

    impact? How can you game it? “How Can I Game It?” Ask your team to play and get a wider perspective @everydaykanban | @TeamLagom
  55. @everydaykanban | @TeamLagom #devopscon18

  56. @everydaykanban | @TeamLagom #devopscon18 What will you do differently on

    Monday?
  57. Do you have any metric tips to share? Any questions?

    @everydaykanban | @TeamLagom #devopscon18
  58. @everydaykanban | @TeamLagom #devopscon18 julia@lagom.solutions @everydaykanban & @TeamLagom https://lagom.solutions http://everydaykanban.com

    https://www.linkedin.com/in/juliawester/ Contact me to talk more