Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Metrics: The Good the Bad and the Ugly

Metrics: The Good the Bad and the Ugly

Sharing business metrics can be a great enabler of collaboration between product, commercial, development and operations. They provide a common reference point and help sort fact from opinion. They inform our decisions and prompt useful conversations.

But are they always all they seem; and when do they stop being helpful and start causing harm?

In this session we will look at examples of metrics from development and operations and use them to explore issues of provenance, cleanliness and bias. We’ll look at how the sharing of metrics can have a different impact on different audiences.

We’ll share takeaways on how to establish and grow a metrics practice that is helpful to everyone in the business.

Chris is the co-author, with Mattia Battiston of Team Guide to Metrics for Business Decisions (sample chapter available at bizmetricsbook.com). He has been leading engineering teams since 1999. He was pivotal in the BBC’s adoption of agile in 2000-2001. Since then, he has worked for broadcasters and video platform owners including Dish Network in the US and Deutsche Telekom in Europe.

1797307564c2566c3e006aac2f44458f?s=128

Chris Young

April 15, 2019
Tweet

More Decks by Chris Young

Other Decks in Technology

Transcript

  1. Metrics The good, the bad and the ugly

  2. None
  3. Imagined Realities “We cooperate effectively with strangers because we believe

    in things like gods, nations, money and human rights. Yet none of these things exists outside the stories that people invent and tell one another.” Yuval Noah Harari https://www.ynharari.com/topic/power-and-imagination/ @worldofchris
  4. Stories @worldofchris

  5. Reality Abstracts from Impacts Stories @worldofchris

  6. Reality Abstracts from Impacts Stories Quality Value @worldofchris

  7. Part One: Quality

  8. What do we mean by quality?

  9. “Quality is Conformance to Requirements” Philip B. Crosby @worldofchris

  10. So what are the requirements?

  11. What’s the context?

  12. Dev Ops Generic Model @worldofchris

  13. Dev Ops Generic Model Toyota Lean Metrics IBM RAS Model

    @worldofchris
  14. Dev Generic Model Toyota Lean Metrics • Throughput • Lead

    Time • Value / Failure Demand @worldofchris
  15. Throughput @worldofchris

  16. Lead Time Story Lead Time @worldofchris

  17. Value vs Failure Demand @worldofchris

  18. Dev Ops Generic Model Toyota Lean Metrics IBM RAS Model

    @worldofchris
  19. Ops Generic Model IBM RAS Model • Reliability • Availability

    • Serviceability @worldofchris
  20. Reliability - Defect discovery rate @worldofchris

  21. Reliability - Number of test cases @worldofchris

  22. Availability - How does it change under load? @worldofchris

  23. Availability - How does it change with load? @worldofchris

  24. Availability - When is it needed? @worldofchris

  25. Serviceability - Who can fix it? HT @PapaChrisMatts for Liquidity

    Matrix @worldofchris
  26. Let me tell you a story

  27. Once upon a time there was a SaaS platform... Platform

    @worldofchris
  28. Platform Admin Input Output Search And that platform had four

    services... @worldofchris
  29. Platform Admin Input Output Search But one of these services

    had problems. @worldofchris
  30. The Business Decision

  31. Admin Input Output Search Sales wanted to add more features...

    Engineering wanted to fix the problems they had inherited @worldofchris
  32. Making the case - Operational Metrics

  33. @worldofchris

  34. Reliability - Cumulative Defect discovery @worldofchris

  35. Availability Search latency in seconds

  36. Serviceability

  37. Making the case - Development Metrics

  38. Throughput @worldofchris

  39. So how did the story end?

  40. @worldofchris

  41. @worldofchris

  42. Deployed Backlog Open Do Next In Progress QA Queue QA

    Deployment Queue @worldofchris
  43. Where does all this data come from?

  44. Dev Ops Generic Model Toyota Lean Metrics IBM RAS Model

    @worldofchris
  45. Quality of Measurement Practice Metrics Practice Dev Ops

  46. Cleanliness

  47. None
  48. @worldofchris

  49. Selection

  50. @worldofchris

  51. @worldofchris

  52. @lisacrispin @janetgregoryca @worldofchris

  53. Part Two: Value

  54. Reality Abstracts from Impacts Stories Quality Value @worldofchris

  55. Quality of Measurement Practice Metrics Practice Dev Ops @worldofchris

  56. Business Value Metrics Practice Cleanliness, Bias, Selection Dev Throughput, Lead

    Time, Failure Demand Ops Reliability, Availability, Serviceability @worldofchris
  57. “Quality is Conformance to Requirements” Philip B. Crosby @worldofchris

  58. Value + “Quality is Conformance to Requirements” Philip B. Crosby

    @worldofchris
  59. Business Need Metrics Practice Cleanliness, Bias, Selection Dev Throughput, Lead

    Time, Failure Demand Ops Reliability, Availability, Serviceability @worldofchris
  60. Business Need AARRR! - Pirate Metrics Metrics Practice Cleanliness, Bias,

    Selection Dev Throughput, Lead Time, Failure Demand Ops Reliability, Availability, Serviceability @worldofchris
  61. ACQUISITION ACTIVATION REVENUE RETENTION REFERRAL @davemcclure Pirate Metrics https://500hats.typepad.com/500blogs/2007/09/startup-metrics.html @worldofchris

  62. Let me tell you a another story

  63. @worldofchris

  64. Business Need AARRR! - Pirate Metrics Metrics Practice Cleanliness, Bias,

    Selection Dev Throughput, Lead Time, Failure Demand Ops Reliability, Availability, Serviceability @worldofchris
  65. ACQUISITION ACTIVATION REVENUE RETENTION REFERRAL GOAL: Higher revenue through retention

    of trial customer as a gym member @worldofchris
  66. Business Need AARRR! - Pirate Metrics Metrics Practice Cleanliness, Bias,

    Selection Dev Throughput, Lead Time, Failure Demand Ops Reliability, Availability, Serviceability @worldofchris
  67. Lead Time @worldofchris

  68. Throughput @worldofchris

  69. Business Need AARRR! - Pirate Metrics Metrics Practice Cleanliness, Bias,

    Selection Dev Throughput, Lead Time, Failure Demand Ops Reliability, Availability, Serviceability @worldofchris
  70. Test cases @worldofchris

  71. None
  72. ACQUISITION ACTIVATION REVENUE RETENTION REFERRAL GOAL: Higher revenue through retention

    of trial customer as a gym member BONUS: Repeat Business for the agency @worldofchris
  73. Business Need AARRR! - Pirate Metrics Metrics Practice Cleanliness, Bias,

    Selection Dev Throughput, Lead Time, Failure Demand Ops Reliability, Availability, Serviceability @worldofchris
  74. Imagined Realities “We cooperate effectively with strangers because we believe

    in things like gods, nations, money and human rights. Yet none of these things exists outside the stories that people invent and tell one another.” Yuval Noah Harari https://www.ynharari.com/topic/power-and-imagination/ @worldofchris
  75. ― https://www.goodreads.com/photo/author/3472.Margaret_Atwood @worldofchris