Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Metrics - You are what you measure (DevOps Perth)

Rob Crowley
September 18, 2014

Metrics - You are what you measure (DevOps Perth)

DevOps is no longer just the concern of cutting edge start-ups in Silicon Valley and is gaining wide scale adoption within established industries. This session focuses on the Metrics pillar of DevOps and explores how we can leverage metrics to drive the software delivery process based on data rather than gut feel and opinions.

Rob Crowley

September 18, 2014

More Decks by Rob Crowley

Other Decks in Technology


  1. Empower you with new ideas to bring your team together!

    Metrics. What are they? What to measure? How to visualise? Tooling. Some secret sauce to make our lives easier Metrics as a catalyst for cultural change Goals for Session
  2. To inform To compare To motivate To understand To improve

    To protect Not: Just because successful companies do! Why do we measure
  3. A quantifiable measure of any component or process used to

    gauge the performance of your business – Application – Business – People – Process – System What is a metric
  4. Basis of measurement Assumptions Level and usage Desired trend When

    to use When to stop How to game Warnings Facets of a metric
  5. “Lack of direction, not lack of time, is the problem.

    We all have twenty-four hour days.” - Zig Zigler
  6. Pivot: Base conversations on metrics instead of non actionable opinions

    Use stakeholder input as the basis for your initial set of metrics – “What do you want out of this?” – “How quickly do you want this?” Metrics can help guide us
  7. Awesome book, poor title Core Principles – Minimum Viable Product

    (MVP) – Continuous Deployment – Split Testing (aka A/B Testing) – Actionable Metrics – Pivot – Innovation Accounting – Build-Measure-Learn Loop
  8. Acquisition – finding new users Activation – getting users to

    give your product a try Retention – making sure those users stick around Referral – have your loyal users invite others Revenue – making some money from all this Note: These steps are not strictly sequential Pirate Funnel (AARRR!)
  9. Do not focus solely on metrics that focus on reducing

    errors Focus on achieving excellence and positive outcomes will ensue
  10. DevOps State of the Union focuses on these four core

    metrics – Deployment Frequency – Lead Time for Changes – Mean Time to Recover (MTTR) – Failed Change Rate Example Metrics
  11. Fact: We only realise business value from the work we

    do when it’s released into production The bottleneck or constraint is the limiting factor on our ability to deliver more quickly Strengthening any link other than the weakest is a waste of time and effort Use metrics to identify current constraint Theory of Constraints
  12. Display all metrics on a dashboard – Business (i.e. Application

    dropouts) – Dev (i.e. Performance metrics) – Ops (i.e. Web Server CPU Usage) Single URL for all data Make it easy for colleagues to access dashboard Visualisation
  13. Use a tool that can handle different kinds of metrics

    A few standouts are: – StatsD/Graphite/Grafana – Heka – Logstash/Elastic Seach/Kibana Tools
  14. Case Study: US Health Service In the 1990s the US

    Health Service decided to make hospital mortality rates public. What better metric for hospital performance? Observed Behaviour: Best way to improve rating was not to admit critically ill patients in the first place You are what you measure
  15. “The actual company values, as opposed to the nice-sounding values,

    are shown by who gets rewarded, promoted or let go” - Patty McCord
  16. Case Study: Enron Enron which went bankrupt from fraud and

    whose leaders went to jail, had the following company values: Integrity, Communication, Respect, Excellence Where these really the core values at Enron? You are what you measure
  17. “In the absence of clarity around an objective, any measure

    will do” - Bob Paladino Too Many Metrics
  18. Depict rosiest picture possible but do not accurately reflect the

    key drivers of your business – Inaccessible – Not auditable – Not actionable Vanity Metrics
  19. Metrics based on output rather than outcomes Aim to maximise

    outcomes and minimise output – Total lines of code – Number of bugs fixed – Total hours worked – Lines of code per developer Useless Metrics
  20. “We’re not doing this data analysis thing just because. We’re

    gathering data points because we’ve seen it work to great effect in other organisations and industries.” - Anonymous Cargo Cult Metrics
  21. Metrics being used for evil rather than for good Confronting

    this organisational anti-pattern can be a good first step Information sharing is the best defense Break down knowledge silos in the organisation Weaponized Metrics