Metrics - You are what you measure (DevOps Perth)

DevOps is no longer just the concern of cutting edge start-ups in Silicon Valley and is gaining wide scale adoption within established industries. This session focuses on the Metrics pillar of DevOps and explores how we can leverage metrics to drive the software delivery process based on data rather than gut feel and opinions.

Transcript

1. Metrics
   You are what you measure
   @robdcrowley
   robdcrowley
   

   View Slide

2. Empower you with new ideas to bring your team together!
   Metrics. What are they? What to measure? How to visualise?
   Tooling. Some secret sauce to make our lives easier
   Metrics as a catalyst for cultural change
   Goals for Session
   

   View Slide

3. Metrics 101
   What metrics are and why we measure
   

   View Slide

4. To inform
   To compare
   To motivate
   To understand
   To improve
   To protect
   Not: Just because successful companies do!
   Why do we measure
   

   View Slide

5. A quantifiable measure of any component or process
   used to gauge the performance of your business
   – Application
   – Business
   – People
   – Process
   – System
   What is a metric
   

   View Slide

6. Basis of measurement
   Assumptions
   Level and usage
   Desired trend
   When to use
   When to stop
   How to game
   Warnings
   Facets of a metric
   

   View Slide

7. Red is good,
   Green is worthless
   

   View Slide

8. Metrics 201
   What should you measure
   

   View Slide

9. “Lack of direction, not lack of time, is the
   problem. We all have twenty-four hour days.”
   - Zig Zigler
   

   View Slide

10. Pivot: Base conversations on metrics instead of
    non actionable opinions
    Use stakeholder input as the basis for your initial
    set of metrics
    – “What do you want out of this?”
    – “How quickly do you want this?”
    Metrics can help guide us
    

    View Slide

11. Awesome book, poor title
    Core Principles
    – Minimum Viable Product (MVP)
    – Continuous Deployment
    – Split Testing (aka A/B Testing)
    – Actionable Metrics
    – Pivot
    – Innovation Accounting
    – Build-Measure-Learn Loop
    

    View Slide

12. View Slide

13. Acquisition – finding new users
    Activation – getting users to give your product a try
    Retention – making sure those users stick around
    Referral – have your loyal users invite others
    Revenue – making some money from all this
    Note: These steps are not strictly sequential
    Pirate Funnel (AARRR!)
    

    View Slide

14. Do not focus solely on metrics
    that focus on reducing errors
    Focus on achieving excellence
    and positive outcomes will
    ensue
    

    View Slide

15. DevOps State of the Union focuses on these four
    core metrics
    – Deployment Frequency
    – Lead Time for Changes
    – Mean Time to Recover (MTTR)
    – Failed Change Rate
    Example Metrics
    

    View Slide

16. Fact: We only realise business value from the
    work we do when it’s released into production
    The bottleneck or constraint is the limiting factor
    on our ability to deliver more quickly
    Strengthening any link other than the weakest is
    a waste of time and effort
    Use metrics to identify current constraint
    Theory of Constraints
    

    View Slide

17. Metrics 301
    How to visualise and share metrics
    

    View Slide

18. Display all metrics on a dashboard
    – Business (i.e. Application dropouts)
    – Dev (i.e. Performance metrics)
    – Ops (i.e. Web Server CPU Usage)
    Single URL for all data
    Make it easy for colleagues to access dashboard
    Visualisation
    

    View Slide

19. Use a tool that can handle different kinds of metrics
    A few standouts are:
    – StatsD/Graphite/Grafana
    – Heka
    – Logstash/Elastic Seach/Kibana
    Tools
    

    View Slide

20. StatsD Whisper
    Carbon
    Graphite
    Dashboard
    Apps
    UDP
    TCP
    HTTP
    Graphite and StatsD
    

    View Slide

21. Metrics 401
    Potential pitfalls when using metrics
    

    View Slide

22. Case Study: US Health Service
    In the 1990s the US Health Service decided to
    make hospital mortality rates public. What better
    metric for hospital performance?
    Observed Behaviour: Best way to improve rating
    was not to admit critically ill patients in the first
    place
    You are what you measure
    

    View Slide

23. “The actual company values, as opposed to
    the nice-sounding values, are shown by who
    gets rewarded, promoted or let go”
    - Patty McCord
    

    View Slide

24. Case Study: Enron
    Enron which went bankrupt from fraud and whose
    leaders went to jail, had the following company
    values:
    Integrity, Communication, Respect, Excellence
    Where these really the core values at Enron?
    You are what you measure
    

    View Slide

25. “In the absence of clarity around an
    objective, any measure will do”
    - Bob Paladino
    Too Many Metrics
    

    View Slide

26. Depict rosiest picture possible but do not
    accurately reflect the key drivers of your business
    – Inaccessible
    – Not auditable
    – Not actionable
    Vanity Metrics
    

    View Slide

27. Metrics based on output rather than outcomes
    Aim to maximise outcomes and minimise output
    – Total lines of code
    – Number of bugs fixed
    – Total hours worked
    – Lines of code per developer
    Useless Metrics
    

    View Slide

28. “We’re not doing this data analysis thing
    just because. We’re gathering data points
    because we’ve seen it work to great
    effect in other organisations and industries.”
    - Anonymous
    Cargo Cult Metrics
    

    View Slide

29. Metrics being used for evil rather than for good
    Confronting this organisational anti-pattern can
    be a good first step
    Information sharing is the best defense
    Break down knowledge silos in the organisation
    Weaponized Metrics
    

    View Slide

30. Questions
    

    View Slide