Avoiding Metrics Mistakes

Metrics. We gather them. We make decisions based on them. We are judged by them. We don’t question them as often as we should. Despite our best intentions, we don’t always know how to decide what to measure. Even when we get it right we can end up doing more damage than good because of how we use them.

This deck was presented to the LeanAgile London Meetup group by Julia Wester, product manager for ActionableAgile, to help people learn about some common metrics mistakes, how to know if they're making them, and, most importantly, how to avoid them in the future.

Transcript

1. SIX
   COMMON
   METRICS
   MISTAKES
   AND HOW
   TO AVOID
   THEM!!
   

   View Slide

2. JULIA
   WESTER
   Co-Founder, 55 Degrees AB
   Scrum.org PST
   ProKanban.org Founding Member
   Agendashift Partner
   @everydaykanban
   Hello! I am
   

   View Slide

3. ACTIONABLE AGILE POWERED BY 55 DEGREES AB
   TM
   http://actionableagile.com
   I’m a product
   manager too!
   

   View Slide

4. NOT HAVING
   A “WHY”
   1
   

   View Slide

5. View Slide

6. View Slide

7. Does it really
   give expected
   insight?
   think of small, day-
   to-day decisions.
   Courtesy of Larry Maccherone (@LMaccherone)
   https://medium.com/@lmaccherone/odim-12d80823222
   Reduced time /
   cost for rework
   Should we increase
   testing time / effort?
   Are we finding
   defects early
   enough?
   Escaped defects
   ODIM
   

   View Slide

8. defects found in production
   ESCAPED DEFECTS
   

   View Slide

9. Count
   DEFECTS PER ENVIRONMENT
   

   View Slide

10. Percentiles
    DEFECTS PER ENVIRONMENT
    

    View Slide

11. Does it really
    give expected
    insight?
    think of small, day-
    to-day decisions.
    Courtesy of Larry Maccherone (@LMaccherone)
    https://medium.com/@lmaccherone/odim-12d80823222
    Reduced time /
    cost for rework
    Should we increase
    testing time / effort?
    Are we finding
    defects early
    enough?
    % of defects found per
    environment
    ODIM
    

    View Slide

12. REFUSING TO
    ACKNOWLEDGE
    THE GAME
    2
    

    View Slide

13. TRUTH:
    What you measure
    shows what you value
    

    View Slide

14. USED WITH PERMISSION
    

    View Slide

15. ELI GOLDRATT
    “Tell me how you’ll measure me
    and I’ll tell you how I’ll behave.
    If you measure me in an
    illogical way, do not complain
    about illogical behavior.”
    

    View Slide

16. View Slide

17. How can you
    game it?
    How can we
    make this safer?
    What is the
    impact?
    A GAME TO PLAY
    WITH YOUR TEAM
    

    View Slide

18. SHAMING PEOPLE
    WITH METRICS
    3
    

    View Slide

19. “Fear invites wrong
    figures. Bearers of bad
    news fare badly. To
    keep his job, anyone
    may present to his boss
    only good news.”
    W. EDWARDS DEMING
    “NEW ECONOMICS”
    

    View Slide

20. COURTESY OF TROY MAGENNIS (@T_MAGENNIS)
    WHAT IF YOU WERE ON THE LIST?
    “What a slacker!”
    “Wow, I bet they’re working on
    other priorities right now.”
    “Makes sense. That person’s
    code was part of a deep
    exploratory testing session.”
    Group A
    Team Member 1
    Team Member 3
    Team Member 5
    Team Member 7
    Team Member 9
    Team Member 11
    Group B
    Team Member 2
    Team Member 4
    Team Member 6
    Team Member 8
    Team Member 10
    Team Member 12
    

    View Slide

21. BY REMOVING THE CHOICE BETWEEN COMPANY AND SELF
    MINIMIZE THE DANGER
    • Focus on desired outcomes.
    • Measure individuals only when it makes sense.
    • Understand the audience for your metrics.
    • Have your team help you identify metrics that put
    them in a bad spot.
    

    View Slide

22. Fooling
    yourself with
    vanity
    metrics
    @everydaykanban
    I sure do
    get a lot
    done!
    I add so
    much
    value!
    CONFUSING
    ACTIVITY WITH
    SUCCESS
    4
    

    View Slide

23. KNOW WHAT EACH METRIC REALLY TELLS YOU
    • lines of code written
    • # of issues closed
    • # of deploys per day
    • # of points finished
    Fooling
    yourself with
    vanity
    metrics
    ydaykanban
    I sure do
    get a lot
    done!
    I add so
    much
    value!
    • revenue generated
    • revenue protected
    • cost avoided
    • cost saved
    

    View Slide

24. Does it:
    1. matter to my customer?
    2. help me make a decision or take action?
    ASK “SO WHAT”?
    

    View Slide

25. BLINDLY PURSUING
    TARGETS
    5
    

    View Slide

26. W. EDWARDS DEMING
    “If you give a manager a
    numerical target, he’ll make it,
    even if he has to destroy the
    organization in the process.”
    

    View Slide

27. Accelerate:
    State of
    DevOps 2019
    https://services.google.com/fh/files/
    misc/state-of-devops-2019.pdf
    Let’s get down to
    10 minutes max!
    

    View Slide

28. 
    
    WATCH FOR DECISION POINTS
    

    View Slide

29. WITH A FEW KEY TIPS
    MAKE TARGETS SAFER
    • Set targets in terms of outcomes rather than
    tactics.
    • Treat tactics as experiments. It’s ok to
    change direction.
    • Look at the broader cost of your target, and
    what you could put at risk.
    • Know when to call it “good enough”
    

    View Slide

30. FIX IT
    HERE,
    BREAK
    IT THERE
    6
    

    View Slide

31. View Slide

32. COURTESY OF LARRY MACCHERONE (@LMACCHERONE)
    

    View Slide

33. COURTESY OF TROY MAGENNIS (@T_MAGENNIS) - HTTPS://BIT.LY/SIMRESOURCES
    A Helpful Template
    Time in Process 1 Sprint
    Escaped Defects
    QUADRANT
    QUADRANT
    RESPONSIVENESS
    RESPONSIVENESS
    (QUALITY)
    

    View Slide

34. COURTESY OF TROY MAGENNIS (@T_MAGENNIS) - HTTPS://BIT.LY/SIMRESOURCES
    BEHAVIORAL
    POLARITY
    WORKSHEET
    Troy Magennis
    Focused Objective
    

    View Slide

35. Spectrum Thinking Worksheet
    For a more detailed worksheet, visit
    https://github.com/55-Degrees/worksheets
    PDSA
    

    View Slide

36. SUMMARY
    

    View Slide

37. Free Resources
    • Blog Posts, Webinars, etc.
    Short Training Sessions
    • Introduction to ActionableAgile Video Sessions
    In-Depth Workshops
    • Professional Scrum with Kanban (PSK)
    • Advanced Agile Metrics, Forecasting and Predictability
    LEARN
    https://55degrees.se/events
    

    View Slide

38. ENGAGE
    Subscribe to our newsletter
    https://55degrees.se/subscribe
    Follow on Twitter
    @55degreesAB
    Engage with us on LinkedIn
    https://www.linkedin.com/company/55degrees
    

    View Slide

39. ACTIONABLE AGILE POWERED BY 55 DEGREES AB
    TM
    http://actionableagile.com
    2 months free on new monthly subscriptions
    20% off of a new yearly subscription
    Using promo code lal-2months
    Using promo code lal-20percent
    OR
    

    View Slide

40. LET’S INTERACT ON MURAL
    CHECK THE ZOOM CHAT FOR THE LINK TO MY MURAL PAGE
    

    View Slide