$30 off During Our Annual Pro Sale. View Details »

Dev and Test Metrics 101 - Agile 2018

Dev and Test Metrics 101 - Agile 2018

Cat Swetel, accompanied by Julia Wester, will outline the most common metrics used for understanding flow and impediments in software development and test functions. This session should provide a clear picture of the types of metrics that help understand Agile development, and how to get started in capturing and using these for improving Agile adoption, improvement, and management.

Julia Wester
PRO

August 08, 2018
Tweet

More Decks by Julia Wester

Other Decks in Business

Transcript

  1. Development and Test
    Metrics 101
    @catswetel | @everydaykanban

    View Slide

  2. @catswetel | @everydaykanban

    View Slide

  3. QUALITY RESPONSIVENESS
    PRODUCTIVITY PREDICTABILITY

    View Slide

  4. QUALITY RESPONSIVENESS
    PRODUCTIVITY PREDICTABILITY
    Software Development
    Performance Index (SDPI)
    Larry Maccherone

    View Slide

  5. QUALITY RESPONSIVENESS
    PRODUCTIVITY PREDICTABILITY

    View Slide

  6. QUALITY RESPONSIVENESS
    PRODUCTIVITY PREDICTABILITY
    What’s
    missing?

    View Slide

  7. QUALITY RESPONSIVENESS
    PRODUCTIVITY PREDICTABILITY

    View Slide

  8. “The RIGHTER we do the
    WRONG thing, the
    WRONGER we become.”
    -- Dr Russell Ackoff
    @catswetel | @everydaykanban

    View Slide

  9. QUALITY RESPONSIVENESS
    PRODUCTIVITY PREDICTABILITY
    QUALITY
    RESPONSIVENESS
    @catswetel | @everydaykanban

    View Slide

  10. @catswetel | @everydaykanban

    View Slide

  11. Time in Process
    Units of time per unit of work
    @catswetel | @everydaykanban

    View Slide

  12. TIME IN PROCESS
    DATE DELIVERED
    TIME IN PROCESS SCATTER PLOT

    View Slide

  13. TIME IN PROCESS
    DATE DELIVERED
    When can we expect X?
    90%
    50%

    View Slide

  14. TIME IN PROCESS
    DATE DELIVERED
    How likely by X?
    30d: 77% 21d: 50%

    View Slide

  15. TIME IN PROCESS
    DATE DELIVERED
    What’s the story?

    View Slide

  16. QUALITY RESPONSIVENESS
    PRODUCTIVITY PREDICTABILITY

    View Slide

  17. FREQUENCY
    TIME IN PROCESS
    TIME IN PROCESS
    DISTRIBUTION
    Groups are
    easier to see
    This one happens
    more often

    View Slide

  18. QUALITY RESPONSIVENESS
    PRODUCTIVITY PREDICTABILITY

    View Slide

  19. @catswetel | @everydaykanban

    View Slide

  20. WEIBULL
    DISTRIBUTION
    @catswetel | @everydaykanban

    View Slide

  21. WEIBULL
    DISTRIBUTION
    ...maybe
    @catswetel | @everydaykanban

    View Slide

  22. FREQUENCY
    TIME IN PROCESS
    NOT Normal

    View Slide

  23. FREQUENCY
    TIME IN PROCESS
    NOT Normal
    @catswetel | @everydaykanban

    View Slide

  24. TIME IN PROCESS
    DATE DELIVERED
    Notice anything else?

    View Slide

  25. TIME IN PROCESS
    DATE DELIVERED
    Notice anything else?

    View Slide

  26. TIME IN PROCESS
    DATE DELIVERED
    90%
    AVERAGE

    View Slide

  27. MULTI MODAL
    @catswetel | @everydaykanban

    View Slide

  28. TIME IN PROCESS
    DATE DELIVERED

    View Slide

  29. QUALITY RESPONSIVENESS
    PRODUCTIVITY PREDICTABILITY

    View Slide

  30. TIME IN PROCESS
    DATE DELIVERED
    What else can we learn?

    View Slide

  31. TIME IN PROCESS
    DATE DELIVERED
    THROUGHPUT
    Units of work per unit of time

    View Slide

  32. TIME IN PROCESS
    DATE DELIVERED
    Boxes show variability of Time in Process,
    NOT Throughput

    View Slide

  33. TIME IN PROCESS
    DATE DELIVERED
    Does it always make sense
    to count all the dots?

    View Slide

  34. TIME IN PROCESS
    DATE DELIVERED
    9 9 11
    6
    4
    6
    4
    10
    13

    View Slide

  35. TIME IN PROCESS
    DATE DELIVERED
    9 9 11
    6
    4
    6
    4
    10
    13
    1 2 4 1 0 3 1 1 1

    View Slide

  36. QUALITY RESPONSIVENESS
    PRODUCTIVITY PREDICTABILITY

    View Slide

  37. THROUGHPUT

    View Slide

  38. 85%
    50%

    View Slide

  39. “We are forced to consider
    knowledge as something that changes
    as new evidence is provided by more data,
    or as new predictions are made
    from the same data by new theories.”
    Statistical Method from the Viewpoint of Quality Control
    by Walter A Shewhart
    @catswetel | @everydaykanban

    View Slide

  40. QUALITY RESPONSIVENESS
    PRODUCTIVITY PREDICTABILITY

    View Slide

  41. Lagging quality
    @catswetel | @everydaykanban

    View Slide

  42. ESCAPED DEFECTS
    Jan
    defects found in production
    Feb Mar Apr May Jun Jul Aug Sep
    4
    5
    8 8
    10
    6
    12
    8
    11
    @catswetel | @everydaykanban
    type

    View Slide

  43. ESCAPED DEFECTS
    Jan
    defects found in production
    Feb Mar Apr May Jun Jul Aug Sep
    4
    5
    8 8
    10
    6
    12
    8
    11
    @catswetel | @everydaykanban
    type
    How can we tell if we’re
    getting better at finding
    defects earlier?

    View Slide

  44. % OF DEFECTS
    DEFECTS
    BY ENV
    percentage
    Key:
    Jan Feb Mar Apr May Jun Jul Aug Sep
    @catswetel | @everydaykanban
    STAGE
    PROD
    DEV
    INT
    env

    View Slide

  45. QUALITY RESPONSIVENESS
    PRODUCTIVITY PREDICTABILITY

    View Slide

  46. @catswetel | @everydaykanban

    View Slide

  47. View Slide

  48. Predicting the movements
    of the one bee is nearly
    impossible without the
    context of the swarm of bees.
    Complex Adaptive Systems: An Introduction
    to Computational Models of Social Life
    by John H. Miller, Scott E. Page

    View Slide

  49. @catswetel | @everydaykanban

    View Slide

  50. MORE FINISHED
    THAN STARTED
    MORE STARTED
    THAN FINISHED
    @catswetel | @everydaykanban

    View Slide

  51. @catswetel | @everydaykanban

    View Slide

  52. @catswetel | @everydaykanban

    View Slide

  53. Total
    WIP
    @catswetel | @everydaykanban

    View Slide

  54. OLDEST
    NEWEST
    @catswetel | @everydaykanban

    View Slide

  55. Average
    WIP
    @catswetel | @everydaykanban

    View Slide

  56. @catswetel | @everydaykanban

    View Slide

  57. @catswetel | @everydaykanban

    View Slide

  58. @catswetel | @everydaykanban

    View Slide

  59. LITTLE’S LAW
    @catswetel | @everydaykanban

    View Slide

  60. It’s ok to make
    necessary trade-offs

    View Slide

  61. Where should we focus
    improvement efforts?
    @catswetel | @everydaykanban

    View Slide

  62. @catswetel | @everydaykanban
    FLOW EFFICIENCY
    Work Wait
    Total Duration

    View Slide

  63. ...per state (incl. waiting states)
    @catswetel | @everydaykanban

    View Slide

  64. @catswetel | @everydaykanban
    You can do any chart for
    any portion of the process!

    View Slide

  65. Are we there yet?
    @catswetel | @everydaykanban

    View Slide

  66. QUALITY RESPONSIVENESS
    PRODUCTIVITY PREDICTABILITY

    View Slide

  67. #NOESTIMATES
    @catswetel | @everydaykanban

    View Slide

  68. “Close examination reveals that
    every meaningful interpretation
    involves a prediction.”
    Statistical Method
    from the Viewpoint of Quality Control
    Walter A Shewhart
    @catswetel | @everydaykanban

    View Slide

  69. 3 Components of Knowledge:
    1. Evidence
    2. Prediction
    3. Degree of Belief
    @catswetel | @everydaykanban

    View Slide

  70. EVIDENCE Agile Quantified (Measuring
    the impact of Agility)
    Larry Maccherone
    Wed
    3:45
    PREDICTION Forecasting and Estimation
    Hands-On
    Ozzie Yuce
    Thu
    10:45
    DEGREE OF
    BELIEF
    Data Driven Coaching
    Julia Wester & Cheryl Hammond
    Thu
    9:00
    @catswetel | @everydaykanban

    View Slide

  71. Julia Wester
    Co-Founder & Principal Consultant
    Lagom Solutions
    [email protected]
    @everydaykanban
    Cat Swetel
    Engineering Manager
    Ticketmaster
    [email protected]
    @catswetel

    View Slide

  72. Pablo Picasso
    “Computers are useless.
    They can only give you
    answers.”

    View Slide

  73. View Slide

  74. “STATISTICS are
    useless. They can only
    give you answers.”

    View Slide

  75. View Slide

  76. View Slide

  77. LITTLE’S LAW
    @catswetel at #qconnyc
    Are your decisions making
    this more or less true?
    FEEL FREE TO CUT

    View Slide

  78. Some joke about defining
    quality

    View Slide

  79. VARIABILITY
    FEEL FREE TO CUT

    View Slide

  80. TIME IN PROCESS
    DATE DELIVERED
    VARIABILITY
    FEEL FREE TO CUT

    View Slide

  81. QUALITY RESPONSIVENESS
    PRODUCTIVITY PREDICTABILITY
    VARIABILITY
    VARIABILITY
    VARIABILITY
    SHEWHART: the cause of v
    in an expected attribute. Re
    JUDGEMENT to know which
    “important quality characte
    (causes of undesirable varia
    within the bounds of reason
    expected economic results.
    FEEL FREE TO CUT

    View Slide

  82. View Slide

  83. DONE
    REVIEW
    DOING
    TO DO
    @catswetel at #qconnyc

    View Slide

  84. CUMULATIVE FLOW DIAGRAM
    @catswetel at #qconnyc

    View Slide

  85. CUMULATIVE FLOW DIAGRAM
    AVERAGE
    ARRIVAL
    RATE
    @catswetel at #qconnyc

    View Slide

  86. CUMULATIVE FLOW DIAGRAM
    AVERAGE
    DEPARTURE
    RATE
    @catswetel at #qconnyc

    View Slide

  87. CUMULATIVE FLOW DIAGRAM
    AVERAGE
    WORK IN
    PROCESS
    @catswetel at #qconnyc

    View Slide

  88. CUMULATIVE FLOW DIAGRAM
    AVERAGE TIME IN PROCESS
    @catswetel at #qconnyc

    View Slide