Dev and Test Metrics 101 - Agile 2018

Cat Swetel, accompanied by Julia Wester, will outline the most common metrics used for understanding flow and impediments in software development and test functions. This session should provide a clear picture of the types of metrics that help understand Agile development, and how to get started in capturing and using these for improving Agile adoption, improvement, and management.

Transcript

1. Development and Test
   Metrics 101
   @catswetel | @everydaykanban
   

   View Slide

2. @catswetel | @everydaykanban
   

   View Slide

3. QUALITY RESPONSIVENESS
   PRODUCTIVITY PREDICTABILITY
   

   View Slide

4. QUALITY RESPONSIVENESS
   PRODUCTIVITY PREDICTABILITY
   Software Development
   Performance Index (SDPI)
   Larry Maccherone
   

   View Slide

5. QUALITY RESPONSIVENESS
   PRODUCTIVITY PREDICTABILITY
   

   View Slide

6. QUALITY RESPONSIVENESS
   PRODUCTIVITY PREDICTABILITY
   What’s
   missing?
   

   View Slide

7. QUALITY RESPONSIVENESS
   PRODUCTIVITY PREDICTABILITY
   

   View Slide

8. “The RIGHTER we do the
   WRONG thing, the
   WRONGER we become.”
   -- Dr Russell Ackoff
   @catswetel | @everydaykanban
   

   View Slide

9. QUALITY RESPONSIVENESS
   PRODUCTIVITY PREDICTABILITY
   QUALITY
   RESPONSIVENESS
   @catswetel | @everydaykanban
   

   View Slide

10. @catswetel | @everydaykanban
    

    View Slide

11. Time in Process
    Units of time per unit of work
    @catswetel | @everydaykanban
    

    View Slide

12. TIME IN PROCESS
    DATE DELIVERED
    TIME IN PROCESS SCATTER PLOT
    

    View Slide

13. TIME IN PROCESS
    DATE DELIVERED
    When can we expect X?
    90%
    50%
    

    View Slide

14. TIME IN PROCESS
    DATE DELIVERED
    How likely by X?
    30d: 77% 21d: 50%
    

    View Slide

15. TIME IN PROCESS
    DATE DELIVERED
    What’s the story?
    

    View Slide

16. QUALITY RESPONSIVENESS
    PRODUCTIVITY PREDICTABILITY
    

    View Slide

17. FREQUENCY
    TIME IN PROCESS
    TIME IN PROCESS
    DISTRIBUTION
    Groups are
    easier to see
    This one happens
    more often
    

    View Slide

18. QUALITY RESPONSIVENESS
    PRODUCTIVITY PREDICTABILITY
    

    View Slide

19. @catswetel | @everydaykanban
    

    View Slide

20. WEIBULL
    DISTRIBUTION
    @catswetel | @everydaykanban
    

    View Slide

21. WEIBULL
    DISTRIBUTION
    ...maybe
    @catswetel | @everydaykanban
    

    View Slide

22. FREQUENCY
    TIME IN PROCESS
    NOT Normal
    

    View Slide

23. FREQUENCY
    TIME IN PROCESS
    NOT Normal
    @catswetel | @everydaykanban
    

    View Slide

24. TIME IN PROCESS
    DATE DELIVERED
    Notice anything else?
    

    View Slide

25. TIME IN PROCESS
    DATE DELIVERED
    Notice anything else?
    

    View Slide

26. TIME IN PROCESS
    DATE DELIVERED
    90%
    AVERAGE
    

    View Slide

27. MULTI MODAL
    @catswetel | @everydaykanban
    

    View Slide

28. TIME IN PROCESS
    DATE DELIVERED
    

    View Slide

29. QUALITY RESPONSIVENESS
    PRODUCTIVITY PREDICTABILITY
    

    View Slide

30. TIME IN PROCESS
    DATE DELIVERED
    What else can we learn?
    

    View Slide

31. TIME IN PROCESS
    DATE DELIVERED
    THROUGHPUT
    Units of work per unit of time
    

    View Slide

32. TIME IN PROCESS
    DATE DELIVERED
    Boxes show variability of Time in Process,
    NOT Throughput
    

    View Slide

33. TIME IN PROCESS
    DATE DELIVERED
    Does it always make sense
    to count all the dots?
    

    View Slide

34. TIME IN PROCESS
    DATE DELIVERED
    9 9 11
    6
    4
    6
    4
    10
    13
    

    View Slide

35. TIME IN PROCESS
    DATE DELIVERED
    9 9 11
    6
    4
    6
    4
    10
    13
    1 2 4 1 0 3 1 1 1
    

    View Slide

36. QUALITY RESPONSIVENESS
    PRODUCTIVITY PREDICTABILITY
    

    View Slide

37. THROUGHPUT
    

    View Slide

38. 85%
    50%
    

    View Slide

39. “We are forced to consider
    knowledge as something that changes
    as new evidence is provided by more data,
    or as new predictions are made
    from the same data by new theories.”
    Statistical Method from the Viewpoint of Quality Control
    by Walter A Shewhart
    @catswetel | @everydaykanban
    

    View Slide

40. QUALITY RESPONSIVENESS
    PRODUCTIVITY PREDICTABILITY
    

    View Slide

41. Lagging quality
    @catswetel | @everydaykanban
    

    View Slide

42. ESCAPED DEFECTS
    Jan
    defects found in production
    Feb Mar Apr May Jun Jul Aug Sep
    4
    5
    8 8
    10
    6
    12
    8
    11
    @catswetel | @everydaykanban
    type
    

    View Slide

43. ESCAPED DEFECTS
    Jan
    defects found in production
    Feb Mar Apr May Jun Jul Aug Sep
    4
    5
    8 8
    10
    6
    12
    8
    11
    @catswetel | @everydaykanban
    type
    How can we tell if we’re
    getting better at finding
    defects earlier?
    

    View Slide

44. % OF DEFECTS
    DEFECTS
    BY ENV
    percentage
    Key:
    Jan Feb Mar Apr May Jun Jul Aug Sep
    @catswetel | @everydaykanban
    STAGE
    PROD
    DEV
    INT
    env
    

    View Slide

45. QUALITY RESPONSIVENESS
    PRODUCTIVITY PREDICTABILITY
    

    View Slide

46. @catswetel | @everydaykanban
    

    View Slide

47. View Slide

48. Predicting the movements
    of the one bee is nearly
    impossible without the
    context of the swarm of bees.
    Complex Adaptive Systems: An Introduction
    to Computational Models of Social Life
    by John H. Miller, Scott E. Page
    

    View Slide

49. @catswetel | @everydaykanban
    

    View Slide

50. MORE FINISHED
    THAN STARTED
    MORE STARTED
    THAN FINISHED
    @catswetel | @everydaykanban
    

    View Slide

51. @catswetel | @everydaykanban
    

    View Slide

52. @catswetel | @everydaykanban
    

    View Slide

53. Total
    WIP
    @catswetel | @everydaykanban
    

    View Slide

54. OLDEST
    NEWEST
    @catswetel | @everydaykanban
    

    View Slide

55. Average
    WIP
    @catswetel | @everydaykanban
    

    View Slide

56. @catswetel | @everydaykanban
    

    View Slide

57. @catswetel | @everydaykanban
    

    View Slide

58. @catswetel | @everydaykanban
    

    View Slide

59. LITTLE’S LAW
    @catswetel | @everydaykanban
    

    View Slide

60. It’s ok to make
    necessary trade-offs
    

    View Slide

61. Where should we focus
    improvement efforts?
    @catswetel | @everydaykanban
    

    View Slide

62. @catswetel | @everydaykanban
    FLOW EFFICIENCY
    Work Wait
    Total Duration
    

    View Slide

63. ...per state (incl. waiting states)
    @catswetel | @everydaykanban
    

    View Slide

64. @catswetel | @everydaykanban
    You can do any chart for
    any portion of the process!
    

    View Slide

65. Are we there yet?
    @catswetel | @everydaykanban
    

    View Slide

66. QUALITY RESPONSIVENESS
    PRODUCTIVITY PREDICTABILITY
    

    View Slide

67. #NOESTIMATES
    @catswetel | @everydaykanban
    

    View Slide

68. “Close examination reveals that
    every meaningful interpretation
    involves a prediction.”
    Statistical Method
    from the Viewpoint of Quality Control
    Walter A Shewhart
    @catswetel | @everydaykanban
    

    View Slide

69. 3 Components of Knowledge:
    1. Evidence
    2. Prediction
    3. Degree of Belief
    @catswetel | @everydaykanban
    

    View Slide

70. EVIDENCE Agile Quantified (Measuring
    the impact of Agility)
    Larry Maccherone
    Wed
    3:45
    PREDICTION Forecasting and Estimation
    Hands-On
    Ozzie Yuce
    Thu
    10:45
    DEGREE OF
    BELIEF
    Data Driven Coaching
    Julia Wester & Cheryl Hammond
    Thu
    9:00
    @catswetel | @everydaykanban
    

    View Slide

71. Julia Wester
    Co-Founder & Principal Consultant
    Lagom Solutions
    julia@lagom.solutions
    @everydaykanban
    Cat Swetel
    Engineering Manager
    Ticketmaster
    cat@catswetel.com
    @catswetel
    

    View Slide

72. Pablo Picasso
    “Computers are useless.
    They can only give you
    answers.”
    

    View Slide

73. View Slide

74. “STATISTICS are
    useless. They can only
    give you answers.”
    

    View Slide

75. View Slide

76. View Slide

77. LITTLE’S LAW
    @catswetel at #qconnyc
    Are your decisions making
    this more or less true?
    FEEL FREE TO CUT
    

    View Slide

78. Some joke about defining
    quality
    

    View Slide

79. VARIABILITY
    FEEL FREE TO CUT
    

    View Slide

80. TIME IN PROCESS
    DATE DELIVERED
    VARIABILITY
    FEEL FREE TO CUT
    

    View Slide

81. QUALITY RESPONSIVENESS
    PRODUCTIVITY PREDICTABILITY
    VARIABILITY
    VARIABILITY
    VARIABILITY
    SHEWHART: the cause of v
    in an expected attribute. Re
    JUDGEMENT to know which
    “important quality characte
    (causes of undesirable varia
    within the bounds of reason
    expected economic results.
    FEEL FREE TO CUT
    

    View Slide

82. View Slide

83. DONE
    REVIEW
    DOING
    TO DO
    @catswetel at #qconnyc
    

    View Slide

84. CUMULATIVE FLOW DIAGRAM
    @catswetel at #qconnyc
    

    View Slide

85. CUMULATIVE FLOW DIAGRAM
    AVERAGE
    ARRIVAL
    RATE
    @catswetel at #qconnyc
    

    View Slide

86. CUMULATIVE FLOW DIAGRAM
    AVERAGE
    DEPARTURE
    RATE
    @catswetel at #qconnyc
    

    View Slide

87. CUMULATIVE FLOW DIAGRAM
    AVERAGE
    WORK IN
    PROCESS
    @catswetel at #qconnyc
    

    View Slide

88. CUMULATIVE FLOW DIAGRAM
    AVERAGE TIME IN PROCESS
    @catswetel at #qconnyc
    

    View Slide