Metrics Driven Development

A talk given first at Nordic Ruby, then polished and refactored a bit for Scottish Ruby Conf. It talks about the process we use @paperlessdev / @paperlesspost for driving our development of features and systems with metrics and 'moving the needle'

Transcript

1. Metrics
   drIVEN
   DEVELOPMENT
   Aaron Quint / Scottish Ruby Conf / June 2012
   Friday, June 29, 12
   

   View Slide

2. In a data-less world
   Friday, June 29, 12
   

   View Slide

3. It’s me, @AQ
   HeYO
   Friday, June 29, 12
   

   View Slide

4. Friday, June 29, 12
   

   View Slide

5. • Food Obsessed (CodeFoodBar)
   Friday, June 29, 12
   

   View Slide

6. • Food Obsessed (CodeFoodBar)
   • Open Source Ruby & JS (Sammy.js, et al)
   Friday, June 29, 12
   

   View Slide

7. • Food Obsessed (CodeFoodBar)
   • Open Source Ruby & JS (Sammy.js, et al)
   • Bicoastal (Brooklyn, NY Berkeley, CA)
   Friday, June 29, 12
   

   View Slide

8. • Food Obsessed (CodeFoodBar)
   • Open Source Ruby & JS (Sammy.js, et al)
   • Bicoastal (Brooklyn, NY Berkeley, CA)
   • Very excited to be here
   Friday, June 29, 12
   

   View Slide

9. • Food Obsessed (CodeFoodBar)
   • Open Source Ruby & JS (Sammy.js, et al)
   • Bicoastal (Brooklyn, NY Berkeley, CA)
   • Very excited to be here
   • twitter: @aq
   Friday, June 29, 12
   

   View Slide

10. • Food Obsessed (CodeFoodBar)
    • Open Source Ruby & JS (Sammy.js, et al)
    • Bicoastal (Brooklyn, NY Berkeley, CA)
    • Very excited to be here
    • twitter: @aq
    • github: quirkey
    Friday, June 29, 12
    

    View Slide

11. CTO of
    Friday, June 29, 12
    

    View Slide

12. Chief Taco Officer
    CTO of
    Friday, June 29, 12
    

    View Slide

13. www.paperlesspost.com
    Friday, June 29, 12
    

    View Slide

14. Friday, June 29, 12
    

    View Slide

15. A lot of these ideas and
    implementations are theirs.
    Amazing team.
    Friday, June 29, 12
    

    View Slide

16. METRICS
    METRICS
    METRICS
    Friday, June 29, 12
    

    View Slide

17. If you’re not improving your product,
    what are you doing?
    System metrics
    ==
    PRoduct Metrics
    Friday, June 29, 12
    

    View Slide

18. The spiel
    Friday, June 29, 12
    

    View Slide

19. Measure
    Experiment
    Pivot
    Metrics
    Friday, June 29, 12
    

    View Slide

20. Measure
    Experiment
    Pivot
    Metrics
    #winning
    Friday, June 29, 12
    

    View Slide

21. WTF is going on
    MEASURE
    Friday, June 29, 12
    

    View Slide

22. Make changes,
    users are like WTF
    Experiment
    Friday, June 29, 12
    

    View Slide

23. Take the best result
    and change WTF is going on
    Pivot
    Friday, June 29, 12
    

    View Slide

24. The data you collect in
    _every_ step
    Metrics
    Friday, June 29, 12
    

    View Slide

25. The Way
    Friday, June 29, 12
    

    View Slide

26. Moving the needle
    Friday, June 29, 12
    

    View Slide

27. Continuous
    Delivery
    Friday, June 29, 12
    

    View Slide

28. Continuous
    Process
    Delivery
    Friday, June 29, 12
    

    View Slide

29. Continuous
    Process
    DeliveryTM
    Friday, June 29, 12
    

    View Slide

30. Moving the needle
    Friday, June 29, 12
    

    View Slide

31. Moving the needle
    1. Measure Everything
    Friday, June 29, 12
    

    View Slide

32. Moving the needle
    1. Measure Everything
    2. Set Goal metrics attached to each
    feature/deploy
    Friday, June 29, 12
    

    View Slide

33. Moving the needle
    1. Measure Everything
    2. Set Goal metrics attached to each
    feature/deploy
    3. Build & Roll out in chunks
    Friday, June 29, 12
    

    View Slide

34. Moving the needle
    1. Measure Everything
    2. Set Goal metrics attached to each
    feature/deploy
    3. Build & Roll out in chunks
    4. Regroup and analyze
    Friday, June 29, 12
    

    View Slide

35. Seriously, everything
    1. Measure everything
    Friday, June 29, 12
    

    View Slide

36. What happened vs What’s happening
    Absolute vs Relative
    Accounting
    vs
    Trending
    Friday, June 29, 12
    

    View Slide

37. Key Performance Indicators
    Accounting = KPI’s
    Friday, June 29, 12
    

    View Slide

38. It’s in your database
    the data is there
    Friday, June 29, 12
    

    View Slide

39. Friday, June 29, 12
    

    View Slide

40. KPI’s
    • From the existing database
    • PLPSQL
    • Run nightly, offline
    • Ad Hoc
    • Returns fixed (absolute) data, saved
    back to a database
    • About the numbers
    Friday, June 29, 12
    

    View Slide

41. Pretty pictures.
    TRends
    Friday, June 29, 12
    

    View Slide

42. Friday, June 29, 12
    

    View Slide

43. TRends
    • Redis backed
    • Low impact
    • Time series, fading resolution
    • GRAPHS
    • About the relative differences,
    actual numbers may be incorrect
    Friday, June 29, 12
    

    View Slide

44. NEW SHIT
    or new to us and maybe you
    Friday, June 29, 12
    

    View Slide

45. Statsd +
    Friday, June 29, 12
    

    View Slide

46. Statsd +
    Graphite +
    Friday, June 29, 12
    

    View Slide

47. graphiti
    Statsd +
    Graphite +
    Friday, June 29, 12
    

    View Slide

48. gh://etsy/statsd
    gh://paperlesspost/statsd
    gh://github/statsd-ruby
    Statsd
    Friday, June 29, 12
    

    View Slide

49. 1 Paperless::Statsd.increment "dashboard.logins"
    2 Paperless::Statsd.timing "media_file.convert", time_in_ms
    Friday, June 29, 12
    

    View Slide

50. Friday, June 29, 12
    

    View Slide

51. gh://quirkey/resque-metrics
    Friday, June 29, 12
    

    View Slide

52. An alternate frontend for graphite,
    optimized for play and discovery
    gh://paperlesspost/graphiti
    Graphiti
    Friday, June 29, 12
    

    View Slide

53. Friday, June 29, 12
    

    View Slide

54. Graphiti
    Friday, June 29, 12
    

    View Slide

55. Graphiti
    • Sinatra, Sammy, Redis
    Friday, June 29, 12
    

    View Slide

56. Graphiti
    • Sinatra, Sammy, Redis
    • Graphs stored as JSON in Redis
    Friday, June 29, 12
    

    View Slide

57. Graphiti
    • Sinatra, Sammy, Redis
    • Graphs stored as JSON in Redis
    • Ace Editor for Play/editing
    Friday, June 29, 12
    

    View Slide

58. Graphiti
    • Sinatra, Sammy, Redis
    • Graphs stored as JSON in Redis
    • Ace Editor for Play/editing
    • Named Dashboards are collections
    of graphs
    Friday, June 29, 12
    

    View Slide

59. Graphiti
    • Sinatra, Sammy, Redis
    • Graphs stored as JSON in Redis
    • Ace Editor for Play/editing
    • Named Dashboards are collections
    of graphs
    • WIP
    Friday, June 29, 12
    

    View Slide

60. Friday, June 29, 12
    

    View Slide

61. Friday, June 29, 12
    

    View Slide

62. Attached to Specs/Issues
    2. SET Goal metrics
    Friday, June 29, 12
    

    View Slide

63. Terminology TANGENT
    Friday, June 29, 12
    

    View Slide

64. Terminology TANGENT
    Friday, June 29, 12
    

    View Slide

65. • Accounts create Events
    Terminology TANGENT
    Friday, June 29, 12
    

    View Slide

66. • Accounts create Events
    • Events have a Card
    Terminology TANGENT
    Friday, June 29, 12
    

    View Slide

67. • Accounts create Events
    • Events have a Card
    • Card [customizations] are paid for
    with Coins
    Terminology TANGENT
    Friday, June 29, 12
    

    View Slide

68. • Accounts create Events
    • Events have a Card
    • Card [customizations] are paid for
    with Coins
    • Events send to Guests
    Terminology TANGENT
    Friday, June 29, 12
    

    View Slide

69. • Accounts create Events
    • Events have a Card
    • Card [customizations] are paid for
    with Coins
    • Events send to Guests
    • Guests Reply to Events
    Terminology TANGENT
    Friday, June 29, 12
    

    View Slide

70. Quantitative
    vs
    Qualitative
    Friday, June 29, 12
    

    View Slide

71. How good your product is.
    qualitative
    Friday, June 29, 12
    

    View Slide

72. Events created per Account
    Guests added per Event
    Coins used per Event
    A single value
    measured on
    a per n axis
    Friday, June 29, 12
    

    View Slide

73. Quality
    is the
    velocity
    of our product
    growth
    Friday, June 29, 12
    

    View Slide

74. How big your funnel is.
    Quantitative
    Friday, June 29, 12
    

    View Slide

75. % of Accounts that enter the create tool
    % of Guests that leave a comment
    % of Events sent with coins
    A number value
    measured in a %
    change over time
    Friday, June 29, 12
    

    View Slide

76. QUANTITY
    is the
    ACCELERATION
    of our product
    growth
    Friday, June 29, 12
    

    View Slide

77. Quantitative
    vs
    Qualitative
    Friday, June 29, 12
    

    View Slide

78. Quantitative
    X
    Qualitative
    GROWTH
    Friday, June 29, 12
    

    View Slide

79. A feature can touch both,
    but in practice it doesn’t.
    Pick ONE type
    And a FEW
    metrics
    Friday, June 29, 12
    

    View Slide

80. Attach them
    to the spec
    or issue
    Friday, June 29, 12
    

    View Slide

81. Friday, June 29, 12
    

    View Slide

82. Friday, June 29, 12
    

    View Slide

83. Duh, you guys know Ruby, this part is easy.
    3. Build &
    Roll OUt in Chunks
    Friday, June 29, 12
    

    View Slide

84. if current_account_has_feature? :hybrid
    # show the hybrid
    Friday, June 29, 12
    

    View Slide

85. Friday, June 29, 12
    

    View Slide

86. Friday, June 29, 12
    

    View Slide

87. Daily, Weekly, Obsessive Checking
    Report along the way
    Friday, June 29, 12
    

    View Slide

88. Friday, June 29, 12
    

    View Slide

89. Was our hunch correct.
    4. roundup and report
    Friday, June 29, 12
    

    View Slide

90. And no negative change.
    Numbers are important
    but most important is
    any positive change
    Friday, June 29, 12
    

    View Slide

91. Making sure everyone knows what did go
    right, and what opportunities there are to
    improve.
    Roundup meetings
    Friday, June 29, 12
    

    View Slide

92. 5. ProfIt?
    Friday, June 29, 12
    

    View Slide

93. Still early days
    Friday, June 29, 12
    

    View Slide

94. The good
    Friday, June 29, 12
    

    View Slide

95. The good
    • For certain features, it can bring
    laser focus to the goals.
    Friday, June 29, 12
    

    View Slide

96. The good
    • For certain features, it can bring
    laser focus to the goals.
    • The separation of the Q’s has
    brought a lot of clarity to what we’re
    actually trying to change.
    Friday, June 29, 12
    

    View Slide

97. The good
    • For certain features, it can bring
    laser focus to the goals.
    • The separation of the Q’s has
    brought a lot of clarity to what we’re
    actually trying to change.
    • Can inspire and give a clear goal to
    developers.
    Friday, June 29, 12
    

    View Slide

98. The opportunities
    Friday, June 29, 12
    

    View Slide

99. The opportunities
    • Can be dangerous if people focus
    on the wrong goals.
    Friday, June 29, 12
    

    View Slide

100. The opportunities
     • Can be dangerous if people focus
     on the wrong goals.
     • Hard to actually test features in
     isolation.
     Friday, June 29, 12
     

     View Slide

101. The opportunities
     • Can be dangerous if people focus
     on the wrong goals.
     • Hard to actually test features in
     isolation.
     • Some metrics are much harder to
     measure accurately.
     Friday, June 29, 12
     

     View Slide

102. The opportunities
     • Can be dangerous if people focus
     on the wrong goals.
     • Hard to actually test features in
     isolation.
     • Some metrics are much harder to
     measure accurately.
     • Teams need more mid-process
     visibility in to goals.
     Friday, June 29, 12
     

     View Slide

103. We’re gathering
     metrics about how
     metrics driven
     works NOW
     Friday, June 29, 12
     

     View Slide

104. IT’s going pretty well.
     Friday, June 29, 12
     

     View Slide

105. github.com/quirkey
     twitter.com/aq
     quirkey.com
     paperlesspost.com
     THANKS!
     Friday, June 29, 12
     

     View Slide