Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Metrics Driven Development

Metrics Driven Development

A talk given first at Nordic Ruby, then polished and refactored a bit for Scottish Ruby Conf. It talks about the process we use @paperlessdev / @paperlesspost for driving our development of features and systems with metrics and 'moving the needle'

Aaron Quint

June 29, 2012
Tweet

More Decks by Aaron Quint

Other Decks in Technology

Transcript

  1. Metrics
    drIVEN
    DEVELOPMENT
    Aaron Quint / Scottish Ruby Conf / June 2012
    Friday, June 29, 12

    View full-size slide

  2. In a data-less world
    Friday, June 29, 12

    View full-size slide

  3. It’s me, @AQ
    HeYO
    Friday, June 29, 12

    View full-size slide

  4. Friday, June 29, 12

    View full-size slide

  5. • Food Obsessed (CodeFoodBar)
    Friday, June 29, 12

    View full-size slide

  6. • Food Obsessed (CodeFoodBar)
    • Open Source Ruby & JS (Sammy.js, et al)
    Friday, June 29, 12

    View full-size slide

  7. • Food Obsessed (CodeFoodBar)
    • Open Source Ruby & JS (Sammy.js, et al)
    • Bicoastal (Brooklyn, NY <-> Berkeley, CA)
    Friday, June 29, 12

    View full-size slide

  8. • Food Obsessed (CodeFoodBar)
    • Open Source Ruby & JS (Sammy.js, et al)
    • Bicoastal (Brooklyn, NY <-> Berkeley, CA)
    • Very excited to be here
    Friday, June 29, 12

    View full-size slide

  9. • Food Obsessed (CodeFoodBar)
    • Open Source Ruby & JS (Sammy.js, et al)
    • Bicoastal (Brooklyn, NY <-> Berkeley, CA)
    • Very excited to be here
    • twitter: @aq
    Friday, June 29, 12

    View full-size slide

  10. • Food Obsessed (CodeFoodBar)
    • Open Source Ruby & JS (Sammy.js, et al)
    • Bicoastal (Brooklyn, NY <-> Berkeley, CA)
    • Very excited to be here
    • twitter: @aq
    • github: quirkey
    Friday, June 29, 12

    View full-size slide

  11. CTO of
    Friday, June 29, 12

    View full-size slide

  12. Chief Taco Officer
    CTO of
    Friday, June 29, 12

    View full-size slide

  13. www.paperlesspost.com
    Friday, June 29, 12

    View full-size slide

  14. Friday, June 29, 12

    View full-size slide

  15. A lot of these ideas and
    implementations are theirs.
    Amazing team.
    Friday, June 29, 12

    View full-size slide

  16. METRICS
    METRICS
    METRICS
    Friday, June 29, 12

    View full-size slide

  17. If you’re not improving your product,
    what are you doing?
    System metrics
    ==
    PRoduct Metrics
    Friday, June 29, 12

    View full-size slide

  18. The spiel
    Friday, June 29, 12

    View full-size slide

  19. Measure
    Experiment
    Pivot
    Metrics
    Friday, June 29, 12

    View full-size slide

  20. Measure
    Experiment
    Pivot
    Metrics
    #winning
    Friday, June 29, 12

    View full-size slide

  21. WTF is going on
    MEASURE
    Friday, June 29, 12

    View full-size slide

  22. Make changes,
    users are like WTF
    Experiment
    Friday, June 29, 12

    View full-size slide

  23. Take the best result
    and change WTF is going on
    Pivot
    Friday, June 29, 12

    View full-size slide

  24. The data you collect in
    _every_ step
    Metrics
    Friday, June 29, 12

    View full-size slide

  25. The Way
    Friday, June 29, 12

    View full-size slide

  26. Moving the needle
    Friday, June 29, 12

    View full-size slide

  27. Continuous
    Delivery
    Friday, June 29, 12

    View full-size slide

  28. Continuous
    Process
    Delivery
    Friday, June 29, 12

    View full-size slide

  29. Continuous
    Process
    DeliveryTM
    Friday, June 29, 12

    View full-size slide

  30. Moving the needle
    Friday, June 29, 12

    View full-size slide

  31. Moving the needle
    1. Measure Everything
    Friday, June 29, 12

    View full-size slide

  32. Moving the needle
    1. Measure Everything
    2. Set Goal metrics attached to each
    feature/deploy
    Friday, June 29, 12

    View full-size slide

  33. Moving the needle
    1. Measure Everything
    2. Set Goal metrics attached to each
    feature/deploy
    3. Build & Roll out in chunks
    Friday, June 29, 12

    View full-size slide

  34. Moving the needle
    1. Measure Everything
    2. Set Goal metrics attached to each
    feature/deploy
    3. Build & Roll out in chunks
    4. Regroup and analyze
    Friday, June 29, 12

    View full-size slide

  35. Seriously, everything
    1. Measure everything
    Friday, June 29, 12

    View full-size slide

  36. What happened vs What’s happening
    Absolute vs Relative
    Accounting
    vs
    Trending
    Friday, June 29, 12

    View full-size slide

  37. Key Performance Indicators
    Accounting = KPI’s
    Friday, June 29, 12

    View full-size slide

  38. It’s in your database
    the data is there
    Friday, June 29, 12

    View full-size slide

  39. Friday, June 29, 12

    View full-size slide

  40. KPI’s
    • From the existing database
    • PLPSQL
    • Run nightly, offline
    • Ad Hoc
    • Returns fixed (absolute) data, saved
    back to a database
    • About the numbers
    Friday, June 29, 12

    View full-size slide

  41. Pretty pictures.
    TRends
    Friday, June 29, 12

    View full-size slide

  42. Friday, June 29, 12

    View full-size slide

  43. TRends
    • Redis backed
    • Low impact
    • Time series, fading resolution
    • GRAPHS
    • About the relative differences,
    actual numbers may be incorrect
    Friday, June 29, 12

    View full-size slide

  44. NEW SHIT
    or new to us and maybe you
    Friday, June 29, 12

    View full-size slide

  45. Statsd +
    Friday, June 29, 12

    View full-size slide

  46. Statsd +
    Graphite +
    Friday, June 29, 12

    View full-size slide

  47. graphiti
    Statsd +
    Graphite +
    Friday, June 29, 12

    View full-size slide

  48. gh://etsy/statsd
    gh://paperlesspost/statsd
    gh://github/statsd-ruby
    Statsd
    Friday, June 29, 12

    View full-size slide

  49. 1 Paperless::Statsd.increment "dashboard.logins"
    2 Paperless::Statsd.timing "media_file.convert", time_in_ms
    Friday, June 29, 12

    View full-size slide

  50. Friday, June 29, 12

    View full-size slide

  51. gh://quirkey/resque-metrics
    Friday, June 29, 12

    View full-size slide

  52. An alternate frontend for graphite,
    optimized for play and discovery
    gh://paperlesspost/graphiti
    Graphiti
    Friday, June 29, 12

    View full-size slide

  53. Friday, June 29, 12

    View full-size slide

  54. Graphiti
    Friday, June 29, 12

    View full-size slide

  55. Graphiti
    • Sinatra, Sammy, Redis
    Friday, June 29, 12

    View full-size slide

  56. Graphiti
    • Sinatra, Sammy, Redis
    • Graphs stored as JSON in Redis
    Friday, June 29, 12

    View full-size slide

  57. Graphiti
    • Sinatra, Sammy, Redis
    • Graphs stored as JSON in Redis
    • Ace Editor for Play/editing
    Friday, June 29, 12

    View full-size slide

  58. Graphiti
    • Sinatra, Sammy, Redis
    • Graphs stored as JSON in Redis
    • Ace Editor for Play/editing
    • Named Dashboards are collections
    of graphs
    Friday, June 29, 12

    View full-size slide

  59. Graphiti
    • Sinatra, Sammy, Redis
    • Graphs stored as JSON in Redis
    • Ace Editor for Play/editing
    • Named Dashboards are collections
    of graphs
    • WIP
    Friday, June 29, 12

    View full-size slide

  60. Friday, June 29, 12

    View full-size slide

  61. Friday, June 29, 12

    View full-size slide

  62. Attached to Specs/Issues
    2. SET Goal metrics
    Friday, June 29, 12

    View full-size slide

  63. Terminology TANGENT
    Friday, June 29, 12

    View full-size slide

  64. Terminology TANGENT
    Friday, June 29, 12

    View full-size slide

  65. • Accounts create Events
    Terminology TANGENT
    Friday, June 29, 12

    View full-size slide

  66. • Accounts create Events
    • Events have a Card
    Terminology TANGENT
    Friday, June 29, 12

    View full-size slide

  67. • Accounts create Events
    • Events have a Card
    • Card [customizations] are paid for
    with Coins
    Terminology TANGENT
    Friday, June 29, 12

    View full-size slide

  68. • Accounts create Events
    • Events have a Card
    • Card [customizations] are paid for
    with Coins
    • Events send to Guests
    Terminology TANGENT
    Friday, June 29, 12

    View full-size slide

  69. • Accounts create Events
    • Events have a Card
    • Card [customizations] are paid for
    with Coins
    • Events send to Guests
    • Guests Reply to Events
    Terminology TANGENT
    Friday, June 29, 12

    View full-size slide

  70. Quantitative
    vs
    Qualitative
    Friday, June 29, 12

    View full-size slide

  71. How good your product is.
    qualitative
    Friday, June 29, 12

    View full-size slide

  72. Events created per Account
    Guests added per Event
    Coins used per Event
    A single value
    measured on
    a per n axis
    Friday, June 29, 12

    View full-size slide

  73. Quality
    is the
    velocity
    of our product
    growth
    Friday, June 29, 12

    View full-size slide

  74. How big your funnel is.
    Quantitative
    Friday, June 29, 12

    View full-size slide

  75. % of Accounts that enter the create tool
    % of Guests that leave a comment
    % of Events sent with coins
    A number value
    measured in a %
    change over time
    Friday, June 29, 12

    View full-size slide

  76. QUANTITY
    is the
    ACCELERATION
    of our product
    growth
    Friday, June 29, 12

    View full-size slide

  77. Quantitative
    vs
    Qualitative
    Friday, June 29, 12

    View full-size slide

  78. Quantitative
    X
    Qualitative
    GROWTH
    Friday, June 29, 12

    View full-size slide

  79. A feature can touch both,
    but in practice it doesn’t.
    Pick ONE type
    And a FEW
    metrics
    Friday, June 29, 12

    View full-size slide

  80. Attach them
    to the spec
    or issue
    Friday, June 29, 12

    View full-size slide

  81. Friday, June 29, 12

    View full-size slide

  82. Friday, June 29, 12

    View full-size slide

  83. Duh, you guys know Ruby, this part is easy.
    3. Build &
    Roll OUt in Chunks
    Friday, June 29, 12

    View full-size slide

  84. if current_account_has_feature? :hybrid
    # show the hybrid
    Friday, June 29, 12

    View full-size slide

  85. Friday, June 29, 12

    View full-size slide

  86. Friday, June 29, 12

    View full-size slide

  87. Daily, Weekly, Obsessive Checking
    Report along the way
    Friday, June 29, 12

    View full-size slide

  88. Friday, June 29, 12

    View full-size slide

  89. Was our hunch correct.
    4. roundup and report
    Friday, June 29, 12

    View full-size slide

  90. And no negative change.
    Numbers are important
    but most important is
    any positive change
    Friday, June 29, 12

    View full-size slide

  91. Making sure everyone knows what did go
    right, and what opportunities there are to
    improve.
    Roundup meetings
    Friday, June 29, 12

    View full-size slide

  92. 5. ProfIt?
    Friday, June 29, 12

    View full-size slide

  93. Still early days
    Friday, June 29, 12

    View full-size slide

  94. The good
    Friday, June 29, 12

    View full-size slide

  95. The good
    • For certain features, it can bring
    laser focus to the goals.
    Friday, June 29, 12

    View full-size slide

  96. The good
    • For certain features, it can bring
    laser focus to the goals.
    • The separation of the Q’s has
    brought a lot of clarity to what we’re
    actually trying to change.
    Friday, June 29, 12

    View full-size slide

  97. The good
    • For certain features, it can bring
    laser focus to the goals.
    • The separation of the Q’s has
    brought a lot of clarity to what we’re
    actually trying to change.
    • Can inspire and give a clear goal to
    developers.
    Friday, June 29, 12

    View full-size slide

  98. The opportunities
    Friday, June 29, 12

    View full-size slide

  99. The opportunities
    • Can be dangerous if people focus
    on the wrong goals.
    Friday, June 29, 12

    View full-size slide

  100. The opportunities
    • Can be dangerous if people focus
    on the wrong goals.
    • Hard to actually test features in
    isolation.
    Friday, June 29, 12

    View full-size slide

  101. The opportunities
    • Can be dangerous if people focus
    on the wrong goals.
    • Hard to actually test features in
    isolation.
    • Some metrics are much harder to
    measure accurately.
    Friday, June 29, 12

    View full-size slide

  102. The opportunities
    • Can be dangerous if people focus
    on the wrong goals.
    • Hard to actually test features in
    isolation.
    • Some metrics are much harder to
    measure accurately.
    • Teams need more mid-process
    visibility in to goals.
    Friday, June 29, 12

    View full-size slide

  103. We’re gathering
    metrics about how
    metrics driven
    works NOW
    Friday, June 29, 12

    View full-size slide

  104. IT’s going pretty well.
    Friday, June 29, 12

    View full-size slide

  105. github.com/quirkey
    twitter.com/aq
    quirkey.com
    paperlesspost.com
    THANKS!
    Friday, June 29, 12

    View full-size slide