Metrics Driven Development

Metrics Driven Development

A talk given first at Nordic Ruby, then polished and refactored a bit for Scottish Ruby Conf. It talks about the process we use @paperlessdev / @paperlesspost for driving our development of features and systems with metrics and 'moving the needle'

F04bfa14141dca6713f0d9caa763e26b?s=128

Aaron Quint

June 29, 2012
Tweet

Transcript

  1. 6.

    • Food Obsessed (CodeFoodBar) • Open Source Ruby & JS

    (Sammy.js, et al) Friday, June 29, 12
  2. 7.

    • Food Obsessed (CodeFoodBar) • Open Source Ruby & JS

    (Sammy.js, et al) • Bicoastal (Brooklyn, NY <-> Berkeley, CA) Friday, June 29, 12
  3. 8.

    • Food Obsessed (CodeFoodBar) • Open Source Ruby & JS

    (Sammy.js, et al) • Bicoastal (Brooklyn, NY <-> Berkeley, CA) • Very excited to be here Friday, June 29, 12
  4. 9.

    • Food Obsessed (CodeFoodBar) • Open Source Ruby & JS

    (Sammy.js, et al) • Bicoastal (Brooklyn, NY <-> Berkeley, CA) • Very excited to be here • twitter: @aq Friday, June 29, 12
  5. 10.

    • Food Obsessed (CodeFoodBar) • Open Source Ruby & JS

    (Sammy.js, et al) • Bicoastal (Brooklyn, NY <-> Berkeley, CA) • Very excited to be here • twitter: @aq • github: quirkey Friday, June 29, 12
  6. 17.

    If you’re not improving your product, what are you doing?

    System metrics == PRoduct Metrics Friday, June 29, 12
  7. 23.
  8. 32.

    Moving the needle 1. Measure Everything 2. Set Goal metrics

    attached to each feature/deploy Friday, June 29, 12
  9. 33.

    Moving the needle 1. Measure Everything 2. Set Goal metrics

    attached to each feature/deploy 3. Build & Roll out in chunks Friday, June 29, 12
  10. 34.

    Moving the needle 1. Measure Everything 2. Set Goal metrics

    attached to each feature/deploy 3. Build & Roll out in chunks 4. Regroup and analyze Friday, June 29, 12
  11. 40.

    KPI’s • From the existing database • PLPSQL • Run

    nightly, offline • Ad Hoc • Returns fixed (absolute) data, saved back to a database • About the numbers Friday, June 29, 12
  12. 43.

    TRends • Redis backed • Low impact • Time series,

    fading resolution • GRAPHS • About the relative differences, actual numbers may be incorrect Friday, June 29, 12
  13. 52.

    An alternate frontend for graphite, optimized for play and discovery

    gh://paperlesspost/graphiti Graphiti Friday, June 29, 12
  14. 57.

    Graphiti • Sinatra, Sammy, Redis • Graphs stored as JSON

    in Redis • Ace Editor for Play/editing Friday, June 29, 12
  15. 58.

    Graphiti • Sinatra, Sammy, Redis • Graphs stored as JSON

    in Redis • Ace Editor for Play/editing • Named Dashboards are collections of graphs Friday, June 29, 12
  16. 59.

    Graphiti • Sinatra, Sammy, Redis • Graphs stored as JSON

    in Redis • Ace Editor for Play/editing • Named Dashboards are collections of graphs • WIP Friday, June 29, 12
  17. 67.

    • Accounts create Events • Events have a Card •

    Card [customizations] are paid for with Coins Terminology TANGENT Friday, June 29, 12
  18. 68.

    • Accounts create Events • Events have a Card •

    Card [customizations] are paid for with Coins • Events send to Guests Terminology TANGENT Friday, June 29, 12
  19. 69.

    • Accounts create Events • Events have a Card •

    Card [customizations] are paid for with Coins • Events send to Guests • Guests Reply to Events Terminology TANGENT Friday, June 29, 12
  20. 72.

    Events created per Account Guests added per Event Coins used

    per Event A single value measured on a per n axis Friday, June 29, 12
  21. 75.

    % of Accounts that enter the create tool % of

    Guests that leave a comment % of Events sent with coins A number value measured in a % change over time Friday, June 29, 12
  22. 79.

    A feature can touch both, but in practice it doesn’t.

    Pick ONE type And a FEW metrics Friday, June 29, 12
  23. 83.

    Duh, you guys know Ruby, this part is easy. 3.

    Build & Roll OUt in Chunks Friday, June 29, 12
  24. 90.

    And no negative change. Numbers are important but most important

    is any positive change Friday, June 29, 12
  25. 91.

    Making sure everyone knows what did go right, and what

    opportunities there are to improve. Roundup meetings Friday, June 29, 12
  26. 95.

    The good • For certain features, it can bring laser

    focus to the goals. Friday, June 29, 12
  27. 96.

    The good • For certain features, it can bring laser

    focus to the goals. • The separation of the Q’s has brought a lot of clarity to what we’re actually trying to change. Friday, June 29, 12
  28. 97.

    The good • For certain features, it can bring laser

    focus to the goals. • The separation of the Q’s has brought a lot of clarity to what we’re actually trying to change. • Can inspire and give a clear goal to developers. Friday, June 29, 12
  29. 99.

    The opportunities • Can be dangerous if people focus on

    the wrong goals. Friday, June 29, 12
  30. 100.

    The opportunities • Can be dangerous if people focus on

    the wrong goals. • Hard to actually test features in isolation. Friday, June 29, 12
  31. 101.

    The opportunities • Can be dangerous if people focus on

    the wrong goals. • Hard to actually test features in isolation. • Some metrics are much harder to measure accurately. Friday, June 29, 12
  32. 102.

    The opportunities • Can be dangerous if people focus on

    the wrong goals. • Hard to actually test features in isolation. • Some metrics are much harder to measure accurately. • Teams need more mid-process visibility in to goals. Friday, June 29, 12