Tweet
Tweet

Slide 1

Slide 1 text

Metrics drIVEN DEVELOPMENT Aaron Quint / Scottish Ruby Conf / June 2012 Friday, June 29, 12

Slide 2

Slide 2 text

In a data-less world Friday, June 29, 12

Slide 3

Slide 3 text

It’s me, @AQ HeYO Friday, June 29, 12

Slide 4

Slide 4 text

Friday, June 29, 12

Slide 5

Slide 5 text

• Food Obsessed (CodeFoodBar) Friday, June 29, 12

Slide 6

Slide 6 text

• Food Obsessed (CodeFoodBar) • Open Source Ruby & JS (Sammy.js, et al) Friday, June 29, 12

Slide 7

Slide 7 text

• Food Obsessed (CodeFoodBar) • Open Source Ruby & JS (Sammy.js, et al) • Bicoastal (Brooklyn, NY <-> Berkeley, CA) Friday, June 29, 12

Slide 8

Slide 8 text

• Food Obsessed (CodeFoodBar) • Open Source Ruby & JS (Sammy.js, et al) • Bicoastal (Brooklyn, NY <-> Berkeley, CA) • Very excited to be here Friday, June 29, 12

Slide 9

Slide 9 text

• Food Obsessed (CodeFoodBar) • Open Source Ruby & JS (Sammy.js, et al) • Bicoastal (Brooklyn, NY <-> Berkeley, CA) • Very excited to be here • twitter: @aq Friday, June 29, 12

Slide 10

Slide 10 text

• Food Obsessed (CodeFoodBar) • Open Source Ruby & JS (Sammy.js, et al) • Bicoastal (Brooklyn, NY <-> Berkeley, CA) • Very excited to be here • twitter: @aq • github: quirkey Friday, June 29, 12

Slide 11

Slide 11 text

CTO of Friday, June 29, 12

Slide 12

Slide 12 text

Chief Taco Officer CTO of Friday, June 29, 12

Slide 13

Slide 13 text

www.paperlesspost.com Friday, June 29, 12

Slide 14

Slide 14 text

Friday, June 29, 12

Slide 15

Slide 15 text

A lot of these ideas and implementations are theirs. Amazing team. Friday, June 29, 12

Slide 16

Slide 16 text

METRICS METRICS METRICS Friday, June 29, 12

Slide 17

Slide 17 text

If you’re not improving your product, what are you doing? System metrics == PRoduct Metrics Friday, June 29, 12

Slide 18

Slide 18 text

The spiel Friday, June 29, 12

Slide 19

Slide 19 text

Measure Experiment Pivot Metrics Friday, June 29, 12

Slide 20

Slide 20 text

Measure Experiment Pivot Metrics #winning Friday, June 29, 12

Slide 21

Slide 21 text

WTF is going on MEASURE Friday, June 29, 12

Slide 22

Slide 22 text

Make changes, users are like WTF Experiment Friday, June 29, 12

Slide 23

Slide 23 text

Take the best result and change WTF is going on Pivot Friday, June 29, 12

Slide 24

Slide 24 text

The data you collect in _every_ step Metrics Friday, June 29, 12

Slide 25

Slide 25 text

The Way Friday, June 29, 12

Slide 26

Slide 26 text

Moving the needle Friday, June 29, 12

Slide 27

Slide 27 text

Continuous Delivery Friday, June 29, 12

Slide 28

Slide 28 text

Continuous Process Delivery Friday, June 29, 12

Slide 29

Slide 29 text

Continuous Process DeliveryTM Friday, June 29, 12

Slide 30

Slide 30 text

Moving the needle Friday, June 29, 12

Slide 31

Slide 31 text

Moving the needle 1. Measure Everything Friday, June 29, 12

Slide 32

Slide 32 text

Moving the needle 1. Measure Everything 2. Set Goal metrics attached to each feature/deploy Friday, June 29, 12

Slide 33

Slide 33 text

Moving the needle 1. Measure Everything 2. Set Goal metrics attached to each feature/deploy 3. Build & Roll out in chunks Friday, June 29, 12

Slide 34

Slide 34 text

Moving the needle 1. Measure Everything 2. Set Goal metrics attached to each feature/deploy 3. Build & Roll out in chunks 4. Regroup and analyze Friday, June 29, 12

Slide 35

Slide 35 text

Seriously, everything 1. Measure everything Friday, June 29, 12

Slide 36

Slide 36 text

What happened vs What’s happening Absolute vs Relative Accounting vs Trending Friday, June 29, 12

Slide 37

Slide 37 text

Key Performance Indicators Accounting = KPI’s Friday, June 29, 12

Slide 38

Slide 38 text

It’s in your database the data is there Friday, June 29, 12

Slide 39

Slide 39 text

Friday, June 29, 12

Slide 40

Slide 40 text

KPI’s • From the existing database • PLPSQL • Run nightly, offline • Ad Hoc • Returns fixed (absolute) data, saved back to a database • About the numbers Friday, June 29, 12

Slide 41

Slide 41 text

Pretty pictures. TRends Friday, June 29, 12

Slide 42

Slide 42 text

Friday, June 29, 12

Slide 43

Slide 43 text

TRends • Redis backed • Low impact • Time series, fading resolution • GRAPHS • About the relative differences, actual numbers may be incorrect Friday, June 29, 12

Slide 44

Slide 44 text

NEW SHIT or new to us and maybe you Friday, June 29, 12

Slide 45

Slide 45 text

Statsd + Friday, June 29, 12

Slide 46

Slide 46 text

Statsd + Graphite + Friday, June 29, 12

Slide 47

Slide 47 text

graphiti Statsd + Graphite + Friday, June 29, 12

Slide 48

Slide 48 text

gh://etsy/statsd gh://paperlesspost/statsd gh://github/statsd-ruby Statsd Friday, June 29, 12

Slide 49

Slide 49 text

1 Paperless::Statsd.increment "dashboard.logins" 2 Paperless::Statsd.timing "media_file.convert", time_in_ms Friday, June 29, 12

Slide 50

Slide 50 text

Friday, June 29, 12

Slide 51

Slide 51 text

gh://quirkey/resque-metrics Friday, June 29, 12

Slide 52

Slide 52 text

An alternate frontend for graphite, optimized for play and discovery gh://paperlesspost/graphiti Graphiti Friday, June 29, 12

Slide 53

Slide 53 text

Friday, June 29, 12

Slide 54

Slide 54 text

Graphiti Friday, June 29, 12

Slide 55

Slide 55 text

Graphiti • Sinatra, Sammy, Redis Friday, June 29, 12

Slide 56

Slide 56 text

Graphiti • Sinatra, Sammy, Redis • Graphs stored as JSON in Redis Friday, June 29, 12

Slide 57

Slide 57 text

Graphiti • Sinatra, Sammy, Redis • Graphs stored as JSON in Redis • Ace Editor for Play/editing Friday, June 29, 12

Slide 58

Slide 58 text

Graphiti • Sinatra, Sammy, Redis • Graphs stored as JSON in Redis • Ace Editor for Play/editing • Named Dashboards are collections of graphs Friday, June 29, 12

Slide 59

Slide 59 text

Graphiti • Sinatra, Sammy, Redis • Graphs stored as JSON in Redis • Ace Editor for Play/editing • Named Dashboards are collections of graphs • WIP Friday, June 29, 12

Slide 60

Slide 60 text

Friday, June 29, 12

Slide 61

Slide 61 text

Friday, June 29, 12

Slide 62

Slide 62 text

Attached to Specs/Issues 2. SET Goal metrics Friday, June 29, 12

Slide 63

Slide 63 text

Terminology TANGENT Friday, June 29, 12

Slide 64

Slide 64 text

Terminology TANGENT Friday, June 29, 12

Slide 65

Slide 65 text

• Accounts create Events Terminology TANGENT Friday, June 29, 12

Slide 66

Slide 66 text

• Accounts create Events • Events have a Card Terminology TANGENT Friday, June 29, 12

Slide 67

Slide 67 text

• Accounts create Events • Events have a Card • Card [customizations] are paid for with Coins Terminology TANGENT Friday, June 29, 12

Slide 68

Slide 68 text

• Accounts create Events • Events have a Card • Card [customizations] are paid for with Coins • Events send to Guests Terminology TANGENT Friday, June 29, 12

Slide 69

Slide 69 text

• Accounts create Events • Events have a Card • Card [customizations] are paid for with Coins • Events send to Guests • Guests Reply to Events Terminology TANGENT Friday, June 29, 12

Slide 70

Slide 70 text

Quantitative vs Qualitative Friday, June 29, 12

Slide 71

Slide 71 text

How good your product is. qualitative Friday, June 29, 12

Slide 72

Slide 72 text

Events created per Account Guests added per Event Coins used per Event A single value measured on a per n axis Friday, June 29, 12

Slide 73

Slide 73 text

Quality is the velocity of our product growth Friday, June 29, 12

Slide 74

Slide 74 text

How big your funnel is. Quantitative Friday, June 29, 12

Slide 75

Slide 75 text

% of Accounts that enter the create tool % of Guests that leave a comment % of Events sent with coins A number value measured in a % change over time Friday, June 29, 12

Slide 76

Slide 76 text

QUANTITY is the ACCELERATION of our product growth Friday, June 29, 12

Slide 77

Slide 77 text

Quantitative vs Qualitative Friday, June 29, 12

Slide 78

Slide 78 text

Quantitative X Qualitative GROWTH Friday, June 29, 12

Slide 79

Slide 79 text

A feature can touch both, but in practice it doesn’t. Pick ONE type And a FEW metrics Friday, June 29, 12

Slide 80

Slide 80 text

Attach them to the spec or issue Friday, June 29, 12

Slide 81

Slide 81 text

Friday, June 29, 12

Slide 82

Slide 82 text

Friday, June 29, 12

Slide 83

Slide 83 text

Duh, you guys know Ruby, this part is easy. 3. Build & Roll OUt in Chunks Friday, June 29, 12

Slide 84

Slide 84 text

if current_account_has_feature? :hybrid # show the hybrid Friday, June 29, 12

Slide 85

Slide 85 text

Friday, June 29, 12

Slide 86

Slide 86 text

Friday, June 29, 12

Slide 87

Slide 87 text

Daily, Weekly, Obsessive Checking Report along the way Friday, June 29, 12

Slide 88

Slide 88 text

Friday, June 29, 12

Slide 89

Slide 89 text

Was our hunch correct. 4. roundup and report Friday, June 29, 12

Slide 90

Slide 90 text

And no negative change. Numbers are important but most important is any positive change Friday, June 29, 12

Slide 91

Slide 91 text

Making sure everyone knows what did go right, and what opportunities there are to improve. Roundup meetings Friday, June 29, 12

Slide 92

Slide 92 text

5. ProfIt? Friday, June 29, 12

Slide 93

Slide 93 text

Still early days Friday, June 29, 12

Slide 94

Slide 94 text

The good Friday, June 29, 12

Slide 95

Slide 95 text

The good • For certain features, it can bring laser focus to the goals. Friday, June 29, 12

Slide 96

Slide 96 text

The good • For certain features, it can bring laser focus to the goals. • The separation of the Q’s has brought a lot of clarity to what we’re actually trying to change. Friday, June 29, 12

Slide 97

Slide 97 text

The good • For certain features, it can bring laser focus to the goals. • The separation of the Q’s has brought a lot of clarity to what we’re actually trying to change. • Can inspire and give a clear goal to developers. Friday, June 29, 12

Slide 98

Slide 98 text

The opportunities Friday, June 29, 12

Slide 99

Slide 99 text

The opportunities • Can be dangerous if people focus on the wrong goals. Friday, June 29, 12

Slide 100

Slide 100 text

The opportunities • Can be dangerous if people focus on the wrong goals. • Hard to actually test features in isolation. Friday, June 29, 12

Slide 101

Slide 101 text

The opportunities • Can be dangerous if people focus on the wrong goals. • Hard to actually test features in isolation. • Some metrics are much harder to measure accurately. Friday, June 29, 12

Slide 102

Slide 102 text

The opportunities • Can be dangerous if people focus on the wrong goals. • Hard to actually test features in isolation. • Some metrics are much harder to measure accurately. • Teams need more mid-process visibility in to goals. Friday, June 29, 12

Slide 103

Slide 103 text

We’re gathering metrics about how metrics driven works NOW Friday, June 29, 12

Slide 104

Slide 104 text

IT’s going pretty well. Friday, June 29, 12

Slide 105

Slide 105 text

github.com/quirkey twitter.com/aq quirkey.com paperlesspost.com THANKS! Friday, June 29, 12