Tweet
Tweet

Slide 1

Slide 1 text

Using Build-Measure-Learn Effectively LUKAS FITTL @lfittl spark59.com LEAN STACK Lean Startup is trademarked by Eric Ries and used with permission. Business Model Canvas is created by Alex Osterwalder and licensed under CC-BY-SA.

Slide 2

Slide 2 text

About Myself & Spark59 Founded 2 tech startups, worked with many others

Slide 3

Slide 3 text

No content

Slide 4

Slide 4 text

From Agile to Lean Startup

Slide 5

Slide 5 text

Waterfall Release! Risk Time BUILD BUILD BUILD

Slide 6

Slide 6 text

Agile / XP / SCRUM Risk Time Release! BUILD Release! BUILD Release! BUILD Release! BUILD

Slide 7

Slide 7 text

Are we making our product better or worse?

Slide 8

Slide 8 text

Are we efficient? vs Are we effective?

Slide 9

Slide 9 text

The Definition of Done should include testing customer behaviour

Slide 10

Slide 10 text

IDEAS PRODUCT DATA MEASURE BUILD LEARN Experiment

Slide 11

Slide 11 text

Putting Build-Measure-Learn on a Kan-ban Board

Slide 12

Slide 12 text

IDEAS PRODUCT DATA MEASURE BUILD LEARN Build Measure Learn ✓ ✘ Kan-ban Board

Slide 13

Slide 13 text

Build Measure Learn ✓ ✘

Slide 14

Slide 14 text

Experiment Reports

Slide 15

Slide 15 text

1. Context - Why is this relevant? 2. Quantify Your Goals 3. Set Scope & Timebox Planning an Experiment

Slide 16

Slide 16 text

Background EXPERIMENT REPORT Author: Title: Created: Experiment Scope Minimum Build Plan Lean Stack by Spark59.com Falsifiable Hypothesis Validated Learning Results Next Action What are you trying to learn or achieve? List the build plan step by step to get your measurement and experiment. How many and/or how long will the experiment last? [DATE] [NAME] [TITLE] Your list of statements on the expected outcome of the experiment. Use this format: [Specific Repeatable Action] will [Expected Measurable Outcome] Enter the data. Summarize your learning from the experiment. VALIDATED or INVALIDATED Whatʼs the next experiment?

Slide 17

Slide 17 text

Background EXPERIMENT REPORT Author: Title: Created: Experiment Scope Minimum Build Plan Lean Stack by Spark59.com Falsifiable Hypothesis Validated Learning Results Next Action What are you trying to learn or achieve? List the build plan step by step to get your measurement and experiment. How many and/or how long will the experiment last? [DATE] [NAME] [TITLE] Your list of statements on the expected outcome of the experiment. Use this format: [Specific Repeatable Action] will [Expected Measurable Outcome] Enter the data. Summarize your learning from the experiment. VALIDATED or INVALIDATED Whatʼs the next experiment? Left Side Defines The Experiment

Slide 18

Slide 18 text

Give Context. What are you trying to learn? Background What are you trying to learn or achieve?

Slide 19

Slide 19 text

Quantify Your Goals Falsifiable Hypothesis Your list of statements on the expected outcome of the experiment. Use this format: [Specific Repeatable Action] will [Expected Measurable Outcome]

Slide 20

Slide 20 text

Define the Maximum Time/Effort Spent on this Experiment Experiment Scope How long will the experiment last?

Slide 21

Slide 21 text

What needs to be done? (define scope, MVP!) Minimum Build Plan List the build plan step by step to get your measurement and experiment.

Slide 22

Slide 22 text

Background EXPERIMENT REPORT Author: Title: Created: Experiment Scope Minimum Build Plan Lean Stack by Spark59.com Falsifiable Hypothesis Validated Learning Results Next Action What are you trying to learn or achieve? List the build plan step by step to get your measurement and experiment. How many and/or how long will the experiment last? [DATE] [NAME] [TITLE] Your list of statements on the expected outcome of the experiment. Use this format: [Specific Repeatable Action] will [Expected Measurable Outcome] Enter the data. Summarize your learning from the experiment. VALIDATED or INVALIDATED Whatʼs the next experiment? Right Side Analyses The Results

Slide 23

Slide 23 text

Actual data you Measured, based on your Hypothesis. Results Enter the data.

Slide 24

Slide 24 text

Validated Invalidated Inconclusive Validated Learning Summarize your learning from the experiment. + Why?

Slide 25

Slide 25 text

What will you do next based on your learning? Next Action Whatʼs the next experiment?

Slide 26

Slide 26 text

Build Measure Learn ✓ ✘

Slide 27

Slide 27 text

✘ Failure is good. Validated Learning is our Measurement of Progress.

Slide 28

Slide 28 text

1 experiment per team member

Slide 29

Slide 29 text

Separate Task Board for tracking the details Swimlane for each Experiment

Slide 30

Slide 30 text

Qualitative Experiments

Slide 31

Slide 31 text

Examples: Your Beta Launch Improving the First-Use Experience

Slide 32

Slide 32 text

This only works if you do sessions with customers multiple times per day

Slide 33

Slide 33 text

Get people to commit to pay (e.g. signing a letter-of-intent) or measure their behaviour (e.g. time spent, functionality found, etc)

Slide 34

Slide 34 text

Quantitative Qualitative Generative Surveys Interviews Contextual Inquiry Mental Models Interviews Diary Studies Evaluative Automated Card Sort Surveys Automated Studies Analytics A/B Testing Multi-Variant Testing Usability Testing Moderated Card Sort Wizard of Oz By @JohannaKoll, based on work by Janice Fraser, Nate Bolt & Christian Rohrer

Slide 35

Slide 35 text

Quantitative Qualitative Generative Surveys Interviews Contextual Inquiry Mental Models Interviews Diary Studies Evaluative Automated Card Sort Surveys Automated Studies Analytics A/B Testing Multi-Variant Testing Usability Testing Moderated Card Sort Wizard of Oz By @JohannaKoll, based on work by Janice Fraser, Nate Bolt & Christian Rohrer

Slide 36

Slide 36 text

Usability Tests http://www.slideshare.net/JohnnyForeigner/active-testing-sketches-leancamp-bcn

Slide 37

Slide 37 text

Quantitative Experiments

Slide 38

Slide 38 text

Team needs to Understand The Impact of their work

Slide 39

Slide 39 text

Are we making progress towards our business goal?

Slide 40

Slide 40 text

$10000 / month $2000 / month Now June 2013 ?

Slide 41

Slide 41 text

Break Revenue Targets, down into Unit Economics.

Slide 42

Slide 42 text

Dave McClure’s AARRR ACQUISITION ACTIVATION RETENTION REVENUE REFERRAL How do users find you? Do users have a reat first experience? Do users come back? How do you make money? Do users tell others?

Slide 43

Slide 43 text

$2000 / month ACQ ACT 20% to paid at $35 RET REV 20% stay for 1 week 8% activate 5% sign up 2 paid $70 / day 10 retained 50 activated 625 signed up 12500 unique visitors Each Day:

Slide 44

Slide 44 text

ACQUISITION ACTIVATION RETENTION REVENUE REFERRAL Value Metrics How do users find you? Do users have a great first experience? Do users come back? How do you make money? Do users tell others?

Slide 45

Slide 45 text

$2000 / month ACQ ACT 20% to paid at $35 RET REV 20% stay for 1 week 8% activate 5% sign up 2 paid $70 / day 10 retained 50 activated 625 signed up 12500 unique visitors Each Day:

Slide 46

Slide 46 text

ACQUISITION ACTIVATION RETENTION REVENUE REFERRAL How do users find you? Do users have a great first experience? Do users come back? How do you make money? Do users tell others? Growth Metrics

Slide 47

Slide 47 text

$2000 / month ACQ ACT 20% to paid at $35 RET REV 20% stay for 1 week 8% activate 5% sign up 2 paid $70 / day 10 retained 50 activated 625 signed up 12500 unique visitors Each Day:

Slide 48

Slide 48 text

Example!

Slide 49

Slide 49 text

Split URL Test

Slide 50

Slide 50 text

VARIANT

Slide 51

Slide 51 text

VARIANT

Slide 52

Slide 52 text

VARIANT

Slide 53

Slide 53 text

VARIANT

Slide 54

Slide 54 text

No content

Slide 55

Slide 55 text

New variant will increase conversion by 5% Falsifiable Hypothesis: 10-14 hours to build & implement 1,000 visitors per page Experiment Scope:

Slide 56

Slide 56 text

VARIANT Visual Website Optimizer CONTROL 50% 50% Measurements via unique visitors

Slide 57

Slide 57 text

VARIANT CONTROL ?? How did we do? ??

Slide 58

Slide 58 text

VARIANT CONTROL 42.2% Conversion 20% Better 35.1% Conversion

Slide 59

Slide 59 text

VARIANT WHY??? New LAYOUT New COPY New SOCIAL PROOF

Slide 60

Slide 60 text

VARIANT Visual Website Optimizer Experiments are additive #4 #3

Slide 61

Slide 61 text

VARIANT #4 #3 Same content, just new layout Visual Website Optimizer #5

Slide 62

Slide 62 text

Design only variant will increase conversion by 2% Falsifiable Hypothesis: ~ 2 hours to build & implement ~ 1,000 visitors per page Experiment Scope:

Slide 63

Slide 63 text

VARIANT 2 42.6% Conversion 48.5% Conversion CONTROL

Slide 64

Slide 64 text

INCREASE ACQUISITION SEO RANKING LANDING PAGE CONVERSION INBOUND CHANNELS # Depends on your resources & experiment scope

Slide 65

Slide 65 text

Timeboxing

Slide 66

Slide 66 text

Build Measure Learn ✓ ✘

Slide 67

Slide 67 text

Build Measure Learn ✓ ✘ Experiment Scope Experiment Scope How long will the experiment last?

Slide 68

Slide 68 text

Interviews/Prototypes: 1-2 Weeks Testing your MVP: 1 Month A/B Tests: 2-3 Months Recommended Timeboxes:

Slide 69

Slide 69 text

Dictates how fast you can learn Experiment Scope Your Runway How long you can survive

Slide 70

Slide 70 text

Lean Stack is a Toolbox

Slide 71

Slide 71 text

Digital Version: leanstack.com

Slide 72

Slide 72 text

Best used with Post-Its

Slide 73

Slide 73 text

And/or a Whiteboard

Slide 74

Slide 74 text

The goal is not to be perfect, but to do more of the right things at the right time.

Slide 75

Slide 75 text

Thanks! lukas@spark59.com