How we define, measure & validate Experiments LUKAS FITTL @lfittl spark59.com LEAN STACK Lean Startup is trademarked by Eric Ries and used with permission. Business Model Canvas is created by Alex Osterwalder and licensed under CC-BY-SA.

About Me •Entrepreneur (Ef cient Cloud, Soup.io) •Member of Spark59 (Lean Canvas, Running Lean) •http://founderswiki.com/

Key Partners Key Activities Value Proposition Value Proposition Customer Relationships Customer Segments Key Partners Key Resources Value Proposition Value Proposition Channels Customer Segments Cost Structure Cost Structure Cost Structure Revenue Streams Revenue Streams Revenue Streams WTF? IDEAS PRODUCT DATA MEASURE BUILD LEARN Business Model Build-Measure-Learn Metrics

IDEAS PRODUCT DATA MEASURE BUILD LEARN Experiment

Step 0: De ne the experiment. Formulate hypothesis before you test them.

No content

Results Validated Learning Experiment Scope Minimum Build Learning Goal Falsi able Hypotheses LEARN BUILD MEASURE

Learning Goal What are you trying to learn or achieve?

Experiment Scope How many and/or how long will the experiment last?

Minimum Build Plan List the build plan step by step to get your measurement and experiment.

Falsi able Hypothesis [Speci c Repeatable Action] will [Expected Measurable Outcome]

Results Enter the data.

Validated Learning Summarise your Learning & Validate/Invalidate.

EXPERIMENT REPORT Learning Goal Author: Title: Created: Experiment Scope Minimum Build Plan Falsifiable Hypothesis Validated Learning Results Lean Stack by Spark59.com Will an email message asking notifying users that they are over they plan limit get them to upgrade? This is a test to see if building a feature paywall would be worth doing so. 1 out of 50 users will upgrade their account 10 out of 50 users will modify the number of collaborators 1. run query to get usersʼ info 2. email via gmail 3. run query again to see if any changes have occurred Email 50 users who are over the # of collaborators for the free Ideate plan 8/8/2012 - Emailed 49 users via gmail (signed up from 6-25 to 7-07) 8/14/2012 - 2 of 49 had removed collaborators (all collaborators) 8/20/2012 - 1 of 49 upgraded account *First cohort emaild had issues: collaborators count was not set to be unique, did not check for Ash as a collaborator. *Second cohort fixed, issues + collected last_login & activtated. 1151 users VALIDATED or INVALIDATED or INCONCLUSIVE

EXPERIMENT REPORT Learning Goal Author: Title: Created: Experiment Scope Minimum Build Plan Falsifiable Hypothesis Validated Learning Results Lean Stack by Spark59.com What are you trying to learn or achieve? Your statement on the expected outcome of the experiment. Use this format: [Specific Repeatable Action] will [Expected Measurable Outcome] List the build plan step by step to get your measurement and experiment. How many and/or how long will the experiment last? Enter the data. Summarise your learning from the experiment. VALIDATED or INVALIDATED or INCONCLUSIVE

How does this work in a team context? Team member owns the experiment and draws resources as needed.

How many experiments at the same time? ~ 1 or 2 per team member

IDEAS PRODUCT DATA MEASURE BUILD LEARN Build Measure Learn ✓ ✘ Kan-ban Board

Build Measure Learn ✓ ✘

[ Exercise ] Lets Run An Experiment

No content

Time Tracker Facebook Integration

Time Tracker Facebook Integration

Time Tracker Improve Activation Facebook Integration

Time Tracker Facebook Integration Improve Activation

Time Tracker Improve Activation Facebook Integration ✘

✘ Failure is good. Validated Learning is our Measurement of Progress.

Time Tracker Improve Activation Facebook Integration ✘ ✓

Time Tracker Invoicing App Facebook Integration Expense Tracking

Understand Problem: Explorative Customer Interviews De ne Solution: User Testing With Prototypes Validate Qualitatively: Release MVP and hand-collect feedback 10s to 100s of customers Verify Quantitatively: Partial Rollout or Split Test and measure improvement 100s to 10,000s of customers

Questions? lukas@spark59.com