Lean Stack: How we define, measure & validate experiments

At Leancamp Berlin - leanca.mp/berlin1/

Licensed under CC-BY-NC-SA - feel free to share, embed and re-use in your presentations, please give attribution. Thanks!

Transcript

1. How we define, measure & validate Experiments LUKAS FITTL @lfittl spark59.com LEAN STACK Lean Startup is trademarked by Eric Ries and used with permission. Business Model Canvas is created by Alex Osterwalder and licensed under CC-BY-SA.
2. About Me •Entrepreneur (Ef cient Cloud, Soup.io) •Member of Spark59 (Lean Canvas, Running Lean) •http://founderswiki.com/
3. Key Partners Key Activities Value Proposition Value Proposition Customer Relationships Customer Segments Key Partners Key Resources Value Proposition Value Proposition Channels Customer Segments Cost Structure Cost Structure Cost Structure Revenue Streams Revenue Streams Revenue Streams WTF? IDEAS PRODUCT DATA MEASURE BUILD LEARN Business Model Build-Measure-Learn Metrics
4. IDEAS PRODUCT DATA MEASURE BUILD LEARN Experiment
5. Step 0: De ne the experiment. Formulate hypothesis before you test them.
7. Results Validated Learning Experiment Scope Minimum Build Learning Goal Falsi able Hypotheses LEARN BUILD MEASURE
8. Learning Goal What are you trying to learn or achieve?
9. Experiment Scope How many and/or how long will the experiment last?
10. Minimum Build Plan List the build plan step by step to get your measurement and experiment.
11. Falsi able Hypothesis [Speci c Repeatable Action] will [Expected Measurable Outcome]
12. Results Enter the data.
13. Validated Learning Summarise your Learning & Validate/Invalidate.
14. EXPERIMENT REPORT Learning Goal Author: Title: Created: Experiment Scope Minimum Build Plan Falsifiable Hypothesis Validated Learning Results Lean Stack by Spark59.com Will an email message asking notifying users that they are over they plan limit get them to upgrade? This is a test to see if building a feature paywall would be worth doing so. 1 out of 50 users will upgrade their account 10 out of 50 users will modify the number of collaborators 1. run query to get usersʼ info 2. email via gmail 3. run query again to see if any changes have occurred Email 50 users who are over the # of collaborators for the free Ideate plan 8/8/2012 - Emailed 49 users via gmail (signed up from 6-25 to 7-07) 8/14/2012 - 2 of 49 had removed collaborators (all collaborators) 8/20/2012 - 1 of 49 upgraded account *First cohort emaild had issues: collaborators count was not set to be unique, did not check for Ash as a collaborator. *Second cohort fixed, issues + collected last_login & activtated. 1151 users VALIDATED or INVALIDATED or INCONCLUSIVE
15. EXPERIMENT REPORT Learning Goal Author: Title: Created: Experiment Scope Minimum Build Plan Falsifiable Hypothesis Validated Learning Results Lean Stack by Spark59.com What are you trying to learn or achieve? Your statement on the expected outcome of the experiment. Use this format: [Specific Repeatable Action] will [Expected Measurable Outcome] List the build plan step by step to get your measurement and experiment. How many and/or how long will the experiment last? Enter the data. Summarise your learning from the experiment. VALIDATED or INVALIDATED or INCONCLUSIVE
16. How does this work in a team context? Team member owns the experiment and draws resources as needed.
17. How many experiments at the same time? ~ 1 or 2 per team member
18. IDEAS PRODUCT DATA MEASURE BUILD LEARN Build Measure Learn ✓ ✘ Kan-ban Board
19. Build Measure Learn ✓ ✘
20. [ Exercise ] Lets Run An Experiment
22. Time Tracker Facebook Integration
23. Time Tracker Facebook Integration
24. Time Tracker Improve Activation Facebook Integration
25. Time Tracker Facebook Integration Improve Activation
26. Time Tracker Improve Activation Facebook Integration ✘
27. ✘ Failure is good. Validated Learning is our Measurement of Progress.
28. Time Tracker Improve Activation Facebook Integration ✘ ✓
29. Time Tracker Invoicing App Facebook Integration Expense Tracking
30. Understand Problem: Explorative Customer Interviews De ne Solution: User Testing With Prototypes Validate Qualitatively: Release MVP and hand-collect feedback 10s to 100s of customers Verify Quantitatively: Partial Rollout or Split Test and measure improvement 100s to 10,000s of customers
31. Questions? lukas@spark59.com