Lean Stack - Using Build-Measure-Learn Effectively

Lean Startup Meetup Wiesbaden & Frankfurt at Seibert Media

Transcript

1. Using Build-Measure-Learn Effectively LUKAS FITTL @lfittl spark59.com LEAN STACK Lean Startup is trademarked by Eric Ries and used with permission. Business Model Canvas is created by Alex Osterwalder and licensed under CC-BY-SA.
2. About Myself & Spark59 Founded 2 tech startups, worked with many others
4. From Agile to Lean Startup
5. Waterfall Release! Risk Time BUILD BUILD BUILD
6. Agile / XP / SCRUM Risk Time Release! BUILD Release! BUILD Release! BUILD Release! BUILD
7. Are we making our product better or worse?
8. Are we efficient? vs Are we effective?
9. The Definition of Done should include testing customer behaviour
10. IDEAS PRODUCT DATA MEASURE BUILD LEARN Experiment
11. Putting Build-Measure-Learn on a Kan-ban Board
12. IDEAS PRODUCT DATA MEASURE BUILD LEARN Build Measure Learn ✓ ✘ Kan-ban Board
13. Build Measure Learn ✓ ✘
14. Experiment Reports
15. 1. Context - Why is this relevant? 2. Quantify Your Goals 3. Set Scope & Timebox Planning an Experiment
16. Background EXPERIMENT REPORT Author: Title: Created: Experiment Scope Minimum Build Plan Lean Stack by Spark59.com Falsifiable Hypothesis Validated Learning Results Next Action What are you trying to learn or achieve? List the build plan step by step to get your measurement and experiment. How many and/or how long will the experiment last? [DATE] [NAME] [TITLE] Your list of statements on the expected outcome of the experiment. Use this format: [Specific Repeatable Action] will [Expected Measurable Outcome] Enter the data. Summarize your learning from the experiment. VALIDATED or INVALIDATED Whatʼs the next experiment?
17. Background EXPERIMENT REPORT Author: Title: Created: Experiment Scope Minimum Build Plan Lean Stack by Spark59.com Falsifiable Hypothesis Validated Learning Results Next Action What are you trying to learn or achieve? List the build plan step by step to get your measurement and experiment. How many and/or how long will the experiment last? [DATE] [NAME] [TITLE] Your list of statements on the expected outcome of the experiment. Use this format: [Specific Repeatable Action] will [Expected Measurable Outcome] Enter the data. Summarize your learning from the experiment. VALIDATED or INVALIDATED Whatʼs the next experiment? Left Side Defines The Experiment
18. Give Context. What are you trying to learn? Background What are you trying to learn or achieve?
19. Quantify Your Goals Falsifiable Hypothesis Your list of statements on the expected outcome of the experiment. Use this format: [Specific Repeatable Action] will [Expected Measurable Outcome]
20. Define the Maximum Time/Effort Spent on this Experiment Experiment Scope How long will the experiment last?
21. What needs to be done? (define scope, MVP!) Minimum Build Plan List the build plan step by step to get your measurement and experiment.
22. Background EXPERIMENT REPORT Author: Title: Created: Experiment Scope Minimum Build Plan Lean Stack by Spark59.com Falsifiable Hypothesis Validated Learning Results Next Action What are you trying to learn or achieve? List the build plan step by step to get your measurement and experiment. How many and/or how long will the experiment last? [DATE] [NAME] [TITLE] Your list of statements on the expected outcome of the experiment. Use this format: [Specific Repeatable Action] will [Expected Measurable Outcome] Enter the data. Summarize your learning from the experiment. VALIDATED or INVALIDATED Whatʼs the next experiment? Right Side Analyses The Results
23. Actual data you Measured, based on your Hypothesis. Results Enter the data.
24. Validated Invalidated Inconclusive Validated Learning Summarize your learning from the experiment. + Why?
25. What will you do next based on your learning? Next Action Whatʼs the next experiment?
26. Build Measure Learn ✓ ✘
27. ✘ Failure is good. Validated Learning is our Measurement of Progress.
28. 1 experiment per team member
29. Separate Task Board for tracking the details Swimlane for each Experiment
30. Qualitative Experiments
31. Examples: Your Beta Launch Improving the First-Use Experience
32. This only works if you do sessions with customers multiple times per day
33. Get people to commit to pay (e.g. signing a letter-of-intent) or measure their behaviour (e.g. time spent, functionality found, etc)
34. Quantitative Qualitative Generative Surveys Interviews Contextual Inquiry Mental Models Interviews Diary Studies Evaluative Automated Card Sort Surveys Automated Studies Analytics A/B Testing Multi-Variant Testing Usability Testing Moderated Card Sort Wizard of Oz By @JohannaKoll, based on work by Janice Fraser, Nate Bolt & Christian Rohrer
35. Quantitative Qualitative Generative Surveys Interviews Contextual Inquiry Mental Models Interviews Diary Studies Evaluative Automated Card Sort Surveys Automated Studies Analytics A/B Testing Multi-Variant Testing Usability Testing Moderated Card Sort Wizard of Oz By @JohannaKoll, based on work by Janice Fraser, Nate Bolt & Christian Rohrer
36. Usability Tests http://www.slideshare.net/JohnnyForeigner/active-testing-sketches-leancamp-bcn
37. Quantitative Experiments
38. Team needs to Understand The Impact of their work
39. Are we making progress towards our business goal?
40. $10000 / month $2000 / month Now June 2013 ?
41. Break Revenue Targets, down into Unit Economics.
42. Dave McClure’s AARRR ACQUISITION ACTIVATION RETENTION REVENUE REFERRAL How do users find you? Do users have a reat first experience? Do users come back? How do you make money? Do users tell others?
43. $2000 / month ACQ ACT 20% to paid at $35 RET REV 20% stay for 1 week 8% activate 5% sign up 2 paid $70 / day 10 retained 50 activated 625 signed up 12500 unique visitors Each Day:
44. ACQUISITION ACTIVATION RETENTION REVENUE REFERRAL Value Metrics How do users find you? Do users have a great first experience? Do users come back? How do you make money? Do users tell others?
45. $2000 / month ACQ ACT 20% to paid at $35 RET REV 20% stay for 1 week 8% activate 5% sign up 2 paid $70 / day 10 retained 50 activated 625 signed up 12500 unique visitors Each Day:
46. ACQUISITION ACTIVATION RETENTION REVENUE REFERRAL How do users find you? Do users have a great first experience? Do users come back? How do you make money? Do users tell others? Growth Metrics
47. $2000 / month ACQ ACT 20% to paid at $35 RET REV 20% stay for 1 week 8% activate 5% sign up 2 paid $70 / day 10 retained 50 activated 625 signed up 12500 unique visitors Each Day:
48. Example!
49. Split URL Test
50. VARIANT
51. VARIANT
52. VARIANT
53. VARIANT
55. New variant will increase conversion by 5% Falsifiable Hypothesis: 10-14 hours to build & implement 1,000 visitors per page Experiment Scope:
56. VARIANT Visual Website Optimizer CONTROL 50% 50% Measurements via unique visitors
57. VARIANT CONTROL ?? How did we do? ??
58. VARIANT CONTROL 42.2% Conversion 20% Better 35.1% Conversion
59. VARIANT WHY??? New LAYOUT New COPY New SOCIAL PROOF
60. VARIANT Visual Website Optimizer Experiments are additive #4 #3
61. VARIANT #4 #3 Same content, just new layout Visual Website Optimizer #5
62. Design only variant will increase conversion by 2% Falsifiable Hypothesis: ~ 2 hours to build & implement ~ 1,000 visitors per page Experiment Scope:
63. VARIANT 2 42.6% Conversion 48.5% Conversion CONTROL
64. INCREASE ACQUISITION SEO RANKING LANDING PAGE CONVERSION INBOUND CHANNELS # Depends on your resources & experiment scope
65. Timeboxing
66. Build Measure Learn ✓ ✘
67. Build Measure Learn ✓ ✘ Experiment Scope Experiment Scope How long will the experiment last?
68. Interviews/Prototypes: 1-2 Weeks Testing your MVP: 1 Month A/B Tests: 2-3 Months Recommended Timeboxes:
69. Dictates how fast you can learn Experiment Scope Your Runway How long you can survive
70. Lean Stack is a Toolbox
71. Digital Version: leanstack.com
72. Best used with Post-Its
73. And/or a Whiteboard
74. The goal is not to be perfect, but to do more of the right things at the right time.
75. Thanks! lukas@spark59.com