Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Lean Stack - Using Build-Measure-Learn Effectively

Lean Stack - Using Build-Measure-Learn Effectively

Lean Startup Meetup Wiesbaden & Frankfurt at Seibert Media

Lukas Fittl

April 08, 2013
Tweet

More Decks by Lukas Fittl

Other Decks in Business

Transcript

  1. Using Build-Measure-Learn
    Effectively
    LUKAS FITTL
    @lfittl
    spark59.com
    LEAN
    STACK
    Lean Startup is trademarked by Eric Ries and used with permission.
    Business Model Canvas is created by Alex Osterwalder and licensed under CC-BY-SA.

    View full-size slide

  2. About Myself & Spark59
    Founded 2 tech startups,
    worked with many others

    View full-size slide

  3. From Agile to Lean Startup

    View full-size slide

  4. Waterfall
    Release!
    Risk
    Time
    BUILD
    BUILD
    BUILD

    View full-size slide

  5. Agile / XP / SCRUM
    Risk
    Time
    Release!
    BUILD
    Release!
    BUILD
    Release!
    BUILD
    Release!
    BUILD

    View full-size slide

  6. Are we making our
    product
    better or worse?

    View full-size slide

  7. Are we efficient?
    vs
    Are we effective?

    View full-size slide

  8. The Definition of Done
    should include
    testing customer behaviour

    View full-size slide

  9. IDEAS
    PRODUCT
    DATA
    MEASURE
    BUILD
    LEARN
    Experiment

    View full-size slide

  10. Putting Build-Measure-Learn
    on a Kan-ban Board

    View full-size slide

  11. IDEAS
    PRODUCT
    DATA
    MEASURE
    BUILD
    LEARN
    Build Measure Learn


    Kan-ban Board

    View full-size slide

  12. Build Measure Learn


    View full-size slide

  13. Experiment
    Reports

    View full-size slide

  14. 1. Context - Why is this relevant?
    2. Quantify Your Goals
    3. Set Scope & Timebox
    Planning an Experiment

    View full-size slide

  15. Background
    EXPERIMENT REPORT Author:
    Title: Created:
    Experiment Scope
    Minimum Build Plan
    Lean Stack by Spark59.com
    Falsifiable Hypothesis
    Validated Learning
    Results
    Next Action
    What are you trying to learn or achieve?
    List the build plan step by step to get your measurement and experiment.
    How many and/or how long will the experiment last?
    [DATE]
    [NAME]
    [TITLE]
    Your list of statements on the expected outcome of the experiment.
    Use this format:
    [Specific Repeatable Action] will [Expected Measurable Outcome]
    Enter the data.
    Summarize your learning from the experiment.
    VALIDATED or INVALIDATED
    Whatʼs the next experiment?

    View full-size slide

  16. Background
    EXPERIMENT REPORT Author:
    Title: Created:
    Experiment Scope
    Minimum Build Plan
    Lean Stack by Spark59.com
    Falsifiable Hypothesis
    Validated Learning
    Results
    Next Action
    What are you trying to learn or achieve?
    List the build plan step by step to get your measurement and experiment.
    How many and/or how long will the experiment last?
    [DATE]
    [NAME]
    [TITLE]
    Your list of statements on the expected outcome of the experiment.
    Use this format:
    [Specific Repeatable Action] will [Expected Measurable Outcome]
    Enter the data.
    Summarize your learning from the experiment.
    VALIDATED or INVALIDATED
    Whatʼs the next experiment?
    Left Side Defines
    The Experiment

    View full-size slide

  17. Give Context.
    What are you trying to learn?
    Background
    What are you trying to learn or achieve?

    View full-size slide

  18. Quantify Your Goals
    Falsifiable Hypothesis
    Your list of statements on the expected outcome of
    the experiment.
    Use this format:
    [Specific Repeatable Action] will
    [Expected Measurable Outcome]

    View full-size slide

  19. Define the Maximum Time/Effort
    Spent on this Experiment
    Experiment Scope
    How long will the experiment last?

    View full-size slide

  20. What needs to be done?
    (define scope, MVP!)
    Minimum Build Plan
    List the build plan step by step to get your
    measurement and experiment.

    View full-size slide

  21. Background
    EXPERIMENT REPORT Author:
    Title: Created:
    Experiment Scope
    Minimum Build Plan
    Lean Stack by Spark59.com
    Falsifiable Hypothesis
    Validated Learning
    Results
    Next Action
    What are you trying to learn or achieve?
    List the build plan step by step to get your measurement and experiment.
    How many and/or how long will the experiment last?
    [DATE]
    [NAME]
    [TITLE]
    Your list of statements on the expected outcome of the experiment.
    Use this format:
    [Specific Repeatable Action] will [Expected Measurable Outcome]
    Enter the data.
    Summarize your learning from the experiment.
    VALIDATED or INVALIDATED
    Whatʼs the next experiment?
    Right Side
    Analyses The
    Results

    View full-size slide

  22. Actual data you Measured,
    based on your Hypothesis.
    Results
    Enter the data.

    View full-size slide

  23. Validated
    Invalidated
    Inconclusive
    Validated Learning
    Summarize your learning from the experiment.
    + Why?

    View full-size slide

  24. What will you do next
    based on your learning?
    Next Action
    Whatʼs the next experiment?

    View full-size slide

  25. Build Measure Learn


    View full-size slide


  26. Failure is good.
    Validated Learning is our
    Measurement of Progress.

    View full-size slide

  27. 1 experiment
    per team member

    View full-size slide

  28. Separate Task Board
    for tracking the details
    Swimlane for each Experiment

    View full-size slide

  29. Qualitative Experiments

    View full-size slide

  30. Examples:
    Your Beta Launch
    Improving the First-Use Experience

    View full-size slide

  31. This only works if you do
    sessions with customers
    multiple times per day

    View full-size slide

  32. Get people to commit to pay
    (e.g. signing a letter-of-intent)
    or measure their behaviour
    (e.g. time spent, functionality found, etc)

    View full-size slide

  33. Quantitative Qualitative
    Generative
    Surveys
    Interviews
    Contextual Inquiry
    Mental Models
    Interviews
    Diary Studies
    Evaluative
    Automated Card Sort
    Surveys
    Automated Studies
    Analytics
    A/B Testing
    Multi-Variant Testing
    Usability Testing
    Moderated Card Sort
    Wizard of Oz
    By @JohannaKoll, based on work by Janice Fraser, Nate Bolt & Christian Rohrer

    View full-size slide

  34. Quantitative Qualitative
    Generative
    Surveys
    Interviews
    Contextual Inquiry
    Mental Models
    Interviews
    Diary Studies
    Evaluative
    Automated Card Sort
    Surveys
    Automated Studies
    Analytics
    A/B Testing
    Multi-Variant Testing
    Usability Testing
    Moderated Card Sort
    Wizard of Oz
    By @JohannaKoll, based on work by Janice Fraser, Nate Bolt & Christian Rohrer

    View full-size slide

  35. Usability Tests
    http://www.slideshare.net/JohnnyForeigner/active-testing-sketches-leancamp-bcn

    View full-size slide

  36. Quantitative Experiments

    View full-size slide

  37. Team needs to
    Understand The Impact
    of their work

    View full-size slide

  38. Are we making progress
    towards our business goal?

    View full-size slide

  39. $10000 / month
    $2000 / month
    Now June 2013
    ?

    View full-size slide

  40. Break Revenue Targets,
    down into Unit Economics.

    View full-size slide

  41. Dave McClure’s AARRR
    ACQUISITION
    ACTIVATION
    RETENTION
    REVENUE
    REFERRAL
    How do users find you?
    Do users have a reat first experience?
    Do users come back?
    How do you make money?
    Do users tell others?

    View full-size slide

  42. $2000 / month
    ACQ
    ACT
    20% to paid at $35
    RET
    REV
    20% stay for 1 week
    8% activate
    5% sign up
    2 paid
    $70 / day
    10 retained
    50 activated
    625 signed up
    12500 unique visitors
    Each Day:

    View full-size slide

  43. ACQUISITION
    ACTIVATION
    RETENTION
    REVENUE
    REFERRAL
    Value Metrics
    How do users find you?
    Do users have a great first experience?
    Do users come back?
    How do you make money?
    Do users tell others?

    View full-size slide

  44. $2000 / month
    ACQ
    ACT
    20% to paid at $35
    RET
    REV
    20% stay for 1 week
    8% activate
    5% sign up
    2 paid
    $70 / day
    10 retained
    50 activated
    625 signed up
    12500 unique visitors
    Each Day:

    View full-size slide

  45. ACQUISITION
    ACTIVATION
    RETENTION
    REVENUE
    REFERRAL
    How do users find you?
    Do users have a great first experience?
    Do users come back?
    How do you make money?
    Do users tell others?
    Growth Metrics

    View full-size slide

  46. $2000 / month
    ACQ
    ACT
    20% to paid at $35
    RET
    REV
    20% stay for 1 week
    8% activate
    5% sign up
    2 paid
    $70 / day
    10 retained
    50 activated
    625 signed up
    12500 unique visitors
    Each Day:

    View full-size slide

  47. Split URL
    Test

    View full-size slide

  48. New variant will increase
    conversion by 5%
    Falsifiable Hypothesis:
    10-14 hours to build & implement
    1,000 visitors per page
    Experiment Scope:

    View full-size slide

  49. VARIANT
    Visual
    Website
    Optimizer
    CONTROL
    50%
    50%
    Measurements
    via
    unique
    visitors

    View full-size slide

  50. VARIANT
    CONTROL
    ??
    How did we do?
    ??

    View full-size slide

  51. VARIANT
    CONTROL
    42.2%
    Conversion
    20%
    Better
    35.1%
    Conversion

    View full-size slide

  52. VARIANT
    WHY???
    New LAYOUT
    New COPY
    New SOCIAL PROOF

    View full-size slide

  53. VARIANT
    Visual
    Website
    Optimizer
    Experiments
    are additive
    #4
    #3

    View full-size slide

  54. VARIANT
    #4
    #3
    Same content,
    just new layout
    Visual
    Website
    Optimizer
    #5

    View full-size slide

  55. Design only variant will increase
    conversion by 2%
    Falsifiable Hypothesis:
    ~ 2 hours to build & implement
    ~ 1,000 visitors per page
    Experiment Scope:

    View full-size slide

  56. VARIANT 2
    42.6%
    Conversion
    48.5%
    Conversion
    CONTROL

    View full-size slide

  57. INCREASE
    ACQUISITION
    SEO
    RANKING LANDING PAGE
    CONVERSION
    INBOUND
    CHANNELS
    # Depends on your resources & experiment scope

    View full-size slide

  58. Build Measure Learn


    View full-size slide

  59. Build Measure Learn


    Experiment Scope
    Experiment Scope
    How long will the experiment last?

    View full-size slide

  60. Interviews/Prototypes: 1-2 Weeks
    Testing your MVP: 1 Month
    A/B Tests: 2-3 Months
    Recommended Timeboxes:

    View full-size slide

  61. Dictates how fast you can learn
    Experiment Scope
    Your Runway
    How long you can survive

    View full-size slide

  62. Lean Stack
    is a Toolbox

    View full-size slide

  63. Digital Version:
    leanstack.com

    View full-size slide

  64. Best used with Post-Its

    View full-size slide

  65. And/or a Whiteboard

    View full-size slide

  66. The goal is not to be perfect,
    but to do more of the
    right things at the right time.

    View full-size slide