Navigating the tool jungle - How to pick the right tools for your hypothesis experiments

You've lined up everything for your experiment: type of experiment, hypotheses & success criteria, team & time - but how to get started now? How to pick the right tools to actually build, run and evaluate the experiment?

In this talk I help you pick the best tools for your user research & prototyping experiments. I'll share what we learned from running experiments - and going through the test-fail-learn cycle with our tool choices numerous times.

Our three key learnings are:
- Iterate within your experiment
- User-test your usertesting tools
- The simpler, the better - and faster

For each learning I provide a real-life case with plenty of insights.

Transcript

1. Navigating the tool jungle -
   How to pick the right tools for your experiments
   Thomas Leitermann
   Product Manager
   MTP Engage Hamburg 24.05.2019
   @thomasleiterman
   

   View Slide

2. View Slide

3. Prerequisites
   3
   

   View Slide

4. © Business Model Foundry AG
   Set target & success
   criteria
   4
   Hypothesis
   Experiment
   Metric
   Criteria
   Users appreciate our value
   proposition so much, they are willing
   to pay for it.
   Insert a paywall
   after a few example questions.
   the conversion rate to
   purchasing the whole course.
   We are …
   … super happy with 3%
   … okay with 1,3%
   … disappointed with 0,5%.
   Assumption
   Measurement
   Success
   „Read your plants“-App
   

   View Slide

5. › Evaluative or generative?
   › Strength of evidence vs. time-to-insight
   Type of experiment
   5
   

   View Slide

6. Inspired by „The Real Startup Book“, www.realstartupbook.com
   Evaluative or Generative Experiment?
   6
   Market Product
   Question Method Question Method
   Generative
   Customer
    Discovery interview
    Ethnography
    Surveys, focus groups Problem
    Solution interview
    Ethnography
    Concierge-test
    Demo pitch
   Evaluiative
   Revenue
    Landing Page-Test
    Sales pitch &
   Pre-sales
    Fake door
    5-second test
    Flyer
   Solution
    Prototypes (paper,
   clickdummy)
    Wizard of Oz-test
    Analytics
    Usability test
    Finished product
   

   View Slide

7. Derived from A.Osterwalder, Strategyzer (https://blog.strategyzer.com/posts/2017/3/21/how-strong-is-your-innovation-evidence)
   Strength of evidence vs. time-to-insight
   7
   Strong
   evidence
   Weak
   evidence
   Short time to set
   up and conduct
   experiment
   Long time to set
   up and conduct
   experiment
   Landing page-test
   Wizard of Oz
   Concierge
   Pre-sales
   Validation interview
   Discovery interview
   Crowdfunding
   

   View Slide

8. Fulfilled all prerequisites?
    Test card: hypothesis, metric, success criteria
    Matching type of experiment
    Budget (team, time, money)
   8
   

   View Slide

9. Learnings
   9
   

   View Slide

10. You are not the user!
    10
    Slider
    Feature w/
    augmented
    reality
    

    View Slide

11. Automate user research recruitment
    11
    Find Set up
    Schedule
    Acquire
    

    View Slide

12. The simpler, the better – and faster
    12
    

    View Slide

13. The simpler, the better – and faster
    13
    

    View Slide

14. The simpler, the better – and faster
    14
    

    View Slide

15. User-test your user testing tools
    15
    

    View Slide

16. Iterate within the experiment
    16
    18 – 9 o'clock
    Advertisment
    live
    9 – 11 o'clock
    Analysis of
    replays & data
    11 - 12 o'clock
    Review & planning
    12 - 17 o'clock
    Write, design and implement
    changes & extensions
    17 - 18 o'clock
    QA & bug fixing
    

    View Slide

17. 17
    v1
    v2
    v3
    

    View Slide

18. Learnings to navigate the tool jungle
     Phrase revisable hypothesis
     Find type of experiment
     Acquire budget
    › You are not the user
    › Automatate user research recruitment
    › The simpler, the better – and faster
    › User-test your user testing tools
    › Iterate within the experiment
    18
    

    View Slide

19. 19
    Payment
    A/B-Testing
    Prototyping
    Analytics
    Page Builder
    Social Media
    Ads
    Forms
    Automation
    Scheduling
    Organizing
    

    View Slide

20. Thank you!
    Thomas Leitermann
    Product Manager
    @thomasleiterman
    inovex GmbH
    blog.inovex.de
    www.inovex.de
    

    View Slide