Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Navigating the tool jungle - How to pick the right tools for your hypothesis experiments

Navigating the tool jungle - How to pick the right tools for your hypothesis experiments

You've lined up everything for your experiment: type of experiment, hypotheses & success criteria, team & time - but how to get started now? How to pick the right tools to actually build, run and evaluate the experiment?

In this talk I help you pick the best tools for your user research & prototyping experiments. I'll share what we learned from running experiments - and going through the test-fail-learn cycle with our tool choices numerous times.

Our three key learnings are:
- Iterate within your experiment
- User-test your usertesting tools
- The simpler, the better - and faster

For each learning I provide a real-life case with plenty of insights.

B7c1fe3603011ec2ad9070edc4dee54c?s=128

Thomas Leitermann

May 24, 2019
Tweet

Transcript

  1. Navigating the tool jungle - How to pick the right

    tools for your experiments Thomas Leitermann Product Manager MTP Engage Hamburg 24.05.2019 @thomasleiterman
  2. 2

  3. Prerequisites 3

  4. © Business Model Foundry AG Set target & success criteria

    4 Hypothesis Experiment Metric Criteria Users appreciate our value proposition so much, they are willing to pay for it. Insert a paywall after a few example questions. the conversion rate to purchasing the whole course. We are … … super happy with 3% … okay with 1,3% … disappointed with 0,5%. Assumption Measurement Success „Read your plants“-App
  5. › Evaluative or generative? › Strength of evidence vs. time-to-insight

    Type of experiment 5
  6. Inspired by „The Real Startup Book“, www.realstartupbook.com Evaluative or Generative

    Experiment? 6 Market Product Question Method Question Method Generative Customer  Discovery interview  Ethnography  Surveys, focus groups Problem  Solution interview  Ethnography  Concierge-test  Demo pitch Evaluiative Revenue  Landing Page-Test  Sales pitch & Pre-sales  Fake door  5-second test  Flyer Solution  Prototypes (paper, clickdummy)  Wizard of Oz-test  Analytics  Usability test  Finished product
  7. Derived from A.Osterwalder, Strategyzer (https://blog.strategyzer.com/posts/2017/3/21/how-strong-is-your-innovation-evidence) Strength of evidence vs. time-to-insight

    7 Strong evidence Weak evidence Short time to set up and conduct experiment Long time to set up and conduct experiment Landing page-test Wizard of Oz Concierge Pre-sales Validation interview Discovery interview Crowdfunding
  8. Fulfilled all prerequisites?  Test card: hypothesis, metric, success criteria

     Matching type of experiment  Budget (team, time, money) 8
  9. Learnings 9

  10. You are not the user! 10 Slider Feature w/ augmented

    reality
  11. Automate user research recruitment 11 Find Set up Schedule Acquire

  12. The simpler, the better – and faster 12

  13. The simpler, the better – and faster 13

  14. The simpler, the better – and faster 14

  15. User-test your user testing tools 15

  16. Iterate within the experiment 16 18 – 9 o'clock Advertisment

    live 9 – 11 o'clock Analysis of replays & data 11 - 12 o'clock Review & planning 12 - 17 o'clock Write, design and implement changes & extensions 17 - 18 o'clock QA & bug fixing
  17. 17 v1 v2 v3

  18. Learnings to navigate the tool jungle  Phrase revisable hypothesis

     Find type of experiment  Acquire budget › You are not the user › Automatate user research recruitment › The simpler, the better – and faster › User-test your user testing tools › Iterate within the experiment 18
  19. 19 Payment A/B-Testing Prototyping Analytics Page Builder Social Media Ads

    Forms Automation Scheduling Organizing
  20. Thank you! Thomas Leitermann Product Manager @thomasleiterman inovex GmbH blog.inovex.de

    www.inovex.de