Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Why Experimentation is Much More Than A/B-Testing

Why Experimentation is Much More Than A/B-Testing

In this talk I explain why I believe that experimentation is much more than A/B-testing.

I show that there are numerous other methods of experimentation and A/B testing is only one of them. It's also an expensive method and only helps understand the "what", but not the "why"-questions.
In order to find the best matching experiment method I show how to work backwards from the assumption.
Then I show some #NoCode tools that allow you to start as simple as possible. No need to write a line of code to get started with experimentation - and learning!

Thomas Leitermann

May 27, 2020
Tweet

More Decks by Thomas Leitermann

Other Decks in Technology

Transcript

  1. 4 ?

  2. A/B test = current solution vs. new solution 6 Building

    is the most expensive way of testing
  3. A/B testing is an Experiment Method 7 Product Market Goal

    Focus Method Focus Method Discover / Generate Problem  Solution interview  Ethnography  Concierge-test  Demo pitch  ... Customer  Discovery interview  Ethnography  Surveys, focus groups  ... Evaluate / Validate Solution  Prototypes (paper, clickdummy)  Wizard of Oz-test  Analytics  Usability test  A/B test  ... Revenue  Landing Page-Test  Sales pitch & Pre-sales  Fake door  5-second test  Flyer  ...
  4. A/B testing is an Experiment Method 8 Product Market Goal

    Focus Method Focus Method Discover / Generate Problem  Solution interview  Ethnography  Concierge-test  Demo pitch  ... Customer  Discovery interview  Ethnography  Surveys, focus groups  ... Evaluate / Validate Solution  Prototypes (paper, clickdummy)  Wizard of Oz-test  Analytics  Usability test  A/B test  ... Revenue  Landing Page-Test  Sales pitch & Pre-sales  Fake door  5-second test  Flyer  ...
  5. A/B testing is an Experiment Method 9 Product Market Goal

    Focus Method Focus Method Discover / Generate Problem  Solution interview  Ethnography  Concierge-test  Demo pitch  ... Customer  Discovery interview  Ethnography  Surveys, focus groups  ... Evaluate / Validate Solution  Prototypes (paper, clickdummy)  Wizard of Oz-test  Analytics  Usability test  A/B test  ... Revenue  Landing Page-Test  Sales pitch & Pre-sales  Fake door  5-second test  Flyer  ...
  6. Inspired by „The Real Startup Book“, www.realstartupbook.com 10 Goal Focus

    Questions Evaluate / Validate Solution  Does it work?  What do customers like?  Which solution works better?
  7. Inspired by „The Real Startup Book“, www.realstartupbook.com 11 Goal Focus

    Questions Evaluate / Validate Revenue  Are customers willing to pay?  How much?  How do we convince customers to buy (more)?  For which solution are customers willing to pay more?
  8. Inspired by „The Real Startup Book“, www.realstartupbook.com 12 Goal Focus

    Questions Discover/ Generate Problem  How do we create value for the customer?  What form does it take?  What is a minimal feature set?
  9. Inspired by „The Real Startup Book“, www.realstartupbook.com 13 Goal Focus

    Questions Discover/ Generate Customer  Who are the customers?  What are their pains and gains?  What do they need done?  How do we find those customers?  For which solution are customers willing to pay more?
  10. Inspired by „The Real Startup Book“, www.realstartupbook.com Find experiment methods

    once you know goal & focus 14 Product Market Goal Focus Method Focus Method Discover / Generate Problem  Solution interview  Ethnography  Concierge-test  Demo pitch  ... Customer  Discovery interview  Ethnography  ... Evaluate / Validate Solution  Prototypes (paper, clickdummy)  Wizard of Oz-test  Analytics  Usability test  A/B test  ... Revenue  Landing Page-Test  Sales pitch & Pre-sales  Fake door  5-second test  Flyer  ...
  11. Inspired by „The Real Startup Book“, www.realstartupbook.com Find experiment methods

    once you know goal & focus 15 Product Market Goal Focus Method Focus Method Discover / Generate Problem  Solution interview  Ethnography  Concierge-test  Demo pitch  ... Customer  Discovery interview  Ethnography  ... Evaluate / Validate Solution  Prototypes (paper, clickdummy)  Wizard of Oz-test  Analytics  Usability test  A/B test  ... Revenue  Landing Page-Test  Sales pitch & Pre-sales  Fake door  5-second test  Flyer  ...
  12. A/B testing usually requires a new solution to be built.

    But building is the most expensive way of testing. 16 A/B testing helps evaluate options, answering “What”-questions. But it does not help answer the “Why”-questions.
  13. © Business Model Foundry AG Experiment at-a-glance 19 Assumption Experiment

    Measurement Success criteria Users appreciate our value proposition so much, they are willing to pay for it. Insert a paywall after a few example questions. the conversion rate to purchasing the whole course. We are … … super happy with 3% … okay with 1,3% … disappointed with 0,5%. „Read your plants“-App
  14. 24

  15. Takeaways A/B testing is only ONE method of experimentation Work

    backwards from the assumption #NoCode aka start as simple as possible 29
  16. Further Reading Directories of no-code-tools:  nocodehq.com/tools  nocodelist.co 

    nocodeessentials.com  nocode.tech  makerpad.co/tools 30 Experiment methods:  RealStartupBook.com  Testing Business Ideas (David Bland)