Save 37% off PRO during our Black Friday Sale! »

Why Experimentation is Much More Than A/B-Testing

Why Experimentation is Much More Than A/B-Testing

In this talk I explain why I believe that experimentation is much more than A/B-testing.

I show that there are numerous other methods of experimentation and A/B testing is only one of them. It's also an expensive method and only helps understand the "what", but not the "why"-questions.
In order to find the best matching experiment method I show how to work backwards from the assumption.
Then I show some #NoCode tools that allow you to start as simple as possible. No need to write a line of code to get started with experimentation - and learning!

B7c1fe3603011ec2ad9070edc4dee54c?s=128

Thomas Leitermann

May 27, 2020
Tweet

Transcript

  1. Why Experimentation is Much More Than A/B-Testing Thomas Leitermann @thomasleiterman

  2. Who has run an A/B-test in the last month?

  3. 4 ?

  4. 5 A/B testing – the answer to everything! ?

  5. A/B test = current solution vs. new solution 6 Building

    is the most expensive way of testing
  6. A/B testing is an Experiment Method 7 Product Market Goal

    Focus Method Focus Method Discover / Generate Problem  Solution interview  Ethnography  Concierge-test  Demo pitch  ... Customer  Discovery interview  Ethnography  Surveys, focus groups  ... Evaluate / Validate Solution  Prototypes (paper, clickdummy)  Wizard of Oz-test  Analytics  Usability test  A/B test  ... Revenue  Landing Page-Test  Sales pitch & Pre-sales  Fake door  5-second test  Flyer  ...
  7. A/B testing is an Experiment Method 8 Product Market Goal

    Focus Method Focus Method Discover / Generate Problem  Solution interview  Ethnography  Concierge-test  Demo pitch  ... Customer  Discovery interview  Ethnography  Surveys, focus groups  ... Evaluate / Validate Solution  Prototypes (paper, clickdummy)  Wizard of Oz-test  Analytics  Usability test  A/B test  ... Revenue  Landing Page-Test  Sales pitch & Pre-sales  Fake door  5-second test  Flyer  ...
  8. A/B testing is an Experiment Method 9 Product Market Goal

    Focus Method Focus Method Discover / Generate Problem  Solution interview  Ethnography  Concierge-test  Demo pitch  ... Customer  Discovery interview  Ethnography  Surveys, focus groups  ... Evaluate / Validate Solution  Prototypes (paper, clickdummy)  Wizard of Oz-test  Analytics  Usability test  A/B test  ... Revenue  Landing Page-Test  Sales pitch & Pre-sales  Fake door  5-second test  Flyer  ...
  9. Inspired by „The Real Startup Book“, www.realstartupbook.com 10 Goal Focus

    Questions Evaluate / Validate Solution  Does it work?  What do customers like?  Which solution works better?
  10. Inspired by „The Real Startup Book“, www.realstartupbook.com 11 Goal Focus

    Questions Evaluate / Validate Revenue  Are customers willing to pay?  How much?  How do we convince customers to buy (more)?  For which solution are customers willing to pay more?
  11. Inspired by „The Real Startup Book“, www.realstartupbook.com 12 Goal Focus

    Questions Discover/ Generate Problem  How do we create value for the customer?  What form does it take?  What is a minimal feature set?
  12. Inspired by „The Real Startup Book“, www.realstartupbook.com 13 Goal Focus

    Questions Discover/ Generate Customer  Who are the customers?  What are their pains and gains?  What do they need done?  How do we find those customers?  For which solution are customers willing to pay more?
  13. Inspired by „The Real Startup Book“, www.realstartupbook.com Find experiment methods

    once you know goal & focus 14 Product Market Goal Focus Method Focus Method Discover / Generate Problem  Solution interview  Ethnography  Concierge-test  Demo pitch  ... Customer  Discovery interview  Ethnography  ... Evaluate / Validate Solution  Prototypes (paper, clickdummy)  Wizard of Oz-test  Analytics  Usability test  A/B test  ... Revenue  Landing Page-Test  Sales pitch & Pre-sales  Fake door  5-second test  Flyer  ...
  14. Inspired by „The Real Startup Book“, www.realstartupbook.com Find experiment methods

    once you know goal & focus 15 Product Market Goal Focus Method Focus Method Discover / Generate Problem  Solution interview  Ethnography  Concierge-test  Demo pitch  ... Customer  Discovery interview  Ethnography  ... Evaluate / Validate Solution  Prototypes (paper, clickdummy)  Wizard of Oz-test  Analytics  Usability test  A/B test  ... Revenue  Landing Page-Test  Sales pitch & Pre-sales  Fake door  5-second test  Flyer  ...
  15. A/B testing usually requires a new solution to be built.

    But building is the most expensive way of testing. 16 A/B testing helps evaluate options, answering “What”-questions. But it does not help answer the “Why”-questions.
  16. © Business Model Foundry AG Experiment at-a-glance 17

  17. © Business Model Foundry AG Experiment at-a-glance 18 Assumption Experiment

    Measurement Success criteria
  18. © Business Model Foundry AG Experiment at-a-glance 19 Assumption Experiment

    Measurement Success criteria Users appreciate our value proposition so much, they are willing to pay for it. Insert a paywall after a few example questions. the conversion rate to purchasing the whole course. We are … … super happy with 3% … okay with 1,3% … disappointed with 0,5%. „Read your plants“-App
  19. Learnings From the Shop Floor 20

  20. 21 #NoCode „build software without writing a line of code“

  21. 22 #NoCode „build software without writing a line of code“

  22. 23 #NoCode „build software without writing a line of code“

  23. 24

  24. The simpler, the better – and faster 25

  25. The simpler, the better – and faster 26

  26. The simpler, the better – and faster 27

  27. The simpler, the better – and faster 28

  28. Takeaways A/B testing is only ONE method of experimentation Work

    backwards from the assumption #NoCode aka start as simple as possible 29
  29. Further Reading Directories of no-code-tools:  nocodehq.com/tools  nocodelist.co 

    nocodeessentials.com  nocode.tech  makerpad.co/tools 30 Experiment methods:  RealStartupBook.com  Testing Business Ideas (David Bland)
  30. 31 Thomas Leitermann @thomasleiterman linkedin.com/in/thomasleitermann If you‘re in Amsterdam, lets

    meet for a coffee!