Why Experimentation is Much More Than A/B-Testing

In this talk I explain why I believe that experimentation is much more than A/B-testing.

I show that there are numerous other methods of experimentation and A/B testing is only one of them. It's also an expensive method and only helps understand the "what", but not the "why"-questions.
In order to find the best matching experiment method I show how to work backwards from the assumption.
Then I show some #NoCode tools that allow you to start as simple as possible. No need to write a line of code to get started with experimentation - and learning!

Transcript

1. Why Experimentation is Much More Than A/B-Testing Thomas Leitermann @thomasleiterman
2. Who has run an A/B-test in the last month?
3. 4 ?
4. 5 A/B testing – the answer to everything! ?
5. A/B test = current solution vs. new solution 6 Building is the most expensive way of testing
6. A/B testing is an Experiment Method 7 Product Market Goal Focus Method Focus Method Discover / Generate Problem  Solution interview  Ethnography  Concierge-test  Demo pitch  ... Customer  Discovery interview  Ethnography  Surveys, focus groups  ... Evaluate / Validate Solution  Prototypes (paper, clickdummy)  Wizard of Oz-test  Analytics  Usability test  A/B test  ... Revenue  Landing Page-Test  Sales pitch & Pre-sales  Fake door  5-second test  Flyer  ...
7. A/B testing is an Experiment Method 8 Product Market Goal Focus Method Focus Method Discover / Generate Problem  Solution interview  Ethnography  Concierge-test  Demo pitch  ... Customer  Discovery interview  Ethnography  Surveys, focus groups  ... Evaluate / Validate Solution  Prototypes (paper, clickdummy)  Wizard of Oz-test  Analytics  Usability test  A/B test  ... Revenue  Landing Page-Test  Sales pitch & Pre-sales  Fake door  5-second test  Flyer  ...
8. A/B testing is an Experiment Method 9 Product Market Goal Focus Method Focus Method Discover / Generate Problem  Solution interview  Ethnography  Concierge-test  Demo pitch  ... Customer  Discovery interview  Ethnography  Surveys, focus groups  ... Evaluate / Validate Solution  Prototypes (paper, clickdummy)  Wizard of Oz-test  Analytics  Usability test  A/B test  ... Revenue  Landing Page-Test  Sales pitch & Pre-sales  Fake door  5-second test  Flyer  ...
9. Inspired by „The Real Startup Book“, www.realstartupbook.com 10 Goal Focus Questions Evaluate / Validate Solution  Does it work?  What do customers like?  Which solution works better?
10. Inspired by „The Real Startup Book“, www.realstartupbook.com 11 Goal Focus Questions Evaluate / Validate Revenue  Are customers willing to pay?  How much?  How do we convince customers to buy (more)?  For which solution are customers willing to pay more?
11. Inspired by „The Real Startup Book“, www.realstartupbook.com 12 Goal Focus Questions Discover/ Generate Problem  How do we create value for the customer?  What form does it take?  What is a minimal feature set?
12. Inspired by „The Real Startup Book“, www.realstartupbook.com 13 Goal Focus Questions Discover/ Generate Customer  Who are the customers?  What are their pains and gains?  What do they need done?  How do we find those customers?  For which solution are customers willing to pay more?
13. Inspired by „The Real Startup Book“, www.realstartupbook.com Find experiment methods once you know goal & focus 14 Product Market Goal Focus Method Focus Method Discover / Generate Problem  Solution interview  Ethnography  Concierge-test  Demo pitch  ... Customer  Discovery interview  Ethnography  ... Evaluate / Validate Solution  Prototypes (paper, clickdummy)  Wizard of Oz-test  Analytics  Usability test  A/B test  ... Revenue  Landing Page-Test  Sales pitch & Pre-sales  Fake door  5-second test  Flyer  ...
14. Inspired by „The Real Startup Book“, www.realstartupbook.com Find experiment methods once you know goal & focus 15 Product Market Goal Focus Method Focus Method Discover / Generate Problem  Solution interview  Ethnography  Concierge-test  Demo pitch  ... Customer  Discovery interview  Ethnography  ... Evaluate / Validate Solution  Prototypes (paper, clickdummy)  Wizard of Oz-test  Analytics  Usability test  A/B test  ... Revenue  Landing Page-Test  Sales pitch & Pre-sales  Fake door  5-second test  Flyer  ...
15. A/B testing usually requires a new solution to be built. But building is the most expensive way of testing. 16 A/B testing helps evaluate options, answering “What”-questions. But it does not help answer the “Why”-questions.
16. © Business Model Foundry AG Experiment at-a-glance 17
17. © Business Model Foundry AG Experiment at-a-glance 18 Assumption Experiment Measurement Success criteria
18. © Business Model Foundry AG Experiment at-a-glance 19 Assumption Experiment Measurement Success criteria Users appreciate our value proposition so much, they are willing to pay for it. Insert a paywall after a few example questions. the conversion rate to purchasing the whole course. We are … … super happy with 3% … okay with 1,3% … disappointed with 0,5%. „Read your plants“-App
19. Learnings From the Shop Floor 20
20. 21 #NoCode „build software without writing a line of code“
21. 22 #NoCode „build software without writing a line of code“
22. 23 #NoCode „build software without writing a line of code“
24. The simpler, the better – and faster 25
25. The simpler, the better – and faster 26
26. The simpler, the better – and faster 27
27. The simpler, the better – and faster 28
28. Takeaways A/B testing is only ONE method of experimentation Work backwards from the assumption #NoCode aka start as simple as possible 29
29. Further Reading Directories of no-code-tools:  nocodehq.com/tools  nocodelist.co  nocodeessentials.com  nocode.tech  makerpad.co/tools 30 Experiment methods:  RealStartupBook.com  Testing Business Ideas (David Bland)
30. 31 Thomas Leitermann @thomasleiterman linkedin.com/in/thomasleitermann If you‘re in Amsterdam, lets meet for a coffee!