AB Testing - Why? What? How? - Web2Day 2017

How often do you change the colour of a button on your website? How often do you change the layout of a part of the website? How do you measure the effect of these changes? Do you just make an assumption that your changes will be liked by your users/customers? I will be talking about a data-driven approach – A/B testing. I plan to cover these aspects –

– Why should you use A/B testing in your business?
– What all can you test using A/B testing?
– How can you do efficient A/B testing of change in your product?

I will also cover some scenarios where A/B testing may not work or may fail and how you can overcome those limitations by being careful.

Transcript

1. @sahildua2305 #web2day #nantes #abtesting A/B Testing Why? What? How? Sahil Dua
2. @sahildua2305 #web2day #nantes #abtesting Booking.com Graduate Software Developer Git Pandas Open Source Contributor Open Source Contributor DuckDuckGo Community Leader Sahil Dua @sahildua2305
3. @sahildua2305 #web2day #nantes #abtesting Agenda 1. A/B Testing - Quick Introduction 2. Why A/B Testing is important? 3. What all can you test? 4. A/B Testing process 5. Why do tests fail? 6. Conclusion
4. @sahildua2305 #web2day #nantes #abtesting A/B Testing Conducting experiments to optimize your customer experience
5. @sahildua2305 #web2day #nantes #abtesting Graphics credits: @endesignonline
6. @sahildua2305 #web2day #nantes #abtesting Graphics credits: @endesignonline
7. @sahildua2305 #web2day #nantes #abtesting WHY?
8. @sahildua2305 #web2day #nantes #abtesting Why A/B Testing? 1. Your customers decide what stays
9. @sahildua2305 #web2day #nantes #abtesting Why A/B Testing? 1. Your customers decide what stays 2. Easy to measure the business impact
10. @sahildua2305 #web2day #nantes #abtesting Why A/B Testing? 1. Your customers decide what stays 2. Easy to measure the business impact 3. Not dependent on opinion or assumption
11. @sahildua2305 #web2day #nantes #abtesting Why A/B Testing? 1. Your customers decide what stays 2. Easy to measure the business impact 3. Not dependent on opinion or assumption 4. Cuts down on functionality bloat
12. @sahildua2305 #web2day #nantes #abtesting Why A/B Testing? 1. Your customers decide what stays 2. Easy to measure the business impact 3. Not dependent on opinion or assumption 4. Cuts down on functionality bloat 5. General product deployment is easier
13. @sahildua2305 #web2day #nantes #abtesting Why A/B Testing? 1. Your customers decide what stays 2. Easy to measure the business impact 3. Not dependent on opinion or assumption 4. Cuts down on functionality bloat 5. General product deployment is easier 6. and finally ...
14. @sahildua2305 #web2day #nantes #abtesting TO LEARN
15. @sahildua2305 #web2day #nantes #abtesting The key to learning is feedback. It is nearly impossible to learn anything without it. “ - Stephen D. Levitt
16. @sahildua2305 #web2day #nantes #abtesting Why A/B Testing? 1. Your customers decide what stays 2. Easy to measure the business impact 3. Not dependent on opinion or assumption 4. Cuts down on functionality bloat 5. General product deployment is easier 6. Learn from your customers
17. @sahildua2305 #web2day #nantes #abtesting If you can not measure it, you can not improve it. “ - Lord Kelvin
18. @sahildua2305 #web2day #nantes #abtesting WHAT?
19. @sahildua2305 #web2day #nantes #abtesting ANYTHING that you want
20. @sahildua2305 #web2day #nantes #abtesting What to A/B test? Webpage 1. Call-to-Action 2. Content 3. Copy 4. Visual media 5. Site navigation 6. Personalization
21. @sahildua2305 #web2day #nantes #abtesting What to A/B test? Webpage 1. Call-to-Action 2. Content 3. Copy 4. Visual media 5. Site navigation 6. Personalization Social 1. Social icons 2. Profile images 3. Social ads 4. Social messaging
22. @sahildua2305 #web2day #nantes #abtesting What to A/B test? Email 1. “From” name 2. Greeting 3. Subject length 4. Images 5. Personalization 6. Optimal time
23. @sahildua2305 #web2day #nantes #abtesting What to A/B test? Email 1. “From” name 2. Greeting 3. Subject length 4. Images 5. Personalization 6. Optimal time Mobile 1. Navigation 2. Display 3. Push notifications 4. Deep linking
24. @sahildua2305 #web2day #nantes #abtesting A/B Testing is a tool to collect data so that you can make accurate decisions. “
25. @sahildua2305 #web2day #nantes #abtesting HOW?
26. @sahildua2305 #web2day #nantes #abtesting A/B Testing Process STEP 1 Observe a problem STEP 2 Write a hypothesis STEP 3 Brainstorm solutions STEP 4 Run the experiment STEP 5 Analyze the results, make a decision
27. @sahildua2305 #web2day #nantes #abtesting STEP 1 Observe a problem QUALITATIVE QUANTITATIVE Graphics credits: @endesignonline
28. @sahildua2305 #web2day #nantes #abtesting Qualitative data tells you what to test Quantitative data tells you where to test
29. @sahildua2305 #web2day #nantes #abtesting Qualitative Data 1. User Testing 2. Survey Data 3. Heat Maps 4. Customer-facing Teams
30. @sahildua2305 #web2day #nantes #abtesting Quantitative Data 1. Web traffic 2. Email marketing 3. Purchase history 4. Customer support tickets
31. @sahildua2305 #web2day #nantes #abtesting STEP 2 Write a hypothesis Graphics credits: @endesignonline
32. @sahildua2305 #web2day #nantes #abtesting STEP 3 Brainstorm solutions Graphics credits: @endesignonline
33. @sahildua2305 #web2day #nantes #abtesting STEP 4 Run the experiment Graphics credits: @endesignonline
34. @sahildua2305 #web2day #nantes #abtesting STEP 5 Analyze the results, make a decision Graphics credits: @endesignonline
35. @sahildua2305 #web2day #nantes #abtesting What are we looking for? 1. Confidence Interval 2. Statistical Significance 95% Statistical Significance = 5% probability that the observed difference was due to chance
36. @sahildua2305 #web2day #nantes #abtesting What are we looking for? High statistical confidence Lower risk that it was due to chance
37. @sahildua2305 #web2day #nantes #abtesting Testing Segmentation Segment by Source Segment by Behaviour Segment by Outcome Segment by Demographic
38. @sahildua2305 #web2day #nantes #abtesting What metrics are important for you?
39. @sahildua2305 #web2day #nantes #abtesting Metrics to measure Webpage 1. Page views 2. Sign ups 3. Clicks on CTAs 4. Conversion Social 1. Images uploaded 2. Sign ups 3. Active users 4. Session duration
40. @sahildua2305 #web2day #nantes #abtesting Metrics to measure Email 1. Open ratio 2. Links clicked 3. Conversion Mobile 1. Notifications clicked 2. Clicks on CTAs 3. Conversion
41. @sahildua2305 #web2day #nantes #abtesting There are far more ways to fail than there are to succeed. “
42. @sahildua2305 #web2day #nantes #abtesting Why do tests fail?
43. @sahildua2305 #web2day #nantes #abtesting Technical Flaws - Increase in page load time? - New errors or warnings? - Cross-browser issues? - Cross-device issues?
44. @sahildua2305 #web2day #nantes #abtesting Missing Context - Is something less visible now? - Is it in the wrong spot? - Did you remove something to “make space”?
45. @sahildua2305 #web2day #nantes #abtesting Untargeted Audience - Did they type the URL directly? - Are they returning or new visitors? - Did they come from an email campaign? - Are they regular customers?
46. @sahildua2305 #web2day #nantes #abtesting Timing Issues - Is your change seasonal? - Showing the feature too early? - Does the tooltip disappear too fast?
47. @sahildua2305 #web2day #nantes #abtesting Low Traffic Do you have enough traffic? https://www.optimizely.com/resources/sample-si ze-calculator/
48. @sahildua2305 #web2day #nantes #abtesting
49. @sahildua2305 #web2day #nantes #abtesting
50. @sahildua2305 #web2day #nantes #abtesting
51. @sahildua2305 #web2day #nantes #abtesting
52. @sahildua2305 #web2day #nantes #abtesting Why do tests fail? - Technical flaws - Missing context - Untargeted audience - Timing issues - Low traffic
53. @sahildua2305 #web2day #nantes #abtesting Conclusion 1. Start testing now 2. Confirm your instincts 3. Learn and take actions
54. @sahildua2305 #web2day #nantes #abtesting LinkedIn @sahildua2305 GitHub Twitter @sahildua2305 @sahildua2305 Website www.sahildua.com Want to get in touch?
55. @sahildua2305 #web2day #nantes #abtesting Let’s be friends @sahildua2305