A/B Testing

Using science to make UX decisions and increase conversions.

Presentation given at EE CONF 2018 in Nashville, TN

Transcript

2. Charles Kolner 2
3. Brian Litzinger 3
5. AB / Split Test 5
6. Basics • Learn fundamentals • Establish process • Goal tracking 6
7. Advantages • Objective decision-making process • Focus on action and outcomes • Mitigate risk • Quantify value of design decisions • Competitive advantage • Growth-driver 7
8. Challenges • HiPPOs & Culture • Time & Resources • Setup & Scale 8
9. Goal of Program Convince the marketing organization: 1. This is how we should make decisions about the website 2. Quantify impact of website on business outcomes 3. Drive incremental improvement toward goals 9
10. Keys to Success • Stakeholder buy-in • Informed test plan • Implementation • Actionable process 10 Image from http:/ /eyeflow.com/4-major-keys-to-social-media-success/
11. Stakeholder Buy-in Generate business case or proof of concept • Identify areas most closely tied to business outcomes & website success • Quantify the value of conversions to show opportunity potential of program 11
12. Informed Test Plan Focus efforts on resolving pain-points. • Internal discussions and research • WHAT users do - web data • WHY users do it - survey and feedback • Pipeline of potential tests based on greatest impact 12
13. Implementation • Client Side • Server Side • Client + Server Side 13
18. Client Side Pros • Easy, Fast, and Cheap Cons • Less control • Flicker effect 18
19. Server Side 19
21. Definining Variants 21
23. Client + Server Side 23
24. Server Side Pros • More Control • No flicker effect Cons • Requires development effort 24
25. 25 optimizely.com/resources/client-vs-server-side-testing-infographic
26. Experiments 26
27. {exp:channel:entries channel="pages" entry_id="1"} {exp:experiments:bloqs} {bloqs_field} # blocks {/bloqs_field} {/exp:experiments:bloqs} {/exp:channel:entries} {exp:experiments:content} {original} Original Content #1 {/original} Always Shown {variant} Variant Content #1 {/variant} {/exp:experiments:content} 27
28. Actionable Process • Scientific method! • Communicate results • Iterate 28
36. Summary • Start now • Move fast • Be aggressive • Take bets • Outsource 36
37. Questions? Please leave feedback! eeconf.com/speaker-feedback 37