Expedia Tech Know How Talks October 2016: Multi-Variate (A/B) Testing at Expedia

Slides for a meetup hosted and run by Expedia London (https://www.meetup.com/Expedia-London/events/233765707/). It is part of a series called "Know How" where subject matter experts from Expedia London talk all sorts of tech, from Java to Front-End.

Transcript

1. EXPEDIA TECH KNOW HOW MULTI-VARIATE (A/B) TESTING AT EXPEDIA Ralph Jones < > Engineering Manager, Experiment Engineering raljones@expedia.com
2. WHAT'S THE POINT? Basics of A/B testing Expedia's tools Timeline / Lessons Learnt
3. WHAT IS AN EXPERIMENT?
4. ANALYSIS
5. LIFECYCLE
6. TEST & LEARN DASHBOARD
7. EXPEDIA'S TOOLS TO CODE EXPERIMENTS
8. HOW DO TEAMS USE ABACUS?
9. HOW DOES THE ABACUS WORK?
10. HOW DOES THE ABACUS WORK?
11. HISTORY DAY 0
12. BATCH
13. HISTORY DAY 1
14. REALTIME
15. HISTORY DAY 2
16. WHY PRESERVE?
17. BAD COOKIE
18. BAD STATE
19. HISTORY DAY 3
20. NON-CUSTOMER EXPERIMENTS
21. HISTORY DAY 4
22. MICROSERVICES
23. KEYS TO SUCCESS
24. END.
25. APPENDIX 1: INTERACTIONS How can I be sure that having all these tests running at the same time does not cause havoc or mayhem?
26. HOLDOUT GROUP
27. DATA MINING
28. APPENDIX 2: OTHER INTERESTING(?) EXPERIMENT THINGS Multi-Armed Bandit CausalImpact