The Science of Product at Expedia

Expedia Engineering Tech Talk on 29 November 2016.

Transcript

1. Expedia Global Product – Sam Bradley The Science of Product #ExpediaEng
2. Use science to bring a focus on what to deliver The Science of Product
3. Definition of the Scientific Method Make Observations Think of Interesting Questions Formulate Hypotheses Develop Testable Predictions Gather Data to Test Predictions Develop General Theories Refine, Alter, Expand or Reject Hypothesis
4. The Scientific Method (predicting the future in 4 easy steps)
5. Getting Started with the Scientific Method
6. Start Small and Work Outwards
7. Organize Around a Single Metric
8. Most Importantly, Establish the Culture Culture eats strategy for breakfast - Peter Drucker
9. Learn quickly when you’re wrong...you’ll be wrong a lot – and that’s a great thing. Predicting Outcomes: The Science of Product Will my customers like a feature? Where should I invest and what could it return? What is working? What is not working? What consumer behaviors might change and what won’t? Where to Invest What’s Working? Customer Behavior What to Build
10. Walking through the steps: Step 1
11. The Core Skill of a Scientist is to Make Observations
12. Customer & Performance Data Talk to Customers Study the Industry and Techniques Utilize Internal Research Teams
13. Internal Research Teams
14. Step 2
15. change observations measurable clarity on learn. Forming a Good Hypothesis One based upon that is with what you expect to
16. Hypothesis Template By [making some change], we will [see this quantifiable, measureable change]
17. Hypothesis Example By [displaying urgency messaging for the “last room available” in Checkout, customers will feel a sense of urgency to complete their booking], And we will [see an increase in conversion]
18. Step 3
19. Question and Listen Like a Scientist Democracy of ideas Bias towards learning Data over opinion
20. Three Main Types of A/B Testing
21. Example AB Test Control Test
22. Learn
23. How Do You Know Your Test Worked?
24. Significance – When to evaluate a test 90% 10% = 10% risk of concluding that a difference exists when there is no actual difference 90%
25. Goal is to Learn, Not Be Right 33% 33% 33%
26. Test Everywhere A few signals Some Signals Many Signals A ton of signals
27. Pre-2005 2005 2006 2007 2008 2009 2010 2011 2012 2013 2014 Total Number of A|B Tests Performed Velocity Mitigates Risk
28. First Principles Thesis Hypothesis Tests Learn Breaking Thinking Down
29. Company priorities Site Strategy 120 Day Plans Inceptions Sprints Monthly Product Review Staying Focused
30. Feedback is a Gift
31. And Finally…Always Adjust
32. Thanks For Your Time @SammyBB @ExpediaEng_Lon