Booking.com

Transcript

1. App experimentation as a strategy at Booking.com 3 reasons for running experiments on apps Jorden Lentze, September 2022 MOBILE APPS
2. Would it surprise you that 37.6% of all experiments at Booking.com are running on apps?
3. Who has ever run experiments on apps?
4. When I asked on Linkedin 62% Source: survey on Linkedin
5. 01 Key takeaways Follow your customers 02 User experience is key for apps 03 Helps execute on your strategy
6. Experimentation at Booking.com 500+ concurrent experiments
7. 90% 10% 9/10 experiments fail for users. Source: VWO 2016
8. 01 Validate business decisions 02 Learn faster 03 Reduce deployment risk 04 Do not hurt our customers 05 Improve experience Benefits of experimentation
9. Run experiments to add value and not features
10. 01 Experimentation is part of our strategy Informs strategy 02 Validates strategic choices 03 Increases experiment quality
11. Source
12. • Test everything • Anybody can run experiments • No hippos • Teams made for experimentation • 100% access to data What does this mean?
13. And this results in………
14. 2011 Start with the development of our first app 1 team 2018 Increased focus 25 teams 2022 Key part of our strategy 60+ teams 2005 First experiment at Booking.com
15. 1. Follow your customers Run experiments on the most important channels for your customers
16. 01 Follow our customers Our customers are on our apps 02 Customers come straight to us 03 They come back to Booking more often using apps Guests love the ease of use of our apps Source: Booking earnings call
18. Experimentation measurably helps improve the customer experience 1. Follow your customers 2. User experience is key for apps
19. Challenges for apps • Smaller screens • High expectations • Speed matters
20. Confidential + Proprietary A better experience trumps everything - Jared Spool
21. Small changes can influence the experience of our customers
22. Small changes can influence the experience of our customers This icon might be confusing guests?
23. 100% bug fix policy
24. Speed matters. We care about app performance
25. The higher up in the funnel, the bigger the impact
26. User experience is key for apps because: - small screens - high expectations - speed matters
27. 3. Helps execute on your strategy Informs and validates your decisions
28. Informs strategy
29. Some thoughts to get you started
31. • Firebase Remote Config • LaunchDarkly • Optimizely • Split.io • VWO • AB tasty • A/B smartly • Build your own Available tools
32. • Release process • Check if you have enough traffic • Understand your customers • Think about parity across platforms • Spreading hypothesis across platforms? Before you start running app experiments
33. 01 Recap Follow your customers 02 User experience is key for apps 03 Execute on your strategy faster
34. Questions?