TEAM GROWTH
2012
2010
a/b testing, iterations, start small.
still works!
Slide 15
Slide 15 text
SLOW ROLLOUT
Release process
Release features to segments of users first.
Slide 16
Slide 16 text
MOBILE AND NATIVE
Release process
Add mobile and native device support eventually.
Slide 17
Slide 17 text
Engineering process
Build it Make it pretty Ship it Make it fast
Make
it
work
Add
tests
Finalize
design
and
user
experience
Verify
and
measure
usage
Scaling
Slide 18
Slide 18 text
Engineering culture
Optimize for quick recovery from mistakes.
Learn from mistakes. Share remediations with team.
Slide 19
Slide 19 text
Test ideas cheaply
Slide 20
Slide 20 text
PURCHASE FEEDBACK
Slide 21
Slide 21 text
TWO ASSUMPTIONS
Consumers would give feedback
1 2
1
2 Merchants would find the feedback valuable
PURCHASE FEEDBACK
Slide 22
Slide 22 text
OVERALL: SUCCESS
Surveyed millions of people. Collected millions of
records.
Could have validated product interest with merchant
using sample and survey data
PURCHASE FEEDBACK
Could have validated with consumers before building
MISSED TARGET
Tested on a small set of deals (good)
Failed to achieve a statistically significant increase in
purchase conversion.
QUICK CHECKOUT
Removed code
Slide 27
Slide 27 text
FACEPILE
Slide 28
Slide 28 text
FACEPILE
WINNER!
Better subscription conversion from paid clicks
originating from Facebook ads.
5%
Social proof
Slide 29
Slide 29 text
Automate something after doing
it manually becomes painful, but
not before.
Slide 30
Slide 30 text
WAIT LIST FEATURE
Slide 31
Slide 31 text
HYPOTHESES
Additional purchases on deals from additional
inventory
1
WAIT LIST FEATURE
$
More inventory allocation from merchant for re-runs
2
Slide 32
Slide 32 text
MISSED TARGET
Learned we typically had the maximum inventory from
hotels
Purchase conversion from wait list subscribers was <
10%.
WAIT LIST FEATURE
Learned the majority of deals do not sell out
Small opportunity,
removed code.
Slide 33
Slide 33 text
Automated reporting
merchant notification
Did not build:
Slide 34
Slide 34 text
Test riskiest assumptions first
Slide 35
Slide 35 text
LOYALTY PRODUCT
Slide 36
Slide 36 text
Would consumers sign up?
1 2 Would merchants sign up?
LOYALTY PRODUCT
Would integration
with third-party work?
Quality of the data?
3
Slide 37
Slide 37 text
DID NOT LAUNCH
Working with third-party added extra time to our
development speed
LOYALTY PRODUCT
Could have validated riskiness assumptions first
(merchant enrollment)
Slide 38
Slide 38 text
ESCAPES NEW DEVELOPMENT
Booking
Air
Tours
Payment
Mobile
Slide 39
Slide 39 text
ESCAPES NEW DEVELOPMENT
Manually loading inventory
Building piece by piece (air, packaged tours, mobile)
Deferring merchant payment project while working
with hotels that don’t require it
Starting small
Slide 40
Slide 40 text
CONCLUSION
Validate riskiest assumptions first
Use a/b testing for optimization and new product
development
Optimize for iteration speed and quick recovery
Test small ideas with big potential
Make customer validation your primary goal