Flows, Funnels, and Metrics, Oh My!

A presentation about experimentation on the home onboarding flow at Airbnb by Theresa Johnson and Laura Kelly

Transcript

1. Flows, Funnels and Metrics, Oh My! Home Onboarding @ Airbnb Theresa Johnson & Laura Kelly / April 18th, 2017 / Taking Flight
2. Bios • Dr. Theresa Johnson is a data scientist on the Co-Hosting team. She completed a PhD in Aeronautics and Astronautics at Stanford University before joining Airbnb in 2016. • Laura Kelly is a software engineer on the Vacation Rentals team. She went to Washington University in Saint Louis and studied Cognitive Neuroscience. She joined Airbnb in 2015. • Theresa and Laura worked together to increase New Homes Metrics by 10%.
3. 1. Growth Metrics 2. Creating a Home on Airbnb 3. Description 4. Location 5. Final Thoughts • New Homes Added • New Homes Published • Home Onboarding Funnel • New Homes Booked
4. Curated Growth We are selective about which experiments to run, given our Minimum Detectable Effect. • Find problems using data • Size the impact potential • Define user problem & brainstorm solutions • Size the engineering effort • Prioritize ruthlessly • New Homes Booked • New Homes Published
6. Experiments • Engineer builds the feature • Together determine experiment delivery conditions • Engineer builds experiment delivery • Data scientists code and track metrics
7. Experiments for Homes Onboarding
8. 1. Growth Metrics 2. Creating a Home on Airbnb 3. Description 4. Location 5. Final Thoughts Advertise Settings Upsell Basic Information about your home Upsell with pictures, description & features Calendar, price, house rules Settings Calendar, price, house rules Advertise Basic Information about your home
9. Logging Engineering worked closely with data science BEFORE iterating to log these events • Impressions • Clicks • Step completion • Prior progress
10. Home Onboarding Funnel Jan Feb Mar Apr 30 May 10 20 30 40 0 20 25 10 40 Only 30% of all users make it through the “Upsell” Step 18% of users drop off from the “Upsell” Step
11. 1. Growth Metrics 2. Creating a Home on Airbnb 3. Description 4. Location 5. Final Thoughts The “highlights” based description hurt total reservations for new homes • Increased step completion but created “cookie cutter” descriptions Data Science challenges • Find more specific ‘tags’ that help hosts, but are appealing to guests • Model bookings vs. description length to find best word limit to enforce Engineering Challenges • Sunk cost of the description builder • Data scientists found “highlights” of landmarks near the home from guidebooks • Engineering found local highlights from google maps. • Use these to auto-create sentences to fill in the description.
12. ‘Tags’ Based Description Simple, but effective +2.1% New Homes Booked
13. Home Onboarding Funnel Jan Feb Mar Apr 30 May 10 20 30 40 0 20 25 10 40 Only 30% of all users make it through the “Upsell” Step 25% of users drop off from the “Advertising” Step
14. 1. Growth Metrics 2. Creating a Home on Airbnb 3. Description 4. Location 5. Final Thoughts Location step accounted for nearly all of the drop-off in the ‘Advertise’ section. Data Science challenges • Deep dive to determine % users suffering from google map errors • Model conditional expected conversion on the new step placement E(Y|X=x) Engineering Challenges • Large amount of state related to user's step in the flow and what has been completed -- we had to design the flow so that steps could be moved or added • Listing creation back-end event tied to address step • Location step revealed personal information • Map dragging introduced complexity
15. Location step experiment +2.4% New Homes Added Location Property type Bedrooms Bathrooms Amenities Property type Bedrooms Bathrooms Location step moves to later in the flow Location Amenities After Before
16. 1. Growth Metrics 2. Creating a Home on Airbnb 3. Description 4. Location 5. Final Thoughts • Best practices for successful data science and engineering collaboration ◦ Communicate early and often in the project, way before you're ready to iterate ▪ Define expectations and success ◦ Trust each other and leverage complementary skills ▪ Be strategic ◦ Brainstorm together ▪ Use asynchronous tools to avoid meeting overload • Next Steps ◦ Mobile App New Home Onboarding
17. Questions? theresa.johnson@airbnb.com laura.kelly@airbnb.com