Slide 1

Slide 1 text

A Human Touch Evaluation Insights for Airbnb Edwin Chen @echen / echen.me

Slide 2

Slide 2 text

Overview

Slide 3

Slide 3 text

Building a Search and Discovery platform is hard. Building it for an expensive travel experience, when most people only make a few trips each year, is even harder.

Slide 4

Slide 4 text

How well does Airbnb do?

Slide 5

Slide 5 text

Is it easy to find a good listing? How about if I want to stay near the Eiffel Tower? What are problems with its “Similar Listings” algorithm? Where else could the experience be improved?

Slide 6

Slide 6 text

No content

Slide 7

Slide 7 text

Where is Disneyland? Why am I being shown locations so far away?

Slide 8

Slide 8 text

Horrible result. The room is pretty and quaint and the price is right. But again, it's just not what I'm looking for. This room is near downtown Madison, it's not a secluded place and there's no skiing. (from a search keyworded on “skiing”)

Slide 9

Slide 9 text

Click logs in a dynamic marketplace can be sparse and difficult to interpret. Did users skip this listing because it was too expensive, or was it just not relevant? Was this search result page bad because of poor ranking or because all the good listings were already taken?

Slide 10

Slide 10 text

Let’s grab additional data.

Slide 11

Slide 11 text

Human Evaluation

Slide 12

Slide 12 text

Tell me about an upcoming trip you’d like to make. “We love going to Disneyland and would like to go back sometime this year around the beginning of the summer. Ideally we would go for a week but it depends on price. I would want to go with my husband and kids because it's a really fun and special experience for all of us.”

Slide 13

Slide 13 text

Now try to find a place to stay on Airbnb.

Slide 14

Slide 14 text

Rate the first listing Fairly bad result. It looks like a nice play to stay but I don't like that it's actually part of another house. I would want to feel like we have total privacy instead of sharing certain areas. There are also an awful lot of rules about when and what you can do. I understand to an extent but seeing it all written out comes across as kind of negative. It's also not really that close to Disneyland.

Slide 15

Slide 15 text

“It’s also not really that close to Disneyland.” Can we show how far listings are from nearby tourist attractions and landmarks? Why are locations that are so far away from Anaheim being shown?

Slide 16

Slide 16 text

Many travelers are tourists unfamiliar with where they’re traveling. Why not make it easier for them?

Slide 17

Slide 17 text

“There are also an awful lot of rules… I understand to an extent but seeing it all written out comes across as kind of negative.”

Slide 18

Slide 18 text

Other users mention disliking rules as well “I try to stay away from hotels. I find that they are overpriced and dirty. I would rather rent a room in a house, if I couldn't find an entire house, than stay in a hotel. You would think that renting a room or house from someone there would be more rules than a hotel but it is not been my experience. Hotels impose silly rules, and I don't mean standard things like not destroying the room and noise, I mean like not allowing you to open windows or keep a door open to the outside. No thanks! I prefer Airbnb by far.” “It looks like a decent place and how when you first step outside you can see the water and it looks like you can go straight down to the ocean. The price is great for the amount of days I am staying as well and it looks a little homey… The seller also does not seem to have any rules that may complicate me in any way.” “From the pictures, I don't think it would be the kind of place that I would necessarily feel at home in… I don't like the fact that shoes have to be removed before entering the house, since I don't like walking around barefoot. Overall, I just don't think that this room would be the right fit for me.”

Slide 19

Slide 19 text

Users turn to Airbnb for its friendliness and home-like feel. Would it help to boost listings with a friendly tone, and penalize listings with long lists of rules?

Slide 20

Slide 20 text

Rate the second listing Horrible result. The 500 dollar security deposit puts me off immediately. We are looking for something affordable and while the nightly rate is okay the security deposit would just put it over the top. I would be worried we wouldn't get it back no matter how good of a shape we left the house. Also they seem particular about how everything is treated and we have two kids so it just wouldn't work well.

Slide 21

Slide 21 text

Are security deposits a feature in the ranking models? Security deposits are a common gripe. New Airbnb users may be especially worried that an unscrupulous host will keep a security deposit without good reason.

Slide 22

Slide 22 text

Rate the third listing Fairly bad result. I love that the price is so low. Plus there is room for my husband and two kids to sleep. However I don't like that you would be staying in an RV in someone's driveway. There isn't much privacy that way. Plus the distance is reasonable to San Diego and LA which means it's not really close enough to either one.

Slide 23

Slide 23 text

Rate the fourth listing Fantastic result. If I'm understanding correctly, you get the entire house which is great. It looks beautiful on the outside and looks like there is a garage where we could hopefully park our van. The security deposit is not too high. The location isn't perfect but it's pretty good because the drive to Disneyland wouldn't be too long.

Slide 24

Slide 24 text

Rate the fifth listing Pretty good result. The house looks nice outside and in. I like that you get the entire house to yourself. The security deposit is high so that makes me hesitant to want to book it. There is plenty of room for my husband and two kids. I like that we would be able to park in the driveway as we will have our own vehicle.

Slide 25

Slide 25 text

Takeaways • How can we make it easier to gauge the distance to nearby attractions? • Should we penalize strict-looking listings with many rules? • Can we add security deposits as a feature in the ranking models?

Slide 26

Slide 26 text

Search

Slide 27

Slide 27 text

Note: may be a biased sample, from a set of 100 raters.

Slide 28

Slide 28 text

No content

Slide 29

Slide 29 text

How can we make search results better?

Slide 30

Slide 30 text

No content

Slide 31

Slide 31 text

This listing is a little too cutesy for me. I am not a fan of the room in terms of decoration. It looks very clean though, and I wouldn't be too upset if I was "stuck" there. It looks like a really nice space. I would definitely look around a little more to find something that might be more comfortable in terms of style. (from a male user)

Slide 32

Slide 32 text

This place is more in line with what i am looking for. Good decor and furnishings. The place looks modern and very HIP. Has a nice amount of space and nice windows and glass doors. It is almost ideal to what i pictured.

Slide 33

Slide 33 text

Style matters Male users may not want pink, fluffy rooms. Younger users may want hipper, more modern listings. Some places just feel old and dingy. Can we learn the style of a listing, and match it against user demographics?

Slide 34

Slide 34 text

Horrible result. I'm not entirely impressed with the host's response rate/time. It makes me think they would be hard to get in touch with if something were to occur or we needed something. The house itself is nice, but it doesn't look quite as good as other ones that are cheaper. I would pass on this one.

Slide 35

Slide 35 text

Fairly bad result. This place looks a little bit bland. I think it's a little too expensive for what you get as well. I don't really think I would choose this place. It doesn't stand out to me.

Slide 36

Slide 36 text

Cost is relative Some listings are on the higher end of the price range, but others may simply be overpriced. Price is probably already a ranking feature, but should overpricedness be as well? Airbnb already has a price prediction algorithm; can we use the residual as a measure of whether a listing is priced too high or too low?

Slide 37

Slide 37 text

From the pictures, I don't think it would be the kind of place that I would necessarily feel at home in. The late check in also worries me, since I like to get on the road early. I don't like the fact that shoes have to be removed before entering the house, since I don't like walking around barefoot. Overall, I just don't think that this room would be the right fit for me.

Slide 38

Slide 38 text

The pictures look nice, and it's appealing that it is near to the beach. However, this is a room, and I actually wanted a whole space to myself. Also, there are restrictive hours on when to arrive, and that's difficult for me. Plus the price is a bit more than I was hoping to pay for. I don't need bed and breakfast, just a bed.

Slide 39

Slide 39 text

Check-in and check-out times can be problematic Are restrictive arrival and departure times used as a feature?

Slide 40

Slide 40 text

This is the best one yet. The view from the patio is amazing. The price is the cheapest. The amenities are spot on. No need for a television when you have a place like this. We talked about going to Sardinia, and this would be a great reason why, just to spend a night at this place. This place is gorgeous.

Slide 41

Slide 41 text

It looked acceptable. The location just wasn't as good and there weren't spectacular views. If I was going to settle for something that wasn't pretty then it would need to be very convenient, which this isn’t. (from a search whose other results had good views)

Slide 42

Slide 42 text

Can we figure out from a listing’s description, reviews, and pictures whether it’s in a location with great views?

Slide 43

Slide 43 text

This place looks beautiful! It is a little big, however, the price range is not too bad. It has romantic in the title and that is what caught my eye first. I am looking for a romantic vacation with my SO.

Slide 44

Slide 44 text

The room is not ideal, it's not romantic at all. The location is unappealing, it is near an urban area and there is no mention of skiing. I'm looking for a place near a skiing area, I don't have a specific location, because I don't know where they are. The price is okay, it's just not what I want at all.

Slide 45

Slide 45 text

Many vacations revolve around romance: honeymoons, weddings, anniversaries, and special dates. Can we detect which listings and trips are more romantic in mood? What are the other canonical reasons for travel?

Slide 46

Slide 46 text

Based on all 21 pictures shown, it is an extremely attractive place and looks very clean. My girlfriend is particularly picky about living in hotels in regards to hygiene and only wants places that are squeaky clean and have proper mosquito nets. In addition, she wants a bed with enough space for two people and has an air conditioner so that we don't sweat during the nighttime. This location fits the bill for all the conditions, as well as my preferences and looks rather pleasing. (for a trip to Penang)

Slide 47

Slide 47 text

I really would want something different with a different style, as well as a pool, we are really looking for a place with a pool because of the hot sun, in Vegas. It also seems a bit for small we like to have more room that this listing has in it.

Slide 48

Slide 48 text

Certain amenities are more important, depending on the location and time Mosquito nets are important in Malaysia. Listings with air conditioning units are often a necessity in the summer. Is it worth learning these context-dependent preferences?

Slide 49

Slide 49 text

Similar Listings

Slide 50

Slide 50 text

“I would like to go to Las Vegas, Nevada. I would like to go with my girlfriend to see all the attractions of the city. I would also like to go hiking in the canyons nearby and visit Lake Mead.” Tell me about an upcoming trip you’d like to make.

Slide 51

Slide 51 text

Fantastic result. This listing appeals to me because of it's location close to the action of the city. The price is very appealing because it is cheaper than the nearby hotels. The main reason I want to stay here instead of a hotel is is because this has cooking facilities- because of my diet I have to prepare my own special meals. I also like that the place is not fancy or pretentious, it is very basic and fits my needs well. Find a place you’d love to stay at.

Slide 52

Slide 52 text

Bad result. The original search was for a whole home, so I don't like that I would be sharing with other people. I want my own private bathroom and space. I would be worried about making too much noise and disturbing roommates. I do like the location, though, and the room is quite nice. I would stay here as a last resort option. If this listing were unavailable, would you want to stay at Similar Listing #1 instead?

Slide 53

Slide 53 text

The original listing was for the Entire Home. Why is a Private Room being shown as a similar listing? Adding a filter for the same listing type should be pretty simple.

Slide 54

Slide 54 text

Pretty good result. I think the place looks nice and it is in a nice location. I do not like that it looks like someone is currently living there, with family photos on the wall and things like that. It would make me feel like i'm in someone elses space. The reviews are not very good, there are lots of complaints about smoke and noise and problems with the gate. If this listing were unavailable, would you want to stay at Similar Listing #2 instead?

Slide 55

Slide 55 text

No content

Slide 56

Slide 56 text

Horrible result. Again, I do not want to stay in a place with others, I want my own space. I also liked the original listing because it was close to everything, but this location is too far from the strip and downtown, and I would need to rent a car to get around. I don't like that the home is in a golf community, because I don't like being in areas with stuck up people. I need to be closer to the city and all the fun attractions of Vegas, not in a residential area. If this listing were unavailable, would you want to stay at Similar Listing #3 instead?

Slide 57

Slide 57 text

Why is a location so far away being suggested? Is distance from the original listing a feature?

Slide 58

Slide 58 text

from a different set of 100 raters.

Slide 59

Slide 59 text

No content

Slide 60

Slide 60 text

A great deal of Amazon purchases and YouTube watches come from Related books and videos. Would bookings similarly increase if Airbnb’s Similar Listings were boosted from their backwater status at the bottom of the page?

Slide 61

Slide 61 text

New Experiences

Slide 62

Slide 62 text

Neighborhood Discovery

Slide 63

Slide 63 text

Travelers are often unfamiliar with the neighborhoods of a city. Can Airbnb design an experience that helps them pick which to stay in?

Slide 64

Slide 64 text

While there's nothing at all wrong with this listing (and the price is fantastic), I'd just want to check that there's nothing better. Also, I'd want to research whether or not it's a safe and nice area. I might also decide to expand my search to include more expensive options. This way, I can weigh up whether the benefits of a more expensive option warrant the cost compared to a place like this.

Slide 65

Slide 65 text

It looks great, and I'd definitely want to contact the host so long as I do some research and remain as enthusiastic. The price is reasonable and the apartment looks great. The main thing I would want to check is the area it's in, Lisburn. I don't know how convenient that is compared to the rest of Belfast and the Titanic quarter. If it was convenient, I would probably go for this.

Slide 66

Slide 66 text

Helpful Reviews

Slide 67

Slide 67 text

Would showcasing helpful reviews increase bookability?

Slide 68

Slide 68 text

“This property is stunning! The pictures are beautiful and I like the proximity to Dublin. Reading the user reviews, it seems the hosts are gracious and warm, but don't get in your way unless you want them to. I like that. I could just sit in a lawn chair outside this building for the entire vacation and be content.” “It's a really well-decorated and lovely space. I also like that it's in Soho. The price is a little more than we'd like to spend ideally, but for the right place we'd make the sacrifice. Biggest worry I'd have is that this property only has two reviews. I'd be a little scared about committing more money than I would normally on a property with 2 reviews.” “Everything I said about the first rental applies to this one as well. The apartment is very modern and looks very clean (almost clinical). Though I always prefer design that is antique and warm in style, it is almost impossible to find and a clean, modern design is always nice as well. I can't think of much more to say about this apartment except that I notice it is missing half a star in the location category, so I would definitely have to read some of the reviews before I would consider booking it.”

Slide 69

Slide 69 text

Useless reviews

Slide 70

Slide 70 text

Helpful reviews

Slide 71

Slide 71 text

No content

Slide 72

Slide 72 text

No content

Slide 73

Slide 73 text

No content

Slide 74

Slide 74 text

Can we crowdsource training data for building a “helpful reviews” classifier? Review-level features (host friendliness, cleanliness, location, etc.) would also be useful in ranking.

Slide 75

Slide 75 text

Activity-Based Search

Slide 76

Slide 76 text

Let’s try to book a skiing trip on Airbnb.

Slide 77

Slide 77 text

“Within the next 2 months, I'd like to go to the northern part of my state (WI) to a skiing resort. I want to go for the weekend with my husband. I want to do this because we need a getaway and it would be romantic and we would still have plenty to do.”

Slide 78

Slide 78 text

Horrible result. The room is not ideal, it’s not romantic at all. The location is unappealing, it is near an urban area and there is no mention of skiing. I'm looking for a place near a skiing area, I don't have a specific location, because I don't know where they are, just that we have them. The price is okay, it's just not what I want at all.

Slide 79

Slide 79 text

Bad keyword matching Can we learn whether a keyword is about the host or the listing, and limit matches to the appropriate sections?

Slide 80

Slide 80 text

Horrible result. The room is pretty and quaint and the price is right. But again, it's just not what I'm looking for. This room is near downtown Madison, it's not a secluded place and there's no skiing.

Slide 81

Slide 81 text

Pretty good result. The house is beautiful and Lake Geneva is one of the nicest places in the state. I was looking to go up North is the only reason this isn't a good fit, this is in the southern part of the state. Otherwise it is beautiful and the price is great. It also lists skiing and other winter activities nearby. I would consider it if I couldn't find anything near where I want to go.

Slide 82

Slide 82 text

“I didn't find anything that really fit what I was looking for. I think that had more to do with the search than what is available. If I don't have a specific location in mind, like I wanted a Northern retreat near skiing or winter activities, it's hard to find places. I think being able to pick a general geographical area could be a good thing to add. I think having tags of things to do (skiing didn't really work well as a tag for me) or activities near the places might make it easier as well.”

Slide 83

Slide 83 text

It’s difficult to book a place when you know what you want to be doing, but not where you should be staying. How can Airbnb make this experience easier?

Slide 84

Slide 84 text

Traveler Types

Slide 85

Slide 85 text

Why do users travel, and with who? Twitter has news junkies and Beliebers. Dropbox has photo takers and collaborators. What are Airbnb’s clusters of travel types, and how can it design a better experience for them?

Slide 86

Slide 86 text

from a sample of 100 respondents on Google Consumer Surveys

Slide 87

Slide 87 text

No content

Slide 88

Slide 88 text

Hotels

Slide 89

Slide 89 text

Many users turn to Airbnb as a replacement for hotels. So when do users still prefer to book a hotel and why?

Slide 90

Slide 90 text

No content

Slide 91

Slide 91 text

65% of users mentioned a quality control problem with Airbnb

Slide 92

Slide 92 text

Users have had bad experiences with their hosts and listings “We've only ever used Airbnb once, and we had mixed feelings. The pricing was quite reasonable compared to a hotel (it was in Manhattan). On the other hand, the host was not very good. We walked into a very messy place with no clean sheets or towels, dirty dishes, and old food in the fridge.” “It can also be the case that the person's house is not very nice or they are unfriendly. This could be a detriment to the vacation experience. Indeed airbnb's fluctuating quality can be bad.”

Slide 93

Slide 93 text

They often don’t trust hosts and descriptions as a result “Well, it's not so much that it's bad about Airbnb, in and of itself, but there are a few unscrupulous types who misrepresent their properties, and you don't really have any easy time getting reparations, from what I've heard.” “Also, it can sometimes be difficult to trust some of the people listing their places to stay. Its not an insured hotel. Is the unit safe to stay in? Did they wash the sheets? is the room clean? Are they really the owners? They could be scamming you or try to rob you while you are there.”

Slide 94

Slide 94 text

Reviews and pictures can help, but they’re often not detailed enough (and pictures can be misleading too) “Airbnb also can be a crap shoot in terms of quality. Sometimes the physical place is not as good as it was advertised. Although this is usually reflected in the reviews of the place, sometimes they are not accurate or comprehensive enough.” “I wish it had more picture requirements. That way I could see what I was going to see what I got there more clearly. A clearer view of what I am going to be experiencing is it. Someway to make me feel more secure in a way.”

Slide 95

Slide 95 text

In contrast, hotels are a more trusted brand that provide a standard level of quality “Sometimes it's hard to know what to expect with Airbnb. You never really know for certain that it'll be a good experience. The people renting out might be rude or overly anal. It's something that you can't really be 100% sure about. Hotels are almost always guaranteed to have good service. If not you can complain (and it's not like they'll kick you out). With Airbnb there's a greater chance for awkward interactions.” “You're also relying on reviews of others to make a decision to book a stay or not. You can do the same with hotels as well, rely on reviews, but with hotels, there also typically comes a standard of service that you come to expect from certain established hotel names. With renters you can hope to expect what a renter lists, sometimes expectations may not be met and that's a risk you take. Booking a hotel is usually a guarantee.”

Slide 96

Slide 96 text

How could this improve? Amazon and Yelp allow user-submitted pictures. Should Airbnb? Can Airbnb highlight useful reviews? The personal, human interaction makes users more reluctant to leave negative feedback. Would anonymous reviews help?

Slide 97

Slide 97 text

45% of users found online booking on Airbnb a hassle

Slide 98

Slide 98 text

Hotels, without the overhead of dealing with a host, are just simpler and faster “Also a hotel can be better in the fact that it can be booked on a moments notice without prior contact. I sometimes just simply want to deal with the simplicity of using a hotel.” “While you can instantly make a booking on airbnb you have to contact the host beforehand, and make arrangements whereas hotels nowadays you're practically guaranteed/hassle free booking from the convenience of an app without much discussion needed between you and the hotel.” “It can be hit or miss, sometimes you really don't want to have to deal with the host. In a hotel you can just check in very fast, and not have to deal with the social aspect of it. I like to have a consistent experience as well, I just want to have a no fuss experience. The booking process is easy on the site, but you still have to discuss things with the host whereas with a hotel you can just show up and check in.”

Slide 99

Slide 99 text

It’s also difficult to book Airbnb at the last minute “I find a hotel more convenient. If you're traveling and don't book ahead you can probably find a hotel and book it easier than trying to find an airbnb listing near where you're traveling last minute.” “Airbnb does not offer a lot of options if you are looking for a short stay. It also doesn’t seem to be as useful if you’re looking for something last minute… I think there should be a section for last minute reservations. Also there should be an alternative available like an insurance just in case things are not as they were advertised. “

Slide 100

Slide 100 text

It’s especially annoying when you have to contact multiple hosts “I found it difficult to book a room on the specific date that I wanted. I was not able to book one place and had to look for another place that was available.” “My host canceled on me and I had to find another place to stay.”

Slide 101

Slide 101 text

Airbnb has an Instant Booking feature already, but many users didn’t even realize it (Some users thought the lightning bolt indicated a hot listing about to be booked, or a flash sale.) Can Airbnb market the feature better to users, and increase adoption among hosts?

Slide 102

Slide 102 text

30% of users found the forced human interaction a turnoff

Slide 103

Slide 103 text

Part of this stems from a worry of disturbing the owner “Also, in a hotel there is a sense you can get a little wild. You can order pizza and leave it out on a table, for example, without worrying I might bring bugs into this nice couple's house or make their guest room smell funny.” “I felt weird coming home late from a wedding and fearing that I was disturbing the owner. “

Slide 104

Slide 104 text

Meeting with the host can be difficult, and especially annoying after a long travel “I had to wait a few minutes to meet the landlord. It was cold out and didn't appreciate that. We couldn't reach the landlord, heard different stories, then we were told the key was under the doormat. So checkin was strange. ” “My flight arrived early and the host was at a party, so I felt awkward asking him to leave to meet me and give me the keys. I ended up just going to a movie theatre while I waited for him to come back, even though I wanted to sleep.”

Slide 105

Slide 105 text

Some people also want to be left alone, without awkward interaction “There was some lack of privacy. In a hotel, I feel more anonymous. I felt obligated to talk to the apartment owner a lot.” “I think possible intrusion on the host part as they do have access to the room, apartment, house. It can be messy, and not cleaning properly. Not having standards and someone to complain about the place. Perhaps communication problems with the owner. It frustrates me when I have to call, email, and spend time arranging a place to stay.” “Hotels are almost always guaranteed to have good service. If not you can complain (and it's not like they'll kick you out). With Airbnb there's a greater chance for awkward interactions.”

Slide 106

Slide 106 text

Companies like Uber grew in large part by removing the bothersome part of human interaction (calling a taxi), and making the rest (chatting with your driver) optional. Could Airbnb do the same?

Slide 107

Slide 107 text

Summary

Slide 108

Slide 108 text

Growth on Airbnb comes from two ways: 1. Making it easy for users to travel more. 2. Getting users to replace hotels or the accommodations they’d use otherwise.

Slide 109

Slide 109 text

To encourage users to travel more, Airbnb could make the entire travel process easier. Where should I vacation if I want to go skiing? What neighborhood should I stay in? What should I do once I’ve arrived at my listing?

Slide 110

Slide 110 text

I want to go skiing in Wisconsin. Where should I stay? This is impossible to figure out right now, and deters me from booking.

Slide 111

Slide 111 text

I’m going to Paris for the first time. What neighborhood should I look under if I want to stay somewhere artsy with great food? This may be a great listing, but is it close to the Eiffel Tower and great restaurants? Is the neighborhood even safe?

Slide 112

Slide 112 text

To replace existing accomodations, Airbnb needs to make a better online and offline booking experience.

Slide 113

Slide 113 text

I want to spend my trip having fun, not navigating directions. Can Airbnb make it easier to find listings close to the attractions I plan to visit?

Slide 114

Slide 114 text

Part of Airbnb’s appeal is its lower price. So why am I worrying about expensive security deposits that may cost more than my stay? Can features like these be incorporated into Airbnb’s discovery models?

Slide 115

Slide 115 text

I want a host and listing I can trust. Can Airbnb surface more helpful reviews?

Slide 116

Slide 116 text

What other problems does Airbnb need to fix?

Slide 117

Slide 117 text

What types of users does it need to fix them for?

Slide 118

Slide 118 text

What are the advantages of its competition?

Slide 119

Slide 119 text

Airbnb knows more about me than a hotel. It knows where I’ve searched. I’ve left reviews of what I liked. I’ve booked multiple listings on the platform. I’ve even connected my Facebook account Taking a trip shouldn’t be a stressful chore. Can Airbnb use all this information to give me the ultimate travel experience?

Slide 120

Slide 120 text

Homes are sweet. Let’s make travel even sweeter.

Slide 121

Slide 121 text

hello@echen.me