How HotelTonight Finds the Best Hotels in the Moment

Learn how HotelTonight uses hosted Elasticsearch to mine millions of documents and analyze diverse data in milliseconds – from hotel inventory systems to data about a user’s desired trip (date, duration, location, price, brand, ratings, etc) – in order to help their users find a hotel at a moment’s notice.

Transcript

1. ‹#› Paul Sorensen, Platform Engineer at HotelTonight Feb 17th, 2016
 @paulnsorensen Finding the Best Hotels in the Moment
2. allow me to introduce myself
3. About Me: Paul Sorensen Platform Engineer at HotelTonight I work on our hotel ranking algorithm I’m currently fascinated by scaling web apps twitter: @paulnsorensen
4. How HotelTonight works Hotels compete for limited display, we show you the best deals
5. How HotelTonight works Hotels compete for limited display, we show you the best deals Most API requests here
6. How HotelTonight works Hotels compete for limited display, we show you the best deals Most API requests here (audience downloads app here)
7. last minute travel
8. be spontaneous
9. time is precious
10. time is precious response
11. [enter elasticsearch]
12. hotel rate records (documents) 200x 10x 2.5x more API traffic reduction in response times Since Introducing Elasticsearch…
13. how?
14. Just use Elasticsearch!
15. Just use Elasticsearch! gem install elasticsearch
16. Just use Elasticsearch! gem install elasticsearch rake scale:hotels
17. Just use Elasticsearch! gem install elasticsearch rake scale:hotels
18. Just use Elasticsearch! gem install elasticsearch rake scale:hotels JUST KIDDING
19. there is no silver bullet
20. scaling is hard
21. scope
22. Overview Why Elasticsearch? How we use Elasticsearch (is this unique?) Some challenges we’ve faced (amidst success)
23. why elasticsearch?
24. we have a good problem
25. launched in 3 cities
26. now in more than 2000 cities
27. we’re never not booking
28. Early 2014: Our system was reaching its capacity MySQL O(n^2 log n) Ranking over hundreds of ActiveRecord objects gimme nearby hotels geolocation query
29. Early 2014: Our system was reaching its capacity MySQL O(n^2 log n) Ranking over hundreds of ActiveRecord objects gimme nearby hotels geolocation query geolocation query
30. Early 2014: Our system was reaching its capacity MySQL O(n^2 log n) Ranking over hundreds of ActiveRecord objects gimme nearby hotels geolocation query geolocation query geolocation query
31. Early 2014: Our system was reaching its capacity MySQL O(n^2 log n) Ranking over hundreds of ActiveRecord objects gimme nearby hotels geolocation query geolocation query geolocation query
32. we cannot have downtime
33. we’re never not booking
34. meanwhile
35. we still needed to grow
36. we used to be same-day only
37. We wanted to expand our booking window from 1 to 7 days Same-day 6 more days, 7x data Advance booking
38. scaling is hard
39. scaling is unique
40. What were our choices? • More Caching? • Use OpenGIS on MySQL (geospatial index extension)? • Switch to PostgreSQL and use PostGIS? • Find something from Hacker News?
41. What were our choices? • More Caching? • Use OpenGIS on MySQL (geospatial index extension)? • Switch to PostgreSQL and use PostGIS? • Find something from Hacker News? • Use Elasticsearch? The full-text indexing engine?
42. Elasticsearch Use cases Full-text search Analytics: Elastic Stack Spell-checking, Autocomplete
 
 Ranking hotel rooms?
43. we took a closer look
44. { “_id” : 4492, “description”: “The quick brown fox jumps over dogs” }, { “_id” : 4493, “description”: “The slow red fox doesn’t say things” } Simple Example: Documents
45. { “query” : { “match” : { “description” : “quick fox” } } } returns documents matching “quick” and/or “fox” Simple Example: Query
46. Simple Example: How is it stored? Inverted index: { “fox” => [4492, 4493], “brown” => [4492], “red” => [4493], }
47. Simple Example: How is it stored? Inverted index: { “fox” => [4492, 4493], “brown” => [4492], “red” => [4493], } THIS MAKES IT FAST
48. cool
49. but we don’t query by text
50. “give me the best hotel rooms for tomorrow night for a 2 night stay in NYC”
51. deeper dive: how do *we* use elasticsearch?
52. “give me the best hotel rooms for tomorrow night for a 2 night stay in NYC”
53. “give me the best hotel rooms for tomorrow night for a 2 night stay in NYC”
54. major : bool query
55. tomorrow night 2 night NYC These become bitset filters: “check_in_date” : “2016-02-18” “length_of_stay” : 2 “market_id” : 14326
56. tomorrow night 2 night NYC Well, most of them do “check_in_date” : “2016-02-18” “length_of_stay” : 2 “market_id” : 14326 40.7127,-74.0059 “geo_distance” : { // etc. }
57. Bitset filters: independent filter caching • queries cache individual filter matches* • very fast to check if a document matches • *but not geo, range or script filters
58. run bitset filters first then run geo
59. okay, but how do you rank documents?
60. “give me the best hotel rooms for tomorrow night for a 2 night stay in NYC”
61. Elasticsearch orders documents by relevance • Term frequency • Inverse document frequency • Field-length norm
62. but we don’t query by text
63. Elasticsearch orders documents by relevance • Term frequency • Inverse document frequency • Field-length norm must replace with our own scoring
64. major : function score query
65. How WE define relevance: • Pre-score document with sub-scores at index time • Use field_value_factor to apply weight to sub-scores at query time
66. outcome?
67. don’t load lots of ActiveRecord objects into memory
68. What were our problems again? MySQL O(n^2 log n) Ranking over hundreds of ActiveRecord objects gimme nearby hotels geolocation query geolocation query geolocation query
69. What were our problems again? MySQL O(n^2 log n) Ranking over hundreds of ActiveRecord objects gimme nearby hotels geolocation query geolocation query geolocation query SOLVED SOLVED
70. ✓less to geo-filter ✓less iteration ✓less memory GC
71. == Speed is money.
72. show me the money JSON
73. sample document
74. { "index": { "_id": "3752442343255", "_index": "rates_2016-02-19", "_type": "rate", "data": { "category_id": 3, "check_in_date": "2016-02-19", "price_in_usd_avg": "100.0", "hotel_id": 791323345, "location": "44.97,-93.35", "market_id": 9352, "price_score": "0.87", "review_score": 0.64, "sold_out": false, "stay_length": 1, }, } }
75. { "index": { "_id": "3752442343255", "_index": "rates_2016-02-19", "_type": "rate", "data": { "category_id": 3, "check_in_date": "2016-02-19", "price_in_usd_avg": "100.0", "hotel_id": 791323345, "location": "44.97,-93.35", "market_id": 9352, "price_score": "0.87", "review_score": 0.64, "sold_out": false, "stay_length": 1, }, } }
76. { "index": { "_id": "3752442343255", "_index": "rates_2016-02-19", "_type": "rate", "data": { "category_id": 3, "check_in_date": "2016-02-19", "price_in_usd_avg": "100.0", "hotel_id": 791323345, "location": "44.97,-93.35", "market_id": 9352, "price_score": "0.87", "review_score": 0.64, "sold_out": false, "stay_length": 1, }, } }
77. Indices are based on check-in date 2016-0 2-15 2016-0 2-16 2016-0 2-17 2016-0 2-18 2016-0 2-19 2016-0 2-20 2016-0 2-21 current day deleted
78. sample query (in parts)
79. { "filter": { "and": [ { "bool": { // most filters here } }, { “geo_bounding_box” : {
 "pin.location" : { "top_left" : [-74.1, 40.7], "bottom_right" : [-73.6, 40.0] } } } ] } } "must": [ { "term": { "start_date": "2016-02-18" } }, { "term": { "length_of_stay": 2 } }, { "term": { "on_sale": true } } ]
80. { "filter": { "and": [ { "bool": { // most filters here } }, { “geo_bounding_box” : {
 "pin.location" : { "top_left" : [-74.1, 40.7], "bottom_right" : [-73.6, 40.0] } } } ] } } "must": [ { "term": { "start_date": "2016-02-18" } }, { "term": { "length_of_stay": 2 } }, { "term": { "on_sale": true } } ]
81. { "filter": { "and": [ { "bool": { // most filters here } }, { “geo_bounding_box” : {
 "pin.location" : { "top_left" : [-74.1, 40.7], "bottom_right" : [-73.6, 40.0] } } } ] } } "must": [ { "term": { "start_date": "2016-02-18" } }, { "term": { "length_of_stay": 2 } }, { "term": { "on_sale": true } } ]
82. { "filter": { "and": [ { "bool": { // most filters here } }, { “geo_bounding_box” : {
 "pin.location" : { "top_left" : [-74.1, 40.7], "bottom_right" : [-73.6, 40.0] } } } ] } } "must": [ { "term": { "start_date": "2016-02-18" } }, { "term": { "length_of_stay": 2 } }, { "term": { "on_sale": true } } ]
83. { "filter": { "and": [ { "bool": { // most filters here } }, { “geo_bounding_box” : {
 "pin.location" : { "top_left" : [-74.1, 40.7], "bottom_right" : [-73.6, 40.0] } } } ] } } "must": [ { "term": { "start_date": "2016-02-18" } }, { "term": { "length_of_stay": 2 } }, { "term": { "on_sale": true } } ]
84. { "functions": [ { "field_value_factor": { "factor": 0.9, "field": "price_score" } }, { "field_value_factor": { "factor": 0.6, "field": “review_score” } } ] }
85. { "functions": [ { "field_value_factor": { "factor": 0.9, "field": "price_score" } }, { "field_value_factor": { "factor": 0.6, "field": “review_score” } } ] }
86. { "aggs":{ "top_rates":{ "terms":{ "field":"hotel_id", "size":50, "order":{ "top_hit":"desc" } }, "aggs":{ "top_rate":{ "top_hits":{ "size":1 } }, "top_hit":{ "max":{ "script":"_score" } } } } } }
87. { "aggs":{ "top_rates":{ "terms":{ "field":"hotel_id", "size":50, "order":{ "top_hit":"desc" } }, "aggs":{ "top_rate":{ "top_hits":{ "size":1 } }, "top_hit":{ "max":{ "script":"_score" } } } } } }
88. { "aggs":{ "top_rates":{ "terms":{ "field":"hotel_id", "size":50, "order":{ "top_hit":"desc" } }, "aggs":{ "top_rate":{ "top_hits":{ "size":1 } }, "top_hit":{ "max":{ "script":"_score" } } } } } }
89. { "aggs":{ "top_rates":{ "terms":{ "field":"hotel_id", "size":50, "order":{ "top_hit":"desc" } }, "aggs":{ "top_rate":{ "top_hits":{ "size":1 } }, "top_hit":{ "max":{ "script":"_score" } } } } } }
90. { "aggs":{ "top_rates":{ "terms":{ "field":"hotel_id", "size":50, "order":{ "top_hit":"desc" } }, "aggs":{ "top_rate":{ "top_hits":{ "size":1 } }, "top_hit":{ "max":{ "script":"_score" } } } } } }
91. { "aggs":{ "top_rates":{ "terms":{ "field":"hotel_id", "size":50, "order":{ "top_hit":"desc" } }, "aggs":{ "top_rate":{ "top_hits":{ "size":1 } }, "top_hit":{ "max":{ "script":"_score" } } } } } } give me one, best-scoring rate by hotel_id; order each hotel_id bucket by top scoring rate’s max score
92. { "aggs":{ "top_rates":{ "terms":{ "field":"hotel_id", "size":50, "order":{ "top_hit":"desc" } }, "aggs":{ "top_rate":{ "top_hits":{ "size":1 } }, "top_hit":{ "max":{ "script":"_score" } } } } } } give me one, best-scoring rate by hotel_id; order each hotel_id bucket by top scoring rate’s max score AKA: give me the best rate per hotel, and order the rates by score
93. { "aggs":{ "top_rates":{ "terms":{ "field":"hotel_id", "size":50, "order":{ "top_hit":"desc" } }, "aggs":{ "top_rate":{ "top_hits":{ "size":1 } }, "top_hit":{ "max":{ "script":"_score" } } } } } } Feature request: make this an agg? (or at least not a script)
94. We also have a post-processing framework • Some operations require seeing a first pass at the results • Example: variety scoring • Example: “cutting edges off” of bounding box
95. We also have a post-processing framework Elasticsearch generate query docs post- processors loop over docs response
96. architectural layout
97. Document ingestion Docs MySQL price updates $$ Elasticsearch denormalization
98. Querying results Elasticsearch MySQL generate response generate query Memcached
99. new challenges unlocked
100. Elasticsearch MySQL must be kept in sync
101. If Elasticsearch goes down, we go down Elasticsearch MySQL generate response generate query Memcached
102. we cannot have downtime
103. we cannot have will deal with inconsistency
104. Leveling up our architecture minimize consistency delays (sync lag) but defend against it when it does happen mapping changes / elasticsearch upgrades zero-downtime mapping changes blue/green deployments
105. and this worked really well
106. until one day…
108. scaling is hard
109. so now what do we do?
110. track everything
111. (screen of dash here)
112. what did we find?
113. Our writes were too spiky
114. Document ingestion Docs MySQL price updates $$ Elasticsearch denormalization Message Queue We added a message queue
115. filter cache is king
116. Filter cache eviction spikes == slow queries
117. we increased cluster size
118. so, what’s next?
119. more scaling
120. horizontal scaling?
121. scaling is hard
122. scaling is unique
123. hotel rate records (documents) 200x 10x 2.5x more API traffic reduction in response times Scaling is big numbers…
124. Scaling is money.
125. scaling is awesome
126. But most of all, scaling is fun. Thank you Twitter @paulnsorensen Email paul@hoteltonight.com Promo Code ($25 off first booking) PAUL