How REA Group Uses Elasticsearch to Power Real Estate Searches

REA Group’s realestate.com.au has been slowly migrating its search function to Elasticsearch. Its millions of listings, plus millions of sales histories and thousands of realtors and agencies, mean thousands of updates at once and more than 100 queries per second.

David Kemp | Elastic{ON} Tour Sydney | December 8, 2015

Transcript

1. How REA Group Uses Elasticsearch to Power Real Estate Searches David Kemp REA Group 1
3. Millions of listings 3
4. Tens of thousands of agencies 4
5. Hundreds of thousands of agents/salespeople 5
6. 6 Only dozens of updates per hour Spikes at thousands of updates at once
7. 7 More than one hundred queries per second
8. The old system 8 RDB Feeder FAST Search API
9. The new system (application specific ES clusters) 9 Elasticsearch For Rent Search API RDB Agent Feeder Elasticsearch Sold Listings Search API Sold Listings Feeder Elasticsearch Find An Agent API Rental Listings Feeder
10. 10 We use Amazon Web Services (AWS)
11. 11 No two AWS Availability Zones are in the same flood plane
12. Cluster across AWS Availability Zones 12 Availability Zone A Node Elasticsearch cluster Node Node Node Node Availability Zone B AWS Load Balancer Feeder API API AWS Load Balancer
13. Active/Active across two AWS Regions 13 Frankfurt Sydney AWS Load Balancer Node Elasticsearch cluster Node Node Node Node Node Elasticsearch cluster Node Node Node Node AWS Load Balancer Route-53 DNS European Clients Australian Clients DNS resolution based on latency API API API API
14. Active/Active across two AWS Regions 14 Frankfurt Sydney AWS Load Balancer Node Elasticsearch cluster Node Node Node Node Node Elasticsearch cluster Node Node Node Node AWS Load Balancer Route-53 DNS European Clients Australian Clients DNS resolution based on latency API API API API
15. Active/Active across two AWS Regions 15 Frankfurt   Sydney   AWS  Load   Balancer   Node   Elas8csearch   cluster   Node   Node   Node   Node   Node   Elas8csearch   cluster   Node   Node   Node   Node   AWS  Load   Balancer   Route-­‐53   DNS   European   Clients   Australian   Clients   DNS  resolu8on   based  on  latency   API   API   API   API  
16. Disposable Indexes 16
17. Disposable indexes 17 Search   API   Index  1.0   Feeder  1.0   Alias  1  
18. Disposable indexes 18 Search   API   Index  1.0   Feeder  1.0   Alias  1   Index  1.1   Feeder  1.1  
19. Disposable indexes 19 Search   API   Index  1.0   Feeder  1.0   Alias  1   Index  1.1   Feeder  1.1  
20. Disposable indexes 20 Search   API   Alias  1   Index  1.1   Feeder  1.1  
21. Disposable indexes 21 Search   API   Index  1.1   Feeder  1.1   Alias  1  
22. Disposable indexes 22 Search   API   Index  1.1   Feeder  1.1   Alias  1   Index  2.0   Feeder  2.0   Alias  2  
23. Disposable indexes 23 Index  1.1   Feeder  1.1   Alias  1   Index  2.0   Feeder  2.0   Alias  2   Search   API   Search   API  
24. Disposable indexes 24 Index  2.0   Feeder  2.0   Alias  2   Search   API  
25. Modeling Challenges 25
26. Standard Listings 26
27. Upgraded Listing 27
28. Multiple fixed duration upgrades per listing 28 Upgrade   Start:  03/04/15   End:    17/04/15   Lis8ng  
29. Map upgrades as “nested” objects 29 "lis8ngUpgrades":  {          "type":  "nested",          "rank":  {"type":  "integer"},          "startDate":  {"type":  "date"},          "endDate":  {"type":  "date"}   }    
30. Rank by upgrade 30  "query":  {              …              "nested":  {                            …                          "func)on_score":  {                                          …                                              "range":  {  "lis8ngUpgrades.startDate":  {  "lt":  "now"}},                                                "range":  {  "lis8ngUpgrades.endDate":  {"gte":  "now"}}                                                  …                                              "field_value_factor":  {  "field":  "lis8ngUpgrades.rank"}  
31. Suburb autosuggest 31
32. Suburb autosuggest 32
33. Suburb autosuggest •  Will “b” match “B” and “rd” match “road”? •  Will “Bunswik” match “Brunswick”? •  Will “Brunswick East” match “East Brunswick”? •  Will “Bruns” match “East Brunswick”? •  Exact matches should outrank fuzzy matches. •  Highlighting? E.g. “Bru” => “East <em>Bru</em>nswick”. •  Has to be fast. 33
34. The many ways of implementing autosuggest •  Elasticsearch’s completion suggesters •  Prefix query •  Match phrase prefix •  Edge n-grams 34
35. 35 Thank  you  for  listening   @david_b_kemp