Upgrade to Pro — share decks privately, control downloads, hide ads and more …

How REA Group Uses Elasticsearch to Power Real Estate Searches

Dd9d954997353b37b4c2684f478192d3?s=47 Elastic Co
December 08, 2015

How REA Group Uses Elasticsearch to Power Real Estate Searches

REA Group’s realestate.com.au has been slowly migrating its search function to Elasticsearch. Its millions of listings, plus millions of sales histories and thousands of realtors and agencies, mean thousands of updates at once and more than 100 queries per second.

David Kemp | Elastic{ON} Tour Sydney | December 8, 2015

Dd9d954997353b37b4c2684f478192d3?s=128

Elastic Co

December 08, 2015
Tweet

More Decks by Elastic Co

Other Decks in Technology

Transcript

  1. How REA Group Uses Elasticsearch to Power Real Estate Searches

    David Kemp REA Group 1
  2. 2

  3. Millions of listings 3

  4. Tens of thousands of agencies 4

  5. Hundreds of thousands of agents/salespeople 5

  6. 6 Only dozens of updates per hour Spikes at thousands

    of updates at once
  7. 7 More than one hundred queries per second

  8. The old system 8 RDB Feeder FAST Search API

  9. The new system (application specific ES clusters) 9 Elasticsearch For

    Rent Search API RDB Agent Feeder Elasticsearch Sold Listings Search API Sold Listings Feeder Elasticsearch Find An Agent API Rental Listings Feeder
  10. 10 We use Amazon Web Services (AWS)

  11. 11 No two AWS Availability Zones are in the same

    flood plane
  12. Cluster across AWS Availability Zones 12 Availability Zone A Node

    Elasticsearch cluster Node Node Node Node Availability Zone B AWS Load Balancer Feeder API API AWS Load Balancer
  13. Active/Active across two AWS Regions 13 Frankfurt Sydney AWS Load

    Balancer Node Elasticsearch cluster Node Node Node Node Node Elasticsearch cluster Node Node Node Node AWS Load Balancer Route-53 DNS European Clients Australian Clients DNS resolution based on latency API API API API
  14. Active/Active across two AWS Regions 14 Frankfurt Sydney AWS Load

    Balancer Node Elasticsearch cluster Node Node Node Node Node Elasticsearch cluster Node Node Node Node AWS Load Balancer Route-53 DNS European Clients Australian Clients DNS resolution based on latency API API API API
  15. Active/Active across two AWS Regions 15 Frankfurt   Sydney  

    AWS  Load   Balancer   Node   Elas8csearch   cluster   Node   Node   Node   Node   Node   Elas8csearch   cluster   Node   Node   Node   Node   AWS  Load   Balancer   Route-­‐53   DNS   European   Clients   Australian   Clients   DNS  resolu8on   based  on  latency   API   API   API   API  
  16. Disposable Indexes 16

  17. Disposable indexes 17 Search   API   Index  1.0  

    Feeder  1.0   Alias  1  
  18. Disposable indexes 18 Search   API   Index  1.0  

    Feeder  1.0   Alias  1   Index  1.1   Feeder  1.1  
  19. Disposable indexes 19 Search   API   Index  1.0  

    Feeder  1.0   Alias  1   Index  1.1   Feeder  1.1  
  20. Disposable indexes 20 Search   API   Alias  1  

    Index  1.1   Feeder  1.1  
  21. Disposable indexes 21 Search   API   Index  1.1  

    Feeder  1.1   Alias  1  
  22. Disposable indexes 22 Search   API   Index  1.1  

    Feeder  1.1   Alias  1   Index  2.0   Feeder  2.0   Alias  2  
  23. Disposable indexes 23 Index  1.1   Feeder  1.1   Alias

     1   Index  2.0   Feeder  2.0   Alias  2   Search   API   Search   API  
  24. Disposable indexes 24 Index  2.0   Feeder  2.0   Alias

     2   Search   API  
  25. Modeling Challenges 25

  26. Standard Listings 26

  27. Upgraded Listing 27

  28. Multiple fixed duration upgrades per listing 28 Upgrade   Start:

     03/04/15   End:    17/04/15   Lis8ng  
  29. Map upgrades as “nested” objects 29 "lis8ngUpgrades":  {    

         "type":  "nested",          "rank":  {"type":  "integer"},          "startDate":  {"type":  "date"},          "endDate":  {"type":  "date"}   }    
  30. Rank by upgrade 30  "query":  {        

         …              "nested":  {                            …                          "func)on_score":  {                                          …                                              "range":  {  "lis8ngUpgrades.startDate":  {  "lt":  "now"}},                                                "range":  {  "lis8ngUpgrades.endDate":  {"gte":  "now"}}                                                  …                                              "field_value_factor":  {  "field":  "lis8ngUpgrades.rank"}  
  31. Suburb autosuggest 31

  32. Suburb autosuggest 32

  33. Suburb autosuggest •  Will “b” match “B” and “rd” match

    “road”? •  Will “Bunswik” match “Brunswick”? •  Will “Brunswick East” match “East Brunswick”? •  Will “Bruns” match “East Brunswick”? •  Exact matches should outrank fuzzy matches. •  Highlighting? E.g. “Bru” => “East <em>Bru</em>nswick”. •  Has to be fast. 33
  34. The many ways of implementing autosuggest •  Elasticsearch’s completion suggesters

    •  Prefix query •  Match phrase prefix •  Edge n-grams 34
  35. 35 Thank  you  for  listening   @david_b_kemp