Beyond Text Similarity - Tune your search for your Business Domain

A good search is more than just Lucene scoring, this talk from the Search Meetup Munich is about how one could get to better search results and how to secure them against regressions

Transcript

1. Beyond Text Similarity_ Tune your search for your Business Domain Search Meetup Munich 26.10.2016 Christian Uhl
2. Agenda Moving from simple text matching towards custom scoring •  Recap: Text similarity and why this stops working in the travel domain •  Using recommendations and user interaction feedback •  Performance! •  Protect yourself against regressions 2
3. Practical scoring and text similarity is not enough this time 3
4. Text Similarity 4 Elasticsearch •  Lucene Practical Scoring   score(q,d)    =                              queryNorm(q)                          ·∙  coord(q,d)                              ·∙  ∑  (                                                        tf(t  in  d)                                    ·∙  idf(t)²                                          ·∙  t.getBoost()                                ·∙  norm(t,d)                                  )  (t  in  q)    
5. Text Similarity 5 Elasticsearch •  Lucene Practical Scoring   score(q,d)    =                              queryNorm(q)                          ·∙  coord(q,d)                              ·∙  ∑  (                                                        tf(t  in  d)                                    ·∙  idf(t)²                                          ·∙  t.getBoost()                                ·∙  norm(t,d)                                  )  (t  in  q)    
6. Text Similarity 6 Elasticsearch •  Lucene Practical Scoring   score(q,d)    =                              queryNorm(q)                          ·∙  coord(q,d)                              ·∙  ∑  (                                                        tf(t  in  d)                                    ·∙  idf(t)²                                          ·∙  t.getBoost()                                ·∙  norm(t,d)                                  )  (t  in  q)    
7. Text Similarity 7 Elasticsearch •  Lucene Practical Scoring   score(q,d)    =                              queryNorm(q)                          ·∙  coord(q,d)                              ·∙  ∑  (                                                        tf(t  in  d)                                    ·∙  idf(t)²                                          ·∙  t.getBoost()                                ·∙  norm(t,d)                                  )  (t  in  q)    
8. Text Similarity 8 Elasticsearch •  Lucene Practical Scoring   score(q,d)    =                              queryNorm(q)                          ·∙  coord(q,d)                              ·∙  ∑  (                                                        tf(t  in  d)                                    ·∙  idf(t)²                                          ·∙  t.getBoost()                                ·∙  norm(t,d)                                  )  (t  in  q)    
9. Text Similarity 9 Inverse Document Frequency   Search  for  “Da  Vinci  Paris”     Few  Da  Vincis  in  the  World,  but  Paris  occurs  a  lot.     But  is  a  “Da  Vinci”  in  Valencia  more  relevant  than  any  other   Hotel  in  Paris?        
10. Text Similarity 10 Elasticsearch •  Lucene Practical Scoring   score(q,d)    =                              queryNorm(q)                          ·∙  coord(q,d)                              ·∙  ∑  (                                                        tf(t  in  d)                                    ·∙  idf(t)²                                          ·∙  t.getBoost()                                ·∙  norm(t,d)                                  )  (t  in  q)    
11. Text Similarity 11 Elasticsearch •  Lucene Practical Scoring   score(q,d)    =                              queryNorm(q)                          ·∙  coord(q,d)                              ·∙  ∑  (                                                        tf(t  in  d)                                    ·∙  idf(t)²                                          ·∙  t.getBoost()                                ·∙  norm(t,d)                                  )  (t  in  q)    
12. Text Similarity 12 Text Similarity Search  for  “Paris”     •  Paris,  Illinois   •  Paris,  Texas   •  Paris,  France   •  even  more  WTF       I don‘t visit no cheese eating surrender monkeys – Uncle Sam
13. Summary 13 Lucene Practical Scoring is finely crafted and tuned to find occurrences in large text bodys     We  do  not  have  large  text  bodys     Well, s***
14. Bring in the real world 14
15. Bring in the real World 15 Change the score! •  Instead of just relying on the practical scoring function, add other parameters •  Use values from the real world that reflect the relevance of a given document in the whole document space
16. Bring in the real World 16 Our users were kind enough to provide valuable feedback about our data •  They rate and recommend things (Hotels) •  They click on things (Everywhere*) *except ads We also have a geospatial relation between hotels and destinations
17. Bring in the real World 17 Rescore! •  Hotels by recommendations •  Destinations by clicks and hotel count •  POIs by clicks
18. Bring in the real World 18 Dont sort by Average rating!
19. Bring in the real World 19 Dont sort by average rating! http://www.evanmiller.org/how-not-to-sort-by-average-rating.html •  Score = (Positive ratings) - (Negative ratings) A(200,100) B(100,1) A > B ??
20. Bring in the real World 20 Dont sort by average rating! http://www.evanmiller.org/how-not-to-sort-by-average-rating.html •  Score = Average rating = (Positive ratings) / (Total ratings) A(200,100) = 0,66 B(3,1) = 0,75 B > A ??
21. Bring in the real World 21 Dont sort by average rating! http://www.evanmiller.org/how-not-to-sort-by-average-rating.html Maybe use the lower bound of Wilson score confindence interval for a Bernoulli Parameter!
22. Bring in the real World 22 Dont fear the math
23. Doesn’t custom scoring kill performance? 23
24. Bring in the real World 24 Yes it does. We started with script score function to determine a better score during search time. Very bad idea 500ms – 1s queries, CPUs screaming for mercy
25. Bring in the real World 25 Rescoring! •  Generate a search result with ES/ Lucene Standard •  Rescore the top 40 •  Fetch the top n of that
26. Bring in the real World 26
27. Protect yourself against regressions 27
28. Testing 28 Regression Testing •  Record ~4500 searches users did that brought in money •  Generate tests that make sure for each search term the relevant result is in the result set •  Define a threshold for OK (qalitative tests) •  Execute on CI!
29. Testing 29
30. Testing 30
31. Testing 31
32. 32 “Unless you‘re a Library you should use additional real word data for scoring” -me
33. #done 33 Get in touch! •  Christian.uhl@codecentric.de •  @chrisuhlcc