Engine Scoring and Predictive Analytics to Boost Search Accuracy

With Big Data, it is possible to harvest user event data, such as search and click logs, for the purpose of computing user-based search accuracy metrics. Learn about the algorithms and processes for computing these metrics which are vital for comparing search engine accuracy before a deployment.

Transcript

1. 1   Boos$ng  Search  Accuracy  with  Engine  Scoring   and  Predic$ve  Analy$cs     Paul  Nelson     Chief  Architect,  Search  Technologies     February  2016  
2. 2   There  will  be  a  demo   (so  don’t  go  away)  
3. 3   195+  Consultants  Worldwide   San  Diego   London,  UK   San  Jose,  CR   CincinnaH   Prague,  CZ   Washington   (HQ)   Frankfurt,  DE   • Founded  2005   • Deep  search  exper$se   • 700+  customers  worldwide   • Consistent  profitability   • Search  engines  &  Big  Data   • Vendor  independent   Manila,  PH  
4. 4   Typical  Conversa$on  with  Customer   Our  search   accuracy   is  bad   How  bad?   Really,   really,   bad.   Uh…  on  a   scale  of     1  to  10,   how  bad?   An  eight.   No  wait…   a  nine.   Maybe  even   a  9.5.   Let’s  call  it   a  9.23  
5. 5   Current  methods  are  woefully  inadequate   • Golden  Query  Set   o  Key  Documents   • Top  100  /  Top  1000  Queries  Analysis   • Zero  result  queries   • Abandonment  rate   • Queries  with  click   • Conversion  
6. 6   What  are  we  trying  to  achieve?   •  Reliable  metrics  for  search  accuracy   •  Can  run  analysis  off-­‐line   o  Does  not  require  producHon  deployment  (!)   •  Can  accurately  compare  two  engines   •  Runs  quickly  =  agility  =  high  quality   •  Can  handle  different  user  types  /  personaliza$on   o  Broad  coverage   •  Provides  lots  of  data  to  analyze  what’s  going  on   o  Data  to  decide  how  best  to  improve  the  engine  
7. 7   Search  Engine   Under  EvaluaHon   Search  Engine   Under  EvaluaHon   Search  Engine   Under  EvaluaHon   Leverage  logs  for  accuracy  tesHng   Query  Logs   Click  Logs   Engine   Scoring   Framework   • Engine  Score(s)   • Other  metrics  &  histograms   • Scoring  database   Search  Engine   Under  EvaluaHon  
8. 8   From  Queries  à  Users   • User  by  User  Metrics   o  Change  in  focus   • Group  ac$vity  by  session  and/or  user   o  Call  this  an  “AcHvity  Set”   o  Merge  sessions  and  users   • Use  Big  Data  to  analyze  all  users   o  There  are  no  stupid  queries  and  no  stupid  users   o  Overall  performance  based  on  the  experience  of  the  users   Queries   Other   Ac$vity   Clicks   Clusters  
9. 9   Engine  Score   •  Group  ac$vity  by  session  and/or  user  (Queries  &  Clicks)   •  Determine  “relevant”  documents   o  What  did  the  user  view?  Add  to  cart?  Purchase?   o  Did  the  search  engine  return  what  the  user  ulHmately  wanted?   •  Determine  engine  score  per  query  per  user:   o  Σ  power(FACTOR,  posiHon)*isRelevant[user,  searchResult[Q,posiHon].DocID]   -­‐  Evaluated  for  each  user’s  point  of  view   (Note:    many  other  formulae  possible,  MRR,  MAP,  DCG,  etc.)   •  Average  score  for  all  user  queries  =  user  score   •  Average  scores  across  all  users  =  final  engine  score  
10. 10   The  FACTOR  (K)  
11. 11   Off-­‐Line  Engine  Analysis   o  Can  we  re-­‐compute  this  array  for  all  queries?   o  ANSWER:    Yes!   Σ  power(FACTOR,  posiJon)*isRelevant[user,  searchResult[Q,posi5on].DocID]   Offline  Re-­‐Query   Search  Engine      Query  Logs                  New              Results   Big  Data  Array   Search  Engine   (possibly  embedded)  
12. 12   Con$nuous  Improvement  Cycle   Modify   Engine   Execute   Queries   Compute   Engine  Score   Evaluate   Results          Log        Files   Search  Engine   Search Score  Per  Engine  Version  
13. 13   Watch  the  Score  Improve  Over  Time  
14. 14   PredicHve  AnalyHcs     What  else  can  we  do  with  Engine  Scoring?    
15. 15   The  Brutal  Truth  about  Search  Engine  Scores   •  Random  ad-­‐hoc  formulae  put  together   o  Using  data  which  happens  to  be  lying  around  the  search  engine  index   •  TF  /  IDF  &  BM25  à  All  kinds  of  inappropriate  biases   o  Bias  towards  document  size  (smaller  /  larger)   o  Bias  towards  rare  (misspelled?  archaic?)  words   o  Not  scalable  (different  scores  on  different  shards)   •  Same  formula  since  the  1970’s  &  80’s   They  are  not  based  on  science.   We  can  do  beeer!  
16. 16    Big  Data  Cluster   We  use  Big  Data  to  Predict  Relevancy   Search  Engine          Content          Sources   Search Project   Docs   Web  Site   Pages   Support   Pages   Landing   Pages   Content   Copy   Search  Click  Logs   Click  Logs   Query  Logs   Financial   Data   Business   Data   Query  Logs   Op   Relevancy   Model   Connectors Index    Search      Index   Content Processing
17. 17   Probability  Scoring  /  Predic$ve  Relevancy   clicked?   purchased?   0   0   1   1   1   0   0   0   1   0   1   1   PredicHve  AnalyHcs   StaHsHcal  Model   to  Predict  Probability   Document   Signals   Query   Signals   User   Signals   Comparison   Signals  
18. 18   The  Power  of  the  Probability  Score   •  The  score  predicts  probability  of  relevancy   •  Value  is  0  à  1   o  Can  be  used  for  threshold  processing   o  All  documents  too  weak?  Try  something  else!   o  Can  combine  results  from  different  sources  /  construcHons  together   •  Iden$fies  what’s  important   o  Machine  learning  opHmizes  for  parameters   -­‐  IdenJfies  the  impact  and  contribuJon  of  every  parameter   o  If  a  parameter  does  not  improve  relevancy  à  REMOVE  IT   o  Scoring  becomes  objecHve,  not  subjecHve  (now  based  on  SCIENCE)   o  Allows  for  experimentaHon  on  parameters  
19. 19   Come  out  of  the  darkness  
20. And  into  the  light!    
21. 21   The  Age  of  Enlightenment   for  search  engine  accuracy   is  upon  us!  
22. 22   Demo  
23. 23   Thank  you!     Paul  Nelson     Chief  Architect,  Search  Technologies     pnelson@searchtechnologies.com   @PaulLovesSearch