Share
Embed
Engine Scoring and Predictive Analytics to Boost Search Accuracy
With Big Data, it is possible to harvest user event data, such as search and click logs, for the purpose of computing user-based search accuracy metrics. Learn about the algorithms and processes for computing these metrics which are vital for comparing search engine accuracy before a deployment.
Elastic Co
February 18, 2016
Transcript
23 slides
-
1 Boos$ng Search Accuracy with Engine Scoring and
Predic$ve Analy$cs Paul Nelson Chief Architect, Search Technologies February 2016 -
2 There will be a demo (so don’t
go away) -
3 195+ Consultants Worldwide San Diego London,
UK San Jose, CR CincinnaH Prague, CZ Washington (HQ) Frankfurt, DE • Founded 2005 • Deep search exper$se • 700+ customers worldwide • Consistent profitability • Search engines & Big Data • Vendor independent Manila, PH -
4 Typical Conversa$on with Customer Our search
accuracy is bad How bad? Really, really, bad. Uh… on a scale of 1 to 10, how bad? An eight. No wait… a nine. Maybe even a 9.5. Let’s call it a 9.23 -
5 Current methods are woefully inadequate • Golden Query
Set o Key Documents • Top 100 / Top 1000 Queries Analysis • Zero result queries • Abandonment rate • Queries with click • Conversion -
6 What are we trying to achieve? •
Reliable metrics for search accuracy • Can run analysis off-‐line o Does not require producHon deployment (!) • Can accurately compare two engines • Runs quickly = agility = high quality • Can handle different user types / personaliza$on o Broad coverage • Provides lots of data to analyze what’s going on o Data to decide how best to improve the engine -
7 Search Engine Under EvaluaHon Search Engine
Under EvaluaHon Search Engine Under EvaluaHon Leverage logs for accuracy tesHng Query Logs Click Logs Engine Scoring Framework • Engine Score(s) • Other metrics & histograms • Scoring database Search Engine Under EvaluaHon -
8 From Queries à Users • User by User
Metrics o Change in focus • Group ac$vity by session and/or user o Call this an “AcHvity Set” o Merge sessions and users • Use Big Data to analyze all users o There are no stupid queries and no stupid users o Overall performance based on the experience of the users Queries Other Ac$vity Clicks Clusters -
9 Engine Score • Group ac$vity by session
and/or user (Queries & Clicks) • Determine “relevant” documents o What did the user view? Add to cart? Purchase? o Did the search engine return what the user ulHmately wanted? • Determine engine score per query per user: o Σ power(FACTOR, posiHon)*isRelevant[user, searchResult[Q,posiHon].DocID] -‐ Evaluated for each user’s point of view (Note: many other formulae possible, MRR, MAP, DCG, etc.) • Average score for all user queries = user score • Average scores across all users = final engine score -
11 Off-‐Line Engine Analysis o Can we re-‐compute
this array for all queries? o ANSWER: Yes! Σ power(FACTOR, posiJon)*isRelevant[user, searchResult[Q,posi5on].DocID] Offline Re-‐Query Search Engine Query Logs New Results Big Data Array Search Engine (possibly embedded) -
12 Con$nuous Improvement Cycle Modify Engine
Execute Queries Compute Engine Score Evaluate Results Log Files Search Engine Search Score Per Engine Version -
14 PredicHve AnalyHcs What else can we
do with Engine Scoring? -
15 The Brutal Truth about Search Engine Scores
• Random ad-‐hoc formulae put together o Using data which happens to be lying around the search engine index • TF / IDF & BM25 à All kinds of inappropriate biases o Bias towards document size (smaller / larger) o Bias towards rare (misspelled? archaic?) words o Not scalable (different scores on different shards) • Same formula since the 1970’s & 80’s They are not based on science. We can do beeer! -
16 Big Data Cluster We use Big Data
to Predict Relevancy Search Engine Content Sources Search Project Docs Web Site Pages Support Pages Landing Pages Content Copy Search Click Logs Click Logs Query Logs Financial Data Business Data Query Logs Op Relevancy Model Connectors Index Search Index Content Processing -
17 Probability Scoring / Predic$ve Relevancy clicked?
purchased? 0 0 1 1 1 0 0 0 1 0 1 1 PredicHve AnalyHcs StaHsHcal Model to Predict Probability Document Signals Query Signals User Signals Comparison Signals -
18 The Power of the Probability Score •
The score predicts probability of relevancy • Value is 0 à 1 o Can be used for threshold processing o All documents too weak? Try something else! o Can combine results from different sources / construcHons together • Iden$fies what’s important o Machine learning opHmizes for parameters -‐ IdenJfies the impact and contribuJon of every parameter o If a parameter does not improve relevancy à REMOVE IT o Scoring becomes objecHve, not subjecHve (now based on SCIENCE) o Allows for experimentaHon on parameters -
21 The Age of Enlightenment for search engine
accuracy is upon us! -
23 Thank you! Paul Nelson
Chief Architect, Search Technologies [email protected] @PaulLovesSearch