Humans vs AI Quality Raters for Search Engines

Transcript

1. Humans vs LLMs As Quality Raters for Search Engines Are major changes coming? Dawn Anderson - March 2024
2. Dawn Anderson • UK based SEO consultant • 17 years in SEO • Occasional SEO conference speaker • EU, UK, US, Global Search Awards judge • Previous digital marketing lecturer & trainer • Industry publication contributor • Now predominantly consulting all of the time Stalker of information retrieval threads and IR conference hashtags since 2017
3. A sea-change is coming for a fundamental part of search On the other side of the ‘front door’
4. The important algorithmic ranking evaluation stage If importance thresholds reached Indexing Discovery & refresh Crawling Dynamic build at runtime Serving In response to a query Ranking (& Re-ranking)
5. The process of search results evaluation (Ranking System) Determine how well a ‘system’ (ranking system) fares either currently (continuous evaluation), or when compared to proposed changes
6. Why relevance evaluation?
7. Whilst some change is temporally predictable Seasonality, temporality, data-driven probability of ‘precision-to-relevance’ intent shift
8. Often, what ‘relevance’ means is changing unpredictably Search CANNOT be static in a changing world
9. The process of evaluation… • Is both continuous - On existing ‘systems’ • And intermittent - On proposed ‘system improvements’
10. A ‘system’ is simply a recipe or multiple recipes SYSTEM == ALGORITHMIC BLEND
11. ‘System’ evaluation… is simply taste-testing Did the recipe turn out as we hoped???
12. Before… roll out
13. AKA Google Algorithm Updates Jagger Big D addy Florida Fritz Everflux Austin Bourbon PageRank Dewey Vince Caffeine Exact-match Penguin Bert RankBrain Pigeon Panda Fred
14. ‘Human in the loop (HITL)’ is the mainstay of ‘system’ evaluation
15. Predominantly two types of ‘HITL’ evaluation • Implicit • Explicit
16. Implicit (‘Human’ in the loop has no awareness) evaluation • Tests on real searcher segments • Anonymous scroll and click behaviour • UX testing on any site (heatmaps / recordings all fall into this category)
17. Explicit (human knows they are actively evaluating) evaluation • E.g. Searchers asked to provide feedback • Netflix users asked to thumbs up a film • Spotify favouriting or playlist building - leads to further recommendations • User groups / user panels • Sites asking for feedback • Professional expert relevance annotators • Paid human contractor evaluators
18. But it all mostly comes down to labels & labelling anyways IMPORTANT… Labels are training data for machine learning
19. Labels are all around us In vast numbers they are converted into mathematical form for machine learning training data
20. We are ALL data labellers… every single day Every day
21. A cohort of similar data labellers help with recommender systems Birds of a feather flock together… they like the same things
22. Data labels teach machines to know the difference between cats and dogs (reinforcement learning) Cat, dog, dog, cat, cat, dog, cat, dog, dog, dog, cat
23. Search engines have used ‘The Crowd’ for HITL evaluation for more than two decades
24. In search… ‘The Crowd’ ‘labels’ sample comparative search result sets ‘Relevant’ or ‘not relevant’
25. Pair-wise SERP results side-by-side comparisons are in the majority of relevance evaluation exercises PAIR-WISE COMPARISON OR
26. Instead of “yum” or “barf” labels, it’s “relevant” or “not-relevant” labels
27. But it’s mostly aggregated binary data Binary labels rolled up into overall relevance scores
28. Effectively a measurement of NDCG (Normalised Discounted Cumulative Gain) and / or DCG (Discounted Cumulative Gain)
30. The ‘recipe’ ingredients or quantities are then adjusted accordingly And the cycle begins again
31. Until… Acceptable Precision at K is achieved (P@k) E.g. The top 10 (k) or 20 (k) (whatever) in enough samples are deemed relevant
33. And… roll it out
34. But… Not all labels are created equally • Gold labels - High quality / lower availability • Silver labels - Lower quality / higher availability
35. Real search engine users in experiments Create ‘gold labels’
36. Before ‘The Crowd’… professional subject matter expert annotators Created gold labels too
37. But… Scale…
38. ‘The Crowd’ came for the scale Professional expert annotators were not scalable
39. Universal Human Relevance System
40. 16,000+ Google human quality raters alone
42. But to the detriment of quality “Such annotation tasks were delegated to crowd workers, with a substantial decrease in terms of quality of the annotation, compensated by a huge increase in annotated data.” (Clarke et al, 2022)
43. ‘The Crowd’ likely produce silver labels
44. There are issues with the data labelling industry overall
45. Data labelling industry crisis… Demand outstrips supply • There is a bottleneck (and it’s going to get worse) • Not enough labels produced to deal with the size of machine learning modes
46. “The global data collection and labeling market size was valued at $2.22 billion in 2022 and it is expected to expand at a compound annual growth rate of 28.9% from 2023 to 2030, with the market then expected to be worth $13.7 billion.” Source: Grand View Research, 2021
47. Data labellers work across many industries, many companies • Maps • Assistant • AI content detection • Search quality evaluation • Image detection labelling • AI content detection training • Any other ML driven application
48. High risk of under-trained ML models due to scaling without label volume increase
49. Deepmind researchers - “We find current large language models are significantly undertrained, a consequence of the recent focus on scaling language models while keeping the amount of training data constant. …we find for compute-optimal training …for every doubling of model size the number of training tokens should also be doubled.” – “Training Compute-Optimal Large Language Models” (Hoffman et al, 2022)
50. There’s a dark side to the data labelling industry too
51. Low paid ‘ghost workers’ in emerging economies
52. Pay protests at Google
53. There’s also a problem with humans as relevance labellers too We ain’t all that… it seems
54. In addition to the very subjective 170 page ‘Quality Rater Guidelines’
55. Notorious subjectivity of ‘relevance’
56. ‘The crowd is made of people - Observations from large scale crowd labelling’ (Thomas et al, 2022)
57. Bing Researchers - ‘The Crowd is Made of People: Observations from large scale crowd-labelling’ (Thomas et al, 2022) Findings: • Fatigue • Time of day & day of week • Anchoring • Task-switching • Left-side bias • General disagreement on relevance
58. Human challenges Ethical concerns Undertrained models Bottlenecks / over-demand More scale needed A perfect storm
59. Helloooooo… ChatGPT & Bing
60. ‘Large language models can accurately predict searcher preferences’ (Thomas et al, 2023) Bing’s LLM & GPT4 research
61. • GPT4 prompt engineering (role playing prompt) • (Up to 5) LLM agents to emulate the behaviour of search relevance evaluators • Produce enough gold and silver labels to build relevance training data for much larger data sets • Train the agents initially on gold labels ‘Large language models can accurately predict searcher preferences’ (Thomas et al, 2023)
62. Bing’s research and implementation Has caused quite a stir in the information retrieval community
63. “To measure agreement with real searchers needs high-quality “gold” labels, but with these we find that models produce better labels than third-party workers, for a fraction of the cost, and these labels let us train notably better rankers.” (Thomas et al, 2023)
64. Bing’s LLM evaluators - “A fraction of the cost and better rankers” (Thomas et al, 2023)
65. Bing’s LLM evaluators are monitored Using several methods
66. A spectrum of LLM & human rater collaborative approach ? ‘Frontiers of Information Access Experimentation for Research and Education’ (Clarke et al, 2022)
67. Will human quality raters go?
68. “It is yet to be understood what the risks associated with such technology are: it is likely that in the next few years, we will assist in a substantial increase in the usage of LLMs to replace human annotators.” (Clarke et al, 2022)
69. But…Concerns about reduced quality in exchange for scale “It is a concern that machine-annotated assessments might degrade the quality, while dramatically increasing the number of annotations available.” Clarke et al, 2022
70. Will Google follow suit?
71. Bing test and release updates seamlessly and quickly Potential for Google to go this way with better evaluation pipelines
72. Google cancels their contract with Appen
73. Some surmised a switch to AI evaluators was part of the reason
74. Algorithms - bigger, broader, multi-modal / multi-aspected Aspected algorithms quickly go into core or simultaneous • Product reviews • Helpful content classifier • Panda historically • Spam updates
75. Machine learning classifiers Google is learning quickly: • what ‘unhelpful content looks like’ • What AI generated content looks like • What paid links look like
76. AI content detection is where it’s at next For the data labelling industry
77. Fun times ahead Enjoy!!!
78. Thank you Twitter - @dawnieando Website: https://bertey.com LinkedIn - https://www.linkedin.com/in/msdawnanderson/