Slide 28
Slide 28 text
ɹConfidentialɹ© TDAI Lab All rights reserved.
Supervised Learning-to Rank
Traditionally, learning to rank is supervised through annotated datasets:
Relevance annotations for query-document pairs provided by human judges.
However, over time several limitations of this approach have become apparent.
ɾexpensive to make (Qin and Liu, 2013; Chapelle and Chang, 2011).
ɾunethical to create in privacy-sensitive settings (Wang et al., 2016a).
ɾimpossible for small scale problems, e.g., personalization.
ɾstationary, cannot capture future changes in relevancy (Lefortier et al., 2014).
ɾnot necessarily aligned with actual user preferences (Sanderson, 2010),
i.e., annotators and users often disagree
ɹɹ
Can we naively use click as annotation?
ɹɹ
The model will be biased
if you are not careful