01 Introduction

Transcript

1. Anne Schuth (Blendle / University of Amsterdam, The Netherlands) Krisztian Balog (University of Stavanger, Norway) Tutorial at ECIR 2016 in Padua, Italy Introduction
2. Why living labs? • IR is increasingly moving towards online evaluation • Experimenting with real, unsuspecting users in their natural task environments • The production search engine operates as a "living lab" • For a long time, this type of evaluation was only available to those that operate a search engine • This is about to change — researchers now can have direct access to such labs • CLEF Living Labs for IR lab and TREC OpenSearch track • Online evaluation will be an exciting area to work on in the future!
3. Acknowledgments • Liadh Kelly, Trinity College Dublin, Ireland • Maarten de Rijke, University of Amsterdam • Peter Dekker, University of Amsterdam • Ilya Markov, University of Amsterdam • Aleksandr Chuklin, University of Amsterdam / Google • ESF ELIAS
4. Today’s Program • Session I (9:30-11:00) 1. (Theory) Introduction to online evaluation (A/B testing, interleaving methods) - Anne ~45mins 2. (Practice) Overview of the LL4IR architecture - Krisztian ~15mins 3. (Practice) Specific use-cases: product search and academic search - Krisztian ~10mins 4. (Interactive) Obtaining an API key, getting queries and candidate items - Krisztian ~20mins • Session II (11:30-13:00) 5. (Practice) Models for the specific use-cases - Krisztian — 15mins 6. (Interactive) Generating and uploading rankings (to be shown to users) - Krisztian ~25mins 7. (Theory) Interpreting feedback using interleaving metrics - Anne ~15mins 8. (Interactive) Obtaining feedback and outcome - Krisztian ~15mins 9. (Theory) Click models, simulations - Anne ~10mins 10. (Discussion) Topics for future research - Anne ~10mins
5. Practicalities • Slides and pointers available at 
 http://living-labs.net/tutorial/