Information Access Systems on the Web How can we evaluate the utility of these systems? Search engines Recommender systems Conversational assistants Search box All Vertical Vertical Vertical Vertical Vertical Number of results Title of the document URL of the document A result snippet that provides a summary of the result and the context in which the search terms occur in it Title of the document URL of the document A result snippet that provides a summary of the result and the context in which the search terms occur in it Title of the document URL of the document A result snippet that provides a summary of the result and the context in which the search terms occur in it Search box Row of result items Item Item Item Item Row of result items Item Item Item Item Ad Chat 1 Chat 2 Chat 3 … Natural language user prompt Q: A: Lorem ipsum dolor sit amet, consectetur adipiscing elit. Etiam euismod felis at consequat scelerisque. Suspendisse potenti. Aenean venenatis mauris ac diam aliquet, id pretium odio condimentum. In luctus mauris eget fermentum ornare. Curabitur tempor nisl quis felis fringilla ornare. Suspendisse nec ligula augue. Curabitur sodales iaculis nulla vel pharetra. Aenean turpis est, maximus tempor diam sit amet, dapibus lobortis nibh. Suspendisse quis nibh est. Nam sollicitudin risus libero, in porttitor est iaculis vitae. Curabitur ultrices porttitor felis, vel egestas nunc rhoncus vel. Sed vulputate ante at euismod fermentum. Nulla tempus lorem orci, sed molestie metus tristique a. Duis ut condimentum neque. Nullam tellus erat, rhoncus aliquet massa non, commodo consequat quam. Aenean eu massa convallis, fringilla . Message Search box Facet Value Value Value Value Value Result Description Result Description Result Description Result Description Facet Value Value Value Value Value Result Description Result Description Result Description Result Description E-commerce platforms 

Evaluation Paradigms • Reusable test collections • facilitates large-scale repeatable and reproducible evaluation, cheap • static, the user is abstracted away • User studies • highest fidelity, captures real users’ interactions in a controlled setting • expensive, not reproducible • Online evaluation • most direct and reliable measurement of quality and user experience • no control over users, results are harder to interpret, not reproducible User simulation has the potential to address these limitations 

User Simulation Idea: having an intelligent agent to simulate how a user interacts with a system In this tutorial: • Evaluation of Web Information Access Systems • Overview of User Simulation • Simulation-based Evaluation Frameworks • User Simulation and Human Decision-making • Simulating Interactions with Search and Recommender Systems • Simulating Interactions with Conversational Assistants • Future Challenges companion website: usersim.ai