Personalization Finder: A Search Interface for Identifying and Self-controlling Web Search Personalization JCDL2020 AP-L-2: User in Search Session Y. Yamamoto et. al: “Personalization Finder: A Search Interface for Identifying and Self-controlling Web Search Personalization”, Proceedings of the 20th ACM/IEEE on Joint Conference on Digital Libraries (JCDL 2020), China, Xi’an, August 2, 2020 Takehiro Yamamoto University of Hyogo, Japan [email protected] Yusuke Yamamoto Shizuoka University, Japan [email protected]

Risk of personalization technology Recommended information = Biased information Personalization can cause a state of information isolation Filter bubble

How much do web search engines personalize search results? DuckDuckGo’s Survey: Measuring the "Filter Bubble” - How Google is influencing what you click, https://spreadprivacy.com/google-filter-bubble-study/ Compared with 87 people’s Google results for “Gun Control” in private browsing mode They saw 19 domains ordered 31 ways DuckDuckGo Survey

PERSONALIZATION Amplification of bias Topics where preferences can differ significantly Effective search Is personalization harmful? Topics where intellectual isolation is detrimental to society

PERSONALIZATION Amplification of bias Topics where preferences can differ significantly Effective search Is personalization harmful? Topics where intellectual isolation is detrimental to society ● Consideration of personalizaiton in web search ● Controlling of personalization according to topics Necessary supports

Objectives of our study User survey about people’s perception of web search personalization (WSP) • Do they know the existence of WSP? • When and how do they want to use WSP? Designing a web search interface to visualize and control WSP For promoting critical web search against filter bubble

1 People’s perception of Web search personalization

Willingness to use web search personalization 0 20 40 60 80 100 Shopping Entertainment Health Sports C lt re Science/Tech Nat re Economics Politics Ver illing Willing Ne tral Un illing Ver n illing Q. How willing are you to use personalization when using a web search engine?

Willingness to use web search personalization 0 20 40 60 80 100 Shopping Entertainment Health Sports C lt re Science/Tech Nat re Economics Politics Ver illing Willing Ne tral Un illing Ver n illing Q. How willing are you to use personalization when using a web search engine? 55% of participants were willing to use the personalization for entertainment/shopping

Willingness to use web search personalization 0 20 40 60 80 100 Shopping Entertainment Health Sports C lt re Science/Tech Nat re Economics Politics Ver illing Willing Ne tral Un illing Ver n illing 62/50% of participants were NOT willing to use the personalization for political/economic topics Q. How willing are you to use personalization when using a web search engine?

Perception of web search personalization 0 20 40 60 80 100 Shopping En e ainmen Heal h Spo C l e Science/Tech Na e Economic Poli ic > 50% 40-50% 30-40% 20-30% 10-20% < 10% Q. What percentage of web search results do you think that search engines personalize?

Perception of web search personalization 0 20 40 60 80 100 Shopping En e ainmen Heal h Spo C l e Science/Tech Na e Economic Poli ic > 50% 40-50% 30-40% 20-30% 10-20% < 10% Q. What percentage of web search results do you think that search engines personalize? Many participants noticed that the personalization often worked in the results of entertainment/shopping

Perception of web search personalization 0 20 40 60 80 100 Shopping En e ainmen Heal h Spo C l e Science/Tech Na e Economic Poli ic > 50% 40-50% 30-40% 20-30% 10-20% < 10% Q. What percentage of web search results do you think that search engines personalize? 30% of participants believed that few(less than 10%) results were personalized for politics/economics.

Summary of user survery 0 20 40 60 80 100 Shopping Entertainment Health Sports C lt re Science/Tech Nat re Economics Politics Ver illing Willing Ne tral Un illing Ver n illing 0 20 40 60 80 100 Shopping En e ainmen Heal h Spo C l e Science/Tech Na e Economic Poli ic > 50% 40-50% 30-40% 20-30% 10-20% < 10% In reality, web search engines personalize such results… & People want search results for politics/economics topics not to be personalized. People often believe that such results are not personalized.

2 Search Interface for Identifying and Self- controlling Web Search Personalization

Personalization Finder Google Chrome extension to visualize and control Web search personalization

Personalization Finder Google Chrome extension to visualize and control Web search personalization Makes searchers aware of personalization and promote critical web search

Function 1: Exposure of personalized results

Function 1: Exposure of personalized results

Function 1: Exposure of personalized results Highlighting of personalized results on SERPs Disclosure of search results hidden by the personalization

Function 1: Exposure of personalized results Highlighting of personalized results on SERPs Disclosure of search results hidden by the personalization

Function 1: Exposure of personalized results Highlighting of personalized results on SERPs Disclosure of a search result hidden by the personalization

Function 2: Summarization of the personalization effect on a sidebar

Function 2: Summarization of the personalization effect on a sidebar

Function 2: Summarization of the personalization effect on a sidebar Extent of personalization 1. Topic difference between personalized results and hidden results 2. Ratio of personalized results in the top 100 ones. What topics frequently appear in hidden search results? 53％

How to detect personalized search results? Google Custom Search API 1. Issues the same query as users input on Google site “Casino law” 2. Obtains API results. (We assume that such results not personalized) API search results 3. Compares user’s results on Google site with API results. (If a user’s result doesn’t appear in API results, we assume that it could be personalized)

3 User Study

Hypotheses on Personalization Finder (PF) 29 H1 H2 H3 With PF, users will spend more time searching the web more carefully than default web searches PF will encourage users to visit more webpages to obtain more information The effect of PF will be greater for political topics than entertainment topics

Search Task 30 • 220 Participants did web search to answer questions about political/entertainment topics. l/VDMFBS QPXFS QIBTFPVU QPMJDZl JT POF PG UIF NPTU DPOUSPWFSTJBM UPQJDT JO +BQBO 1MFBTF JNBHF UIBU ZPV IBWF CFFO BTLFE UP BOTXFS JG ZPV BHSFF PS EJTBHSFF XJUI OVDMFBS QPXFS QIBTFPVU QPMJDZ "TTVNF UIBU ZPV DPMMFDU JOGPSNBUJPO BCPVU UIF QPMJDZ WJB B XFC TFBSDI UP GPSNVMBUF ZPVS PQJOJPO 4FFL B EJTQMBZFE MJTU PG XFC TFBSDI SFTVMUT BOE WJTJU XFC QBHFT PO UIF MJTU 8IFO ZPV HFU TBUJTGBDUJPO DPODMVTJPO TUPQ UIF TFBSDI BOE SFQPSU ZPVS GJOBM PQJOJPO XJUI SFBTPOT • When they reached satisfactory conclusion, they reported their final opinion with reasons.

Experimental design 31 2 x 2 mixed factorial design UI （between-subject） Search topics （within-subject） • Control UI = default Google Search UI • Experimental UI = Control UI + proposed function • Politics • Entertainment

Two search interfaces 32 Control • Default Google Search UI． • Number of displayed results per page was fixed to 100 Experimental has proposed functions on Control UI ※ Knowledge graph and vertical search results were removed from both conditions’ results

Result 1: Dwell time on SERPs Experimental UI made users spend longer browsing SERPs for both political/entertainment topics. ● ● ● ● 0 50 100 150 200 entertainment politics Task type Dwell time on SERP UI Condition ● ● Control Experimental ● ● ● ● 0 200 400 600 800 entertainment politics Task type Session time UI Condition ● ● Control Experimental UP UP

Result 2: Click probability by rank position (1/3) What percentage of participants clicked each search result by rank position? Q.

Result 2: Click probability by rank position (1/3) Experimental UI made more users shallower click search results positions for entertainment topics.

Result 2: Click probability by rank position (2/3) Experimental UI made more users click deeper search results for political topics.

Result 2: Click probability by rank position (3/3) Experimental UI did NOT encourage users to view more webpages. If users searched for political topics with the experimental UI, they were likely to seek SERPs to lower positions

Result 3: Exit questionnaire Question Control UI Exp. UI Usefulness for objective information collection 2.95 (0.70) 3.28* (0.88) Usefulness for objective decision making 2.80 (0.75) 2.98 (0.79) Ease of use 3.10 (0.89) 3.26 (0.96) Willingness for continuous use 2.80 (0.90) 3.09* (0.90) Experimental UI could promote objective information seeking without reducing usability. ✓ ✓

Hypotheses verification (1/3) H1 With PF, users will spend more time searching the web more carefully than default web searches A. Partially Yes - PF made participants spend longer in SERPs - It did NOT make them view webpages longer

Hypotheses verification (2/3) H2 A. No, but.. - PF made users click search results with deeper positions for political topics. - Users with PF clicked search results with shallower positions for entertainment topics. PF will encourage users to visit more webpages to obtain more information

Hypotheses verification (3/3) H3 A. Partially Yes - PF had a completely different influence on click probability by rank position. The effect of PF will be greater for political topics than entertainment topics

Discussion Belief change through web search o PF made users aware of personalization effect and examine search result list longer, but it did not change their belief about topics. Personalization detection accuracy o The prototype uses Google Search API to approximate the difference between user’s search results on the Google website and non-personalized search results. o The API results are slightly older than original Google results, and so the prototype may detect personalized results incorrectly o One possible improvement would be to provide supplementary information which can make people view unfavorable information

Summary Personalization Finder: A Search Interface for Identifying and Self-controlling Web Search Personalization 44 • It made participants seek SERPs with lower positions in searching for political topics • The prototype made participants spent longer in SERPs for political/entertainment topics • A lot of people believed that web search results about political topics are NOT personalized. Email for questions: [email protected]