Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Personalization Finder: A Search Interface for Identifying and Self-controlling Web Search Personalization

Y. Yamamoto
August 02, 2020

Personalization Finder: A Search Interface for Identifying and Self-controlling Web Search Personalization

Yusuke Yamamoto and Takehiro Yamamoto. 2020. Personalization Finder: A Search Interface for Identifying and Self-controlling Web Search Personalization. In Proceedings of the ACM/IEEE Joint Conference on Digital Libraries in 2020 (JCDL ’20). Association for Computing Machinery, New York, NY, USA, 37–46. DOI:https://doi.org/10.1145/3383583.3398519

Y. Yamamoto

August 02, 2020
Tweet

More Decks by Y. Yamamoto

Other Decks in Research

Transcript

  1. Personalization Finder: A Search Interface
    for Identifying and Self-controlling Web
    Search Personalization
    JCDL2020 AP-L-2: User in Search Session
    Y. Yamamoto et. al: “Personalization Finder: A Search Interface for Identifying and Self-controlling Web Search Personalization”,
    Proceedings of the 20th ACM/IEEE on Joint Conference on Digital Libraries (JCDL 2020), China, Xi’an, August 2, 2020
    Takehiro Yamamoto
    University of Hyogo, Japan
    [email protected]
    Yusuke Yamamoto
    Shizuoka University, Japan
    [email protected]

    View full-size slide

  2. Risk of personalization technology
    Recommended information
    =
    Biased information
    Personalization can cause a state of information isolation
    Filter bubble

    View full-size slide

  3. How much do web search engines personalize search results?
    DuckDuckGo’s Survey: Measuring the "Filter Bubble” - How Google is influencing what you click,
    https://spreadprivacy.com/google-filter-bubble-study/
    Compared with 87 people’s Google results
    for “Gun Control” in private browsing mode
    They saw 19 domains ordered 31 ways
    DuckDuckGo Survey

    View full-size slide

  4. PERSONALIZATION
    Amplification of bias
    Topics where preferences
    can differ significantly
    Effective search
    Is personalization harmful?
    Topics where
    intellectual isolation is
    detrimental to society

    View full-size slide

  5. PERSONALIZATION
    Amplification of bias
    Topics where preferences
    can differ significantly
    Effective search
    Is personalization harmful?
    Topics where
    intellectual isolation is
    detrimental to society
    ● Consideration of personalizaiton in web search
    ● Controlling of personalization according to topics
    Necessary supports

    View full-size slide

  6. Objectives of our study
    User survey about people’s perception
    of web search personalization (WSP)
    • Do they know the existence of WSP?
    • When and how do they want to use WSP?
    Designing a web search interface to
    visualize and control WSP
    For promoting critical web search against filter bubble

    View full-size slide

  7. 1 People’s perception of
    Web search personalization

    View full-size slide

  8. Willingness to use web search personalization
    0 20 40 60 80 100
    Shopping
    Entertainment
    Health
    Sports
    C lt re
    Science/Tech
    Nat re
    Economics
    Politics
    Ver illing Willing
    Ne tral Un illing
    Ver n illing
    Q. How willing are you to use personalization
    when using a web search engine?

    View full-size slide

  9. Willingness to use web search personalization
    0 20 40 60 80 100
    Shopping
    Entertainment
    Health
    Sports
    C lt re
    Science/Tech
    Nat re
    Economics
    Politics
    Ver illing Willing
    Ne tral Un illing
    Ver n illing
    Q. How willing are you to use personalization
    when using a web search engine?
    55% of participants were willing to use
    the personalization for entertainment/shopping

    View full-size slide

  10. Willingness to use web search personalization
    0 20 40 60 80 100
    Shopping
    Entertainment
    Health
    Sports
    C lt re
    Science/Tech
    Nat re
    Economics
    Politics
    Ver illing Willing
    Ne tral Un illing
    Ver n illing
    62/50% of participants were NOT willing to use
    the personalization for political/economic topics
    Q. How willing are you to use personalization
    when using a web search engine?

    View full-size slide

  11. Perception of web search personalization
    0 20 40 60 80 100
    Shopping
    En e ainmen
    Heal h
    Spo
    C l e
    Science/Tech
    Na e
    Economic
    Poli ic
    > 50% 40-50% 30-40%
    20-30% 10-20% < 10%
    Q. What percentage of web search results do you
    think that search engines personalize?

    View full-size slide

  12. Perception of web search personalization
    0 20 40 60 80 100
    Shopping
    En e ainmen
    Heal h
    Spo
    C l e
    Science/Tech
    Na e
    Economic
    Poli ic
    > 50% 40-50% 30-40%
    20-30% 10-20% < 10%
    Q. What percentage of web search results do you
    think that search engines personalize?
    Many participants noticed that the personalization
    often worked in the results of entertainment/shopping

    View full-size slide

  13. Perception of web search personalization
    0 20 40 60 80 100
    Shopping
    En e ainmen
    Heal h
    Spo
    C l e
    Science/Tech
    Na e
    Economic
    Poli ic
    > 50% 40-50% 30-40%
    20-30% 10-20% < 10%
    Q. What percentage of web search results do you
    think that search engines personalize?
    30% of participants believed that few(less than 10%)
    results were personalized for politics/economics.

    View full-size slide

  14. Summary of user survery
    0 20 40 60 80 100
    Shopping
    Entertainment
    Health
    Sports
    C lt re
    Science/Tech
    Nat re
    Economics
    Politics
    Ver illing Willing
    Ne tral Un illing
    Ver n illing
    0 20 40 60 80 100
    Shopping
    En e ainmen
    Heal h
    Spo
    C l e
    Science/Tech
    Na e
    Economic
    Poli ic
    > 50% 40-50% 30-40%
    20-30% 10-20% < 10%
    In reality, web search engines personalize such results… &
    People want search results for
    politics/economics topics not to be personalized.
    People often believe that such results are
    not personalized.

    View full-size slide

  15. 2 Search Interface for Identifying and Self-
    controlling Web Search Personalization

    View full-size slide

  16. Personalization Finder
    Google Chrome extension to visualize and
    control Web search personalization

    View full-size slide

  17. Personalization Finder
    Google Chrome extension to visualize and
    control Web search personalization
    Makes searchers aware of personalization
    and promote critical web search

    View full-size slide

  18. Function 1: Exposure of personalized results

    View full-size slide

  19. Function 1: Exposure of personalized results

    View full-size slide

  20. Function 1: Exposure of personalized results
    Highlighting of personalized
    results on SERPs
    Disclosure of search results
    hidden by the personalization

    View full-size slide

  21. Function 1: Exposure of personalized results
    Highlighting of personalized
    results on SERPs
    Disclosure of search results
    hidden by the personalization

    View full-size slide

  22. Function 1: Exposure of personalized results
    Highlighting of personalized
    results on SERPs
    Disclosure of a search result
    hidden by the personalization

    View full-size slide

  23. Function 2: Summarization of the personalization effect on a sidebar

    View full-size slide

  24. Function 2: Summarization of the personalization effect on a sidebar

    View full-size slide

  25. Function 2: Summarization of the personalization effect on a sidebar
    Extent of personalization
    1.
    Topic difference between
    personalized results and
    hidden results
    2.
    Ratio of personalized results
    in the top 100 ones.
    What topics frequently appear in
    hidden search results?
    53%

    View full-size slide

  26. How to detect personalized search results?
    Google Custom
    Search API
    1. Issues the same query
    as users input on Google site
    “Casino law”
    2. Obtains API results.
    (We assume that such results
    not personalized)
    API search results
    3. Compares user’s results on
    Google site with API results.
    (If a user’s result doesn’t appear in
    API results, we assume that it could
    be personalized)

    View full-size slide

  27. 3 User Study

    View full-size slide

  28. Hypotheses on Personalization Finder (PF)
    29
    H1
    H2
    H3
    With PF, users will spend more time searching the
    web more carefully than default web searches
    PF will encourage users to visit more webpages to
    obtain more information
    The effect of PF will be greater for political topics
    than entertainment topics

    View full-size slide

  29. Search Task
    30
    • 220 Participants did web search to answer
    questions about political/entertainment topics.
    l/VDMFBS QPXFS QIBTFPVU QPMJDZl JT POF PG UIF NPTU
    DPOUSPWFSTJBM UPQJDT JO +BQBO 1MFBTF JNBHF UIBU ZPV
    IBWF CFFO BTLFE UP BOTXFS JG ZPV BHSFF PS EJTBHSFF XJUI
    OVDMFBS QPXFS QIBTFPVU QPMJDZ "TTVNF UIBU ZPV
    DPMMFDU JOGPSNBUJPO BCPVU UIF QPMJDZ WJB B XFC TFBSDI UP
    GPSNVMBUF ZPVS PQJOJPO 4FFL B EJTQMBZFE MJTU PG XFC
    TFBSDI SFTVMUT BOE WJTJU XFC QBHFT PO UIF MJTU 8IFO ZPV
    HFU TBUJTGBDUJPO DPODMVTJPO TUPQ UIF TFBSDI BOE SFQPSU
    ZPVS GJOBM PQJOJPO XJUI SFBTPOT
    • When they reached satisfactory conclusion,
    they reported their final opinion with reasons.

    View full-size slide

  30. Experimental design
    31
    2 x 2 mixed factorial design
    UI (between-subject)
    Search topics (within-subject)
    • Control UI = default Google Search UI
    • Experimental UI = Control UI + proposed function
    • Politics
    • Entertainment

    View full-size slide

  31. Two search interfaces
    32
    Control
    • Default Google Search UI.
    • Number of displayed results
    per page was fixed to 100
    Experimental
    has proposed functions on
    Control UI
    ※ Knowledge graph and vertical search results were removed from both conditions’ results

    View full-size slide

  32. Result 1: Dwell time on SERPs
    Experimental UI made users spend longer browsing
    SERPs for both political/entertainment topics.




    0
    50
    100
    150
    200
    entertainment politics
    Task type
    Dwell time on SERP
    UI Condition


    Control
    Experimental

    ● ●

    0
    200
    400
    600
    800
    entertainment politics
    Task type
    Session time
    UI Condition


    Control
    Experimental
    UP UP

    View full-size slide

  33. Result 2: Click probability by rank position (1/3)
    What percentage of participants clicked
    each search result by rank position?
    Q.

    View full-size slide

  34. Result 2: Click probability by rank position (1/3)
    Experimental UI made more users shallower click
    search results positions for entertainment topics.

    View full-size slide

  35. Result 2: Click probability by rank position (2/3)
    Experimental UI made more users click
    deeper search results for political topics.

    View full-size slide

  36. Result 2: Click probability by rank position (3/3)
    Experimental UI did NOT encourage users
    to view more webpages.
    If users searched for political topics with the
    experimental UI, they were likely to seek SERPs
    to lower positions

    View full-size slide

  37. Result 3: Exit questionnaire
    Question Control UI Exp. UI
    Usefulness for objective
    information collection
    2.95
    (0.70)
    3.28*
    (0.88)
    Usefulness for objective decision
    making
    2.80
    (0.75)
    2.98
    (0.79)
    Ease of use
    3.10
    (0.89)
    3.26
    (0.96)
    Willingness for continuous use
    2.80
    (0.90)
    3.09*
    (0.90)
    Experimental UI could promote objective
    information seeking without reducing usability.


    View full-size slide

  38. Hypotheses verification (1/3)
    H1
    With PF, users will spend more time searching the
    web more carefully than default web searches
    A. Partially Yes
    - PF made participants spend longer in SERPs
    - It did NOT make them view webpages longer

    View full-size slide

  39. Hypotheses verification (2/3)
    H2
    A. No, but..
    - PF made users click search results with deeper
    positions for political topics.
    - Users with PF clicked search results with
    shallower positions for entertainment topics.
    PF will encourage users to visit more webpages to
    obtain more information

    View full-size slide

  40. Hypotheses verification (3/3)
    H3
    A. Partially Yes
    - PF had a completely different influence on click
    probability by rank position.
    The effect of PF will be greater for political topics
    than entertainment topics

    View full-size slide

  41. Discussion
    Belief change through web search
    o PF made users aware of personalization effect and examine search
    result list longer, but it did not change their belief about topics.
    Personalization detection accuracy
    o The prototype uses Google Search API to approximate the
    difference between user’s search results on the Google website and
    non-personalized search results.
    o The API results are slightly older than original Google results, and
    so the prototype may detect personalized results incorrectly
    o One possible improvement would be to provide supplementary
    information which can make people view unfavorable information

    View full-size slide

  42. Summary
    Personalization Finder: A Search Interface for
    Identifying and Self-controlling Web Search
    Personalization
    44
    • It made participants seek SERPs
    with lower positions in searching for political topics
    • The prototype made participants spent longer in SERPs for
    political/entertainment topics
    • A lot of people believed that web search results about
    political topics are NOT personalized.
    Email for questions: [email protected]

    View full-size slide