Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Analyzing the Effects of Document's Opinion and Credibility on Search Behaviors and Belief Dynamics

Analyzing the Effects of Document's Opinion and Credibility on Search Behaviors and Belief Dynamics

711a843fe96222cc55dc9ff9b8549ecd?s=128

Takehiro Yamamoto

November 04, 2019
Tweet

More Decks by Takehiro Yamamoto

Other Decks in Research

Transcript

  1. Analyzing the effects of document’s opinion and credibility on search

    behaviors and belief dynamics 1 Yoshikawa-Ma Laboratory, Kyoto University Suppanut Pothirattanachaikul Kyoto University Takehiro Yamamoto University of Hyogo Yusuke Yamamoto Shizuoka University Masatoshi Yoshikawa Kyoto University
  2. Background 2

  3. Background (1/3) To obtain accurate information, searchers should explore diverse

    information and carefully verify information[1,2] 3 [1] K.R. Garrett and P. Resnick (2011).: Resisting political fragmentation on the internet. [2] Y. Yamamoto et al. (2018) .: Web access literacy scale to evaluate how critically users can browse and search for web information. i.e. issue more queries, verify authors’ information green tea pills lose weight green tea pills evidence green tea pills risks (a) issue more queries (b) verify author’s information Green tea for weight loss ? Dr. Yamada Taro (MD.)
  4. Background (2/3) Unfortunately, searchers are likely to seek the information

    that confirms their beliefs[3] 4 [3] R. W. White (2013) .: Beliefs and biases in web search This phenomenon is known as “confirmation bias” Green tea is helpful for weight loss green tea pills Benefit of green tea pills Buy green tea pills
  5. Background (2/3) Unfortunately, searchers are likely to seek the information

    that confirms their beliefs[3] 5 [3] R. W. White (2013) .: Beliefs and biases in web search Green tea is helpful for weight loss green tea pills Benefit of green tea pills Buy green tea pills Evidence that green tea is not helpful for weight loss This phenomenon is known as “confirmation bias”
  6. Background (3/3) Previous studies suggested that opinion and quality of

    information affect confirmation bias[4] 6 i.e. low quality and belief-consistent information mitigates confirmation bias[5] [4] L. Festinger (1964) .: Conflict, decision and dissonance [5] I. Hernandez (2012).: Disfluency disrupts the confirmation bias Green tea is helpful for weight loss Green tea is not helpful Anonymous author
  7. Background (3/3) Previous studies suggested that opinion and quality of

    information affect confirmation bias[4] 7 i.e. low quality and belief-consistent information mitigates confirmation bias[5] [4] L. Festinger (1964) .: Conflict, decision and dissonance [5] I. Hernandez (2012).: Disfluency disrupts the confirmation bias Is green tea really helpful ? Green tea is not helpful Anonymous author
  8. Research Goal Understanding the effect of the documents’ opinion and

    credibility on the users’ search behaviors and belief dynamics 8 Green tea is helpful for weight loss Green tea is not helpful Dr. Yamada Taro Opinion: Inconsistent Credibility: High
  9. Research Goal Understanding the effect of the documents’ opinion and

    credibility on the users’ search behaviors and belief dynamics 9 green tea pills lose weight green tea pills evidence green tea pills risks Green tea is not helpful
  10. - Issue more queries - Click on more documents -

    etc. Research Question 1 10 RQ1 How do opinions and credibility encourage or discourage searchers to perform the careful information search ? Green tea is not helpful Dr. Yamada Taro Opinion: Inconsistent Credibility: High
  11. Research Question 2 11 RQ2 How do opinions and credibility

    affect searchers’ belief dynamics ? Green tea is not helpful Dr. Yamada Taro Green tea is effective Before ! Green tea is NOT effective After ! Opinion: Inconsistent Credibility: High
  12. Research Question 3 12 RQ3 Is there any relationship between

    search behaviors and belief dynamics ? Green tea is effective Before ! Green tea is NOT effective After ! - Issue more queries - Click on more documents - etc. ?
  13. Take Away • Belief-inconsistent opinions encourage participants to issue more

    queries • Belief-inconsistent opinions also encourage participants to alter their beliefs • Belief dynamics and the effort on search process had few relationships 13
  14. Concept 14

  15. 15 Polarity of the document’s content compare to the users’

    prior belief, a belief before performing a search Factor (1/2) - Opinion Prior Belief: Dairy Products are helpful Consistent Dairy products are not helpful Inconsistent Dairy products are helpful
  16. 16 Opinion: We manually prepared the content that supports/opposes for

    each search task T1 : Are dairy products effective in improving hypertension ? Dairy products are effective Dairy products are NOT effective Supporting Opposing Factor (2/2) - Opinion
  17. 17 Credibility: Believability of information [11] [11] B. J. Fogg

    (2003) .: Persuasive technology Using computers to change what we think and do Low Credibility High Credibility [Clean, clear, and simple] In this study, credibility is judged in accordance to the visual styles Factor - Credibility
  18. 18 Dairy products are effective Supporting + High Credibility [Clean,

    clear, and simple] Controlled Document Controlled Document
  19. 19 Controlled document was inserted in the second rank for

    all queries dairy products hypertension Dairy product does not improve hypertension … hypertension Dairy product does not improve hypertension … Search System
  20. Method 20

  21. 21 Experimental Design • 2x2 within subject design Opinion Credibility

    Green tea extract is helpful Consistent Green tea extract is not helpful Inconsistent High Credibility Low Credibility X
  22. 22 Experimental Design • For each condition, the participants were

    randomly assigned to one of the four medical yes-no questions[10] Task# Questions 1 Are dairy products effective in improving hypertension? 2 Are Isoflavones effective in relieving high cholesterol? … [10] Y Yamamoto et al. (2018) .: Web access literacy scale to evaluate how critically users can browse and search for Web information
  23. 23 Procedure: Task Instruction • Instructions comprised background of the

    symptom and remedy Instruction “ Hypertension tends to be a disease of the heart and vasculature… that dairy products Symptom Remedy please conduct a web search to examine whether dairy products are effective in improving hypertension.”
  24. 24 Procedure: Pre-Task Questionnaire • Participants have to provide beliefs

    to the remedy prior to the search (prior-belief) no lean-no lean-yes yes Do you believe that dairy products are effective in improving hypertension?
  25. 25 Procedure: Search Task dairy products hypertension Belief: Yes +

    Condition: Inconsistent, High Credibility Dairy product does not improve hypertension Controlled Document … “Participants were allowed to issue any queries or click on any documents”
  26. 26 Procedure: Search Task Screenshot of the actual search system

  27. 27 Procedure: Post-Task Questionnaire • Participants have to provide beliefs

    to the remedy after search (posterior-belief) no lean-no lean-yes yes Do you believe that dairy products are effective in improving hypertension?
  28. 28 Procedure: Post-Task Questionnaire • Also, evidence that supports the

    posterior beliefs Benefit of Diary Products http://www.dairyp.com Buy Diary Products http://www.buydairy.com Please select the document that support your belief
  29. 29 Procedure: Exit Questionnaire • Participants were asked if anything

    had disturbed or bothered them during the search task ◦ To filter out the participants who noticed the search results manipulation If anything bother you about our search task、 please feel free to write here (optional)
  30. 30 Participants[*] • # of participants: 275 (401 before filtered)

    • Duration: July 10 to August 3, 2018 • # of search sessions: 822 Gender Education Search Engine Usage Male: 120 Female: 149 N/A: 6 University 70 Non- University 160 N/A: 45 Rarely use: 1 Several times a week: 22 Once a day: 47 More than once a day: 205 [*] Participants were recruited from Lancers.jp; Japanese crowd-sourcing service
  31. Result 31

  32. - Issue more queries - Click on more documents -

    etc. RQ1 (1/4) - Background 32 RQ1 How do opinions and credibility encourage or discourage searchers to perform the careful information search ? Green tea is not helpful Dr. Yamada Taro Opinion: Inconsistent Credibility: High
  33. 33 Belief-inconsistent encourages participants to issue more queries RQ1 (2/4)

    - Result Inconsistent Consistent M M # of Queries 2.70 2.41 (" = 0.12, ) < 0.05)
  34. 34 Participants prefer spending more time on the high credibility

    controlled document RQ1 (3/4) - Result High Low M M Controlled document dwell time (sec) 19.57 12.19 (" = 0.10, ( < 0.01)
  35. RQ1 (4/4) - Conclusion 35 RQ1 How do opinions and

    credibility encourage or discourage searchers to perform the careful information search ? • Answer: Belief-inconsistent opinions encouraged the participants to issue more queries. • However, a further study is needed since the difference seems not large
  36. RQ2 (1/4) - Background 36 RQ2 How do opinions and

    credibility affect searchers’ belief dynamics ? Green tea is not helpful Dr. Yamada Taro Green tea is effective Before ! Green tea is NOT effective After ! Opinion: Inconsistent Credibility: High
  37. RQ2 (2/4) - Results 37 Belief-inconsistent opinion encourages participants to

    alter their belief to the opposite polar( ) Inconsistent, High Credibility No Lean-No Lean- Yes Yes No 51.62% Lean No 71.43% Prior Belief Posterior Belief Inconsistent, Low Credibility No Lean-No Lean-Yes Yes No 51.62% Lean No 75.76% Prior Belief Posterior Belief 1 1 1
  38. RQ2 (3/4): Results • Belief-consistent opinions encourage participants to retain

    their beliefs in the same polar ( , ) ◦ They were less likely to retain their beliefs when it presented in low credibility ( ) 38 Consistent, High Credibility No Lean- No Lean- Yes Yes No 80.85% Lean No 71.88% Lean Yes 84.61% Yes 100.00% Prior Belief Posterior Belief Consistent, Low Credibility No Lean- No Lean- Yes Yes No 75.68% Lean No 55.55% Lean Yes 84.21% Yes 100.00% Prior Belief Posterior Belief 3 3 4 4 3 4 4
  39. RQ2 (4/4) - Conclusion 39 • Answer: Belief-inconsistent encouraged the

    participants to alter the belief • Belief-consistent strengthen the confirmation bias RQ2 How do opinions and credibility affect searchers’ belief dynamics ?
  40. Result Summary • Belief-inconsistent opinions encourage participants to issue more

    queries • Belief-inconsistent opinions also encourage participants to alter their beliefs • Belief dynamics and the effort on search process had few relationships 40
  41. RQ3: Results ◦ Belief dynamics and their effort on search

    process had few relationships 41 Alter Retain M M # of Consistent Clicks 0.70 1.83 # of Inconsistent Clicks 3.10 1.65 RQ3 Is there any relationship between search behaviors and belief dynamics ?
  42. Implication (1/2) • Search engine should present the belief-inconsistent documents

    at the higher rank ◦ So that, it encourage users to perform careful information search 42 Green tea is helpful for losing weight Green tea weight loss Green Tea pills are risky …
  43. Implication (2/2) • Displaying belief-consistent documents under the high credibility

    exacerbate confirmation bias 43 Green tea is helpful for losing weight Green tea is helpful Dr. Yamada Taro (MD.)
  44. Conclusion • Research Goal ◦ To understand the effect of

    opinion and credibility of a document on the users’ search behaviors and belief dynamics • Method ◦ Conducted a user study to investigate the effect of different opinions and credibility levels on the participants’ search behaviors • Result ◦ Belief-inconsistent opinions encourage participants to issue more queries ◦ Belief-inconsistent also encourage participants to alter their beliefs ◦ Belief dynamics and the effort on search process had few relationships 44