Analyzing the effects of document’s opinion and credibility on search behaviors and belief dynamics 1 Yoshikawa-Ma Laboratory, Kyoto University Suppanut Pothirattanachaikul Kyoto University Takehiro Yamamoto University of Hyogo Yusuke Yamamoto Shizuoka University Masatoshi Yoshikawa Kyoto University

Background 2

Background (1/3) To obtain accurate information, searchers should explore diverse information and carefully verify information[1,2] 3 [1] K.R. Garrett and P. Resnick (2011).: Resisting political fragmentation on the internet. [2] Y. Yamamoto et al. (2018) .: Web access literacy scale to evaluate how critically users can browse and search for web information. i.e. issue more queries, verify authors’ information green tea pills lose weight green tea pills evidence green tea pills risks (a) issue more queries (b) verify author’s information Green tea for weight loss ? Dr. Yamada Taro (MD.)

Background (2/3) Unfortunately, searchers are likely to seek the information that confirms their beliefs[3] 4 [3] R. W. White (2013) .: Beliefs and biases in web search This phenomenon is known as “confirmation bias” Green tea is helpful for weight loss green tea pills Benefit of green tea pills Buy green tea pills

Background (2/3) Unfortunately, searchers are likely to seek the information that confirms their beliefs[3] 5 [3] R. W. White (2013) .: Beliefs and biases in web search Green tea is helpful for weight loss green tea pills Benefit of green tea pills Buy green tea pills Evidence that green tea is not helpful for weight loss This phenomenon is known as “confirmation bias”

Background (3/3) Previous studies suggested that opinion and quality of information affect confirmation bias[4] 6 i.e. low quality and belief-consistent information mitigates confirmation bias[5] [4] L. Festinger (1964) .: Conflict, decision and dissonance [5] I. Hernandez (2012).: Disfluency disrupts the confirmation bias Green tea is helpful for weight loss Green tea is not helpful Anonymous author

Background (3/3) Previous studies suggested that opinion and quality of information affect confirmation bias[4] 7 i.e. low quality and belief-consistent information mitigates confirmation bias[5] [4] L. Festinger (1964) .: Conflict, decision and dissonance [5] I. Hernandez (2012).: Disfluency disrupts the confirmation bias Is green tea really helpful ? Green tea is not helpful Anonymous author

Research Goal Understanding the effect of the documents’ opinion and credibility on the users’ search behaviors and belief dynamics 8 Green tea is helpful for weight loss Green tea is not helpful Dr. Yamada Taro Opinion: Inconsistent Credibility: High

Research Goal Understanding the effect of the documents’ opinion and credibility on the users’ search behaviors and belief dynamics 9 green tea pills lose weight green tea pills evidence green tea pills risks Green tea is not helpful

- Issue more queries - Click on more documents - etc. Research Question 1 10 RQ1 How do opinions and credibility encourage or discourage searchers to perform the careful information search ? Green tea is not helpful Dr. Yamada Taro Opinion: Inconsistent Credibility: High

Research Question 2 11 RQ2 How do opinions and credibility affect searchers’ belief dynamics ? Green tea is not helpful Dr. Yamada Taro Green tea is effective Before ! Green tea is NOT effective After ! Opinion: Inconsistent Credibility: High

Research Question 3 12 RQ3 Is there any relationship between search behaviors and belief dynamics ? Green tea is effective Before ! Green tea is NOT effective After ! - Issue more queries - Click on more documents - etc. ?

Take Away ● Belief-inconsistent opinions encourage participants to issue more queries ● Belief-inconsistent opinions also encourage participants to alter their beliefs ● Belief dynamics and the effort on search process had few relationships 13

Concept 14

15 Polarity of the document’s content compare to the users’ prior belief, a belief before performing a search Factor (1/2) - Opinion Prior Belief: Dairy Products are helpful Consistent Dairy products are not helpful Inconsistent Dairy products are helpful

16 Opinion: We manually prepared the content that supports/opposes for each search task T1 : Are dairy products effective in improving hypertension ? Dairy products are effective Dairy products are NOT effective Supporting Opposing Factor (2/2) - Opinion

17 Credibility: Believability of information [11] [11] B. J. Fogg (2003) .: Persuasive technology Using computers to change what we think and do Low Credibility High Credibility [Clean, clear, and simple] In this study, credibility is judged in accordance to the visual styles Factor - Credibility

18 Dairy products are effective Supporting + High Credibility [Clean, clear, and simple] Controlled Document Controlled Document

19 Controlled document was inserted in the second rank for all queries dairy products hypertension Dairy product does not improve hypertension … hypertension Dairy product does not improve hypertension … Search System

Method 20

21 Experimental Design ● 2x2 within subject design Opinion Credibility Green tea extract is helpful Consistent Green tea extract is not helpful Inconsistent High Credibility Low Credibility X

22 Experimental Design ● For each condition, the participants were randomly assigned to one of the four medical yes-no questions[10] Task# Questions 1 Are dairy products effective in improving hypertension? 2 Are Isoflavones effective in relieving high cholesterol? … [10] Y Yamamoto et al. (2018) .: Web access literacy scale to evaluate how critically users can browse and search for Web information

23 Procedure: Task Instruction ● Instructions comprised background of the symptom and remedy Instruction “ Hypertension tends to be a disease of the heart and vasculature… that dairy products Symptom Remedy please conduct a web search to examine whether dairy products are effective in improving hypertension.”

24 Procedure: Pre-Task Questionnaire ● Participants have to provide beliefs to the remedy prior to the search (prior-belief) no lean-no lean-yes yes Do you believe that dairy products are effective in improving hypertension?

25 Procedure: Search Task dairy products hypertension Belief: Yes + Condition: Inconsistent, High Credibility Dairy product does not improve hypertension Controlled Document … “Participants were allowed to issue any queries or click on any documents”

26 Procedure: Search Task Screenshot of the actual search system

27 Procedure: Post-Task Questionnaire ● Participants have to provide beliefs to the remedy after search (posterior-belief) no lean-no lean-yes yes Do you believe that dairy products are effective in improving hypertension?

28 Procedure: Post-Task Questionnaire ● Also, evidence that supports the posterior beliefs Benefit of Diary Products http://www.dairyp.com Buy Diary Products http://www.buydairy.com Please select the document that support your belief

29 Procedure: Exit Questionnaire ● Participants were asked if anything had disturbed or bothered them during the search task ○ To filter out the participants who noticed the search results manipulation If anything bother you about our search task､ please feel free to write here (optional)

30 Participants[*] ● # of participants: 275 (401 before filtered) ● Duration: July 10 to August 3, 2018 ● # of search sessions: 822 Gender Education Search Engine Usage Male: 120 Female: 149 N/A: 6 University 70 Non- University 160 N/A: 45 Rarely use: 1 Several times a week: 22 Once a day: 47 More than once a day: 205 [*] Participants were recruited from Lancers.jp; Japanese crowd-sourcing service

Result 31

- Issue more queries - Click on more documents - etc. RQ1 (1/4) - Background 32 RQ1 How do opinions and credibility encourage or discourage searchers to perform the careful information search ? Green tea is not helpful Dr. Yamada Taro Opinion: Inconsistent Credibility: High

33 Belief-inconsistent encourages participants to issue more queries RQ1 (2/4) - Result Inconsistent Consistent M M # of Queries 2.70 2.41 (" = 0.12, ) < 0.05)

34 Participants prefer spending more time on the high credibility controlled document RQ1 (3/4) - Result High Low M M Controlled document dwell time (sec) 19.57 12.19 (" = 0.10, ( < 0.01)

RQ1 (4/4) - Conclusion 35 RQ1 How do opinions and credibility encourage or discourage searchers to perform the careful information search ? • Answer: Belief-inconsistent opinions encouraged the participants to issue more queries. • However, a further study is needed since the difference seems not large

RQ2 (1/4) - Background 36 RQ2 How do opinions and credibility affect searchers’ belief dynamics ? Green tea is not helpful Dr. Yamada Taro Green tea is effective Before ! Green tea is NOT effective After ! Opinion: Inconsistent Credibility: High

RQ2 (2/4) - Results 37 Belief-inconsistent opinion encourages participants to alter their belief to the opposite polar( ) Inconsistent, High Credibility No Lean-No Lean- Yes Yes No 51.62% Lean No 71.43% Prior Belief Posterior Belief Inconsistent, Low Credibility No Lean-No Lean-Yes Yes No 51.62% Lean No 75.76% Prior Belief Posterior Belief 1 1 1

RQ2 (3/4): Results ● Belief-consistent opinions encourage participants to retain their beliefs in the same polar ( , ) ○ They were less likely to retain their beliefs when it presented in low credibility ( ) 38 Consistent, High Credibility No Lean- No Lean- Yes Yes No 80.85% Lean No 71.88% Lean Yes 84.61% Yes 100.00% Prior Belief Posterior Belief Consistent, Low Credibility No Lean- No Lean- Yes Yes No 75.68% Lean No 55.55% Lean Yes 84.21% Yes 100.00% Prior Belief Posterior Belief 3 3 4 4 3 4 4

RQ2 (4/4) - Conclusion 39 • Answer: Belief-inconsistent encouraged the participants to alter the belief • Belief-consistent strengthen the confirmation bias RQ2 How do opinions and credibility affect searchers’ belief dynamics ?

Result Summary ● Belief-inconsistent opinions encourage participants to issue more queries ● Belief-inconsistent opinions also encourage participants to alter their beliefs ● Belief dynamics and the effort on search process had few relationships 40

RQ3: Results ○ Belief dynamics and their effort on search process had few relationships 41 Alter Retain M M # of Consistent Clicks 0.70 1.83 # of Inconsistent Clicks 3.10 1.65 RQ3 Is there any relationship between search behaviors and belief dynamics ?

Implication (1/2) ● Search engine should present the belief-inconsistent documents at the higher rank ○ So that, it encourage users to perform careful information search 42 Green tea is helpful for losing weight Green tea weight loss Green Tea pills are risky …

Implication (2/2) ● Displaying belief-consistent documents under the high credibility exacerbate confirmation bias 43 Green tea is helpful for losing weight Green tea is helpful Dr. Yamada Taro (MD.)

Conclusion ● Research Goal ○ To understand the effect of opinion and credibility of a document on the users’ search behaviors and belief dynamics ● Method ○ Conducted a user study to investigate the effect of different opinions and credibility levels on the participants’ search behaviors ● Result ○ Belief-inconsistent opinions encourage participants to issue more queries ○ Belief-inconsistent also encourage participants to alter their beliefs ○ Belief dynamics and the effort on search process had few relationships 44