WISE2019 発表資料

Ef88a42bd35d7ca0b1a3a9d6bc9eee3e?s=47 ymmt3lab
January 12, 2020

WISE2019 発表資料

発表タイトル:Highlighting Weasel Sentences for Promoting Critical Information Seeking on the Web

Ef88a42bd35d7ca0b1a3a9d6bc9eee3e?s=128

ymmt3lab

January 12, 2020
Tweet

Transcript

  1. 1.

    Highlighting Weasel Sentences for Promoting Critical Information Seeking on the

    Web Fumiaki Saito1 , Yoshiyuki Shoji2 ,Yusuke Yamamoto1 1:Shizuoka university, Japan 2: Aoyama Gakuin ,Japan January 19,2020 1 WISE 2019: Session S7
  2. 2.

    Background: Web information is not always correct 2 The number

    of medical Web sites authorized by medical experts: < 50%* * E. Sillence et al., “Trust and Mistrust of Online Health Sites”, ACM CHI, pp.663-670, 2004
  3. 4.

    Examples of credibility analysis systems *2 Y. Yamamoto and K.

    Tanaka. Enhancing Credibility Judgment of Web Search Results. In Proceedings of the 29th ACM SIGCHI Conference on Human Factors in Computing Systems (CHI 2011), pages 1235–1244, 2011. *1 Yin, X., Han, J., & Philip, S. Y. (2008). Truth discovery with multiple conflicting information providers on the web. IEEE Transactions on Knowledge and Data Engineering, 20(6), 796-808. TruthFinder*1 Scores the consistency of fact describing objects CowSearch*2 Provides supporting information for credibility judgment The analysis does not guarantee the correctness of information Limitation 4
  4. 5.

    Possible approach in information science 5 Obtaining correct information (Semi-)

    automatic analysis on information credibility Careful examination on information by users
  5. 6.

    57% Many people are not aware of Web information credibility!

    82% # of young Japanese people who trust Web information *1 *1 Adobe Inc., “The State of Content : Rules of Engagement”, 2015 *2 S.Nakamura et al., “Trustworthiness analysis of Web search results (ECDL 2007) Never mind, never mind. # of people who trust information on SERP *2 I trust Google 6
  6. 7.

    Why don’t people often take care of Web information credibility

    lTrust in search engine’s ranking1 lWrong metrics for quality judgment (e.g. appearance of websites2) lCognitive bias3 7 1:Kahneman, D.: Thinking, fast and slow, Macmillan (2011) 2:Pan, B., Hembrooke, H., Joachims, et al In Google We Trust: Users’ Decisions on Rank, Position, and Relevance 3:Fogg, B. J, Soohoo, Cathy ,Danielson, David R, et al How Do Users Evaluate the Credibility of Web Sites? A Study with over 2,500 Participants ♥
  7. 10.

    Proposed system 10 Automatically detects and highlights ambiguous sentences on

    the Web They lack in evidences but seem to have no problems in their claims Ambiguous sentences → In this study
  8. 11.

    Proposed system to highlight ambiguous sentences 11 Makes users aware

    of information credibility and promotes careful information seeking !!
  9. 14.

    What are weasel expressions ? 14 l“Certain person concerned is

    ... ” l“Research has shown...” 14 l“ It is often said ... ” l“ It is widely thought ... ” Who said that? What is the truth? Weasel Expressions create an impression that something specific and meaningful has been said, although their claim is ambiguous and lacks in evidence.
  10. 15.

    Classification of weasel sentences We built a simple weasel sentence

    classifier with Wikipedia data 15 Wikipedia Ambiguous sentences Non-ambiguous ones Training/test set ML (SVM)
  11. 16.

    Training/test dataset to classify weasel sentences (1/2) We focused on

    Wikipedia editting rules. Some weasel expressions are annotated with the special tags in Wikipedia. Ex: Who said? 16 It is said that Company A has said this hoax. [By whom?]
  12. 17.

    Weasel expressions in Wikipedia l“The hoax is said to have

    been company A, a media and advertising company that wanted to show off their influence.[by whom?] ” l“ ~ is an established theory, but there are some objections. [who?]” 17 Image reference https://en.wikipedia.org/wiki/Wikipedia_logo
  13. 18.

    Training/test dataset to classify weasel sentences (2/2) Positive examples (2236

    senteneces) Sentences annotated with [who?][by whom?] on Wikipedia 18 Negative examples(2236 senteneces) Sentences without [who?][by whom?] tags in Wikipedia articles where weasel expressions appear.
  14. 19.

    Features for classification lBag-of-Words :nouns, verbs, and adjectives lTypical example

    of ambiguous expression listed in Wikipedia (27 phrases) – "It is said that ..." – "An expert argues that ..." and so on 19
  15. 21.

    Improvements for better classification (1/2) 21 We need to get

    more data from editing histories on Wikipedia Frequency of weasel annotation varies depending on topics on Wikipedia Topic categories of positive examples may be biased Improvements in dataset
  16. 22.

    Improvements for better classification (2/2) 22 Improvements in classification features

    Because bag-of-words is simple as a feature, features that consider surrounding sentences and word order are required.
  17. 24.

    How does our prototype affect users behaviors? 24 ? 8IBUEPFTPVSQSPUPUZQFBGGFDUVTFSˏTCFIBWJPST

    24 Can our prototype encourage users to browse webpages more carefully? If so, how do user behaviors change? Q.
  18. 25.

    Hypothesis lH1 The proposed system extends the time expended in

    web information seeking. lH2 The proposed system increases the number of visited webpages. lH3 The proposed system improves the users’ confidence in their decisions made through web information seeking. lH4 The above-mentioned effects vary with users’ familiarity with the search topics. 25
  19. 26.

    User study 26 Participants were asked to search for webpages

    to answer medical questions. • Fixed search results: Participants sought 100 fixed search results • 4 search tasks: We prepared 4 search tasks about medical topics. “Is cinnamon effective for diabetes? Report your answer by Web search ”
  20. 27.

    Participant The participants were recruited by crowdsourcing and randomly assigned

    to two groups by UI condition lProposed group(105 participants ) – The system highlighted the weasel sentences while viewing the webpages – Authors manually decided which sentence should be highlighted as weasel sentences – Participants learnt what highlighted sentences meant before the task starts lControlled group(83 participants ) –Highlighting function was disabled 27
  21. 28.

    Focused behaviors Measuring behavior on the web –Session time –SERP

    dwelling time –Average dwelling time on one webpage –Pageview (the number of visited webpages) 28
  22. 29.

    Questionnaire lPre-questionnaire(6 levels) – Degree of prior knowledge about search

    topic – Expected answer – Confidence about the answer lPost questionnaire(6 levels) – Answers for search task – Confidence about the answers 29
  23. 30.

    Analysis method lWe use Bayesian GLMM (generalized linear mixture model)

    to model user behaviors lFixed effects – UI condition – Topic familiarity – Interaction between UI condition & topic familiarity lRandom effects – User – Search topic 30
  24. 32.

    Session time If participants were more familiar with search topic,

    there was greater difference in session time due to UI conditions 32
  25. 35.

    Change of confidence 35 The interaction between UI and topic

    familiarity affected confidence change. If participants were not familiar with topics, our prototype changed prior confidence more significantly. If participants were familiar with topics, our prototype changed prior confidence less significantly.
  26. 36.

    Discussion lThe user study suggests that the proposed system can

    enhance user engagement in careful information seeking on the web – The system increased pageview and session time. lWe need to investigate how user feel and use highlighted sentences – we didn’t understand why participants with our prototype viewed more webpages and spent longer time in search session 36
  27. 37.

    Conclusion lDesigning a system to highlight weasel sentences – The

    classification performance of weasel sentences were generally good performance but needs more improvement lUser study to examine the system effect – The proposed system can enhance user engagement in critical information seeking on the web. lFuture works – Improvement of weasel classifier – Investigating weasel sentences on the Web 37