Recognizing Contextual Polarity (Wilson et. al., 2009)

An Exploration of Features for Phrase-Level Sentiment Analysis

Transcript

1. Recognizing Contextual Polarity: An Exploration of Features for Phrase-Level Sentiment Analysis Theresa Wilson, Janyce Wiebe, Paul Hoffmann Presented by Kalan MacRow 1
2. Introduction • Sentiment analysis is a type of subjectivity analysis • Focus on identifying positive/negative opinions, emotions • Identify angry messages, classify reviews, answer opinion questions, etc. 2
3. Introduction • Need to determine positive/negative opinions and sentiments • But also need to know when no opinion is expressed • “The elementary school was condemned...” 3
4. Introduction • Typically start with a lexicon of words annotated with polarity • reason, reasonable, trust, good, ... • condemned, pollute, fail, bad, ... 4
5. Introduction • Call these prior polarities • Can be manually compiled, or • learned automatically (various techniques) • But we also have contextual polarity... 5
6. “...the contextual polarity of the phrase in which a particular instance of a word appears may be quite different from the word’s prior polarity. Positive words are used in phrases expressing negative sentiments, or vice versa. Also, quite often words that are positive or negative out of context are neutral in context...” Wilson et. al., 2008 6
7. Introduction • Words have different senses (e.g., trust) • Polarity can be negated • Can be neutral • Often prior == contextual, but not always! 7
8. Introduction • Focus on contextual polarity • Annotate MPQA corpus with contextual polarities • Try to automatically distinguish between prior and contextual polarities 8
9. Overview • Polarity influencers • Corpora and annotations • Prior polarity lexicon • The Gold Standard • Prior polarity classifier • Features • Experiments 9
10. Polarity Influencers • Sentiment analysis is not simple • Negation can be local, longer distance dependency, or even intensify polarity • Word sense, syntactic role, diminishers (little) play into contextual polarity 10
11. Polarity Influencers • Domain/topic is also an influencer • A cool car vs. cool demeanor • fever might be neutral in a medical context • Perspective: USA failed to defeat terrorists... • These will be features in the experiments 11
12. Overview • Polarity influencers • Corpora and annotations • Prior polarity lexicon • The Gold Standard • Prior polarity classifier • Features • Experiments 12
13. Corpora & Annotations • Need a corpus annotated for sentiment expressions & contextual polarity • Add these to the Multi-perspective Question Answering Corpus! • Sentiments are a subset of already annotated private states 13
14. Corpora & Annotations • Direct subjective frames mark references to private states • Each direct subjective frame also has an expression intensity •Subjective expressions: direct subjective frames with non-neutral intensity •Sentiment expressions: subjective expressions tagged positive, negative or both 14
15. Corpus & Annotations Agreement = 90% K = 0.84 Wilson et. al., 2008 Agreement Study 15
16. Overview • Polarity influencers • Corpora and annotations • Prior polarity lexicon • The Gold Standard • Prior polarity classifier • Features • Experiments 16
17. Prior Polarity Lexicon • 8,000 single-word subjectivity clues • Based on the list by Riloff et. al. (2003) • Add some clues from thesaurus, General Inquirer list 17
18. Prior Polarity Lexicon • Need to tag the clues with prior polarity: positive, negative, both or neutral • Positive evokes something positive, both imparts both (bittersweet, brag, etc.) •Sentiment clues: clues marked as positive, negative or both. • Neutrals are good clues that a sentiment is being expressed (feel, look, think, deeply, etc.) 18
19. Overview • Polarity influencers • Corpora and annotations • Prior polarity lexicon • The Gold Standard • Prior polarity classifier • Features • Experiments 19
20. The Gold Standard • Goal is to classify contextual polarity of expressions that contain clues in the lexicon • Each clue instance is labelled individually because identifying expr. boundaries is hard • Define the gold-standard in terms of the manual annotations... (Definition Thereof) 20
21. The Gold Standard If a clue is... • Not in a subjective expression: neutral • In a mixture of positive and neutral expressions: positive • In a mixture of negative and neutral expressions: negative • Positive and negative expressions: both 21
22. Overview • Polarity influencers • Corpora and annotations • Prior polarity lexicon • The Gold Standard • Prior polarity classifier • Features • Experiments 22
23. Prior Polarity Classifier • How useful is a classifier based solely in prior polarities? • Not very: 48% accuracy • Most errors come from polar words appearing in neutral contexts • Very few errors from neutral words appearing with polarity 23
24. Prior Polarity Classifier • Try a 2-stage classification process • First: classify instances as polar or neutral • Second: classify polar instances as positive, negative or both 24
25. Wilson et. al., 2008 25
26. Overview • Polarity influencers • Corpora and annotations • Prior polarity lexicon • The Gold Standard • Prior polarity classifier • Features • Experiments 26
27. Features • Motivated by the literature (Polanyi and Zaenan, 2004) • Exploration of the contextual polarity annotations in the data • And other features found useful for recognizing subjective sentences 27
28. Features Neutral-Polar Classification Wilson et. al., 2008 28
29. Features Neutral-Polar Classification Dependency parse tree supports the modifies/modified by features Wilson et. al., 2008 29
30. Features Polarity Classification Wilson et. al., 2008 30
31. Overview • Polarity influencers • Corpora and annotations • Prior polarity lexicon • The Gold Standard • Prior polarity classifier • Features • Experiments 31
32. Experiments • Two major goals: • Evaluate the aforementioned features for recognizing contextual polarity • Investigate the importance of recognizing neutral instances of clues 32
33. Experiments • Evaluate features’ performance together and separately • Verify features are robust and performance is not an artifact of the algorithm used • Vary the learning algorithm used... 33
34. Experiments • BoosTexter, AdaBoost.MH (boosting) • Ripper (rule learning) • TiMBL, IB1 (memory-based) • SVM-light, SVM-multiclass (support vector) SVM-light for binary classification, multiclass for more than two classes 34
35. Experiments • 10-fold cross validation over 10,287 sentences • 494 MPQA corpus documents • Look at accuracy, precision, recall and F-measure 35
36. Experiments Neutral-Polar Classification Learning algorithm configurations based on exploration of development data Wilson et. al., 2008 36
37. Experiments Neutral-Polar Classification Wilson et. al., 2008 37
38. • Results suggest highest performance for neutral-polar classification requires a wide variety of features working together • Smaller feature sets did not produce significant improvements over the baseline Experiments Neutral-Polar Classification 38
39. • Two conditions: • Perfect neutral-polar recognition (according to gold standard) • Automatic neutral-polar recognition (best classifier from each algorithm) Experiments Polarity Classification 39
40. Experiments Polarity Classification 40
41. • Negative polarity is much easier to recognize! (at least in this corpus) • Negation features and polarity-modification features result in significant improvements when added, polarity shifters were weak • Word token actually has little effect Experiments Polarity Classification 41
42. • With automatic neutral-polar classification, results are much lower • Still get classifiers that outperform the baseline! • For most algorithms, classifier with all polarity features has highest accuracy Experiments Polarity Classification 42
43. Conclusions • One-step approaches can perform about as well (or better) than two-step • Negation features are the most critical for contextual polarity • Identifying when contextual polarity is neutral is key • Identifying more complex interdependencies between polarity clues will be useful 43
44. Discussion Q: While this method identifies polarity information, I am not sure whether it finds which aspect (e.g., lens, picture quality, battery) of an entity (camera) this polarity is related to? 44
45. Discussion Q: In section 5, when defining the gold standard class of a clue instance, if the instance appears in a mixture of negative (or positive) and neutral subjective expressions, its class is negative (or positive). I am thinking maybe a majority vote approach here would be better, especially for those extreme cases (a mixture of one negative and many neutral). 45
46. Discussion Q: In the two step method the phrases are classified first and then the polar phrases are disambiguated. Why are netural phrases not disambiguated? Can they not contain polar clues still? 46
47. Discussion Q: The system presented in this paper uses a two-stage classification scheme where sentences are first classified as neutral or polar and the second stage classifies polar sentences as positive, negative, both, or neutral. What are the main advantages of splitting classification into two stages like this? Are there any drawbacks? 47
48. Discussion Q: While finding the subjectivity clues, they only consider single-word clues. However, phrases can also contribute as subjectivity clues. Wouldn't it increase the system's accuracy by doing so? 48
49. Discussion Q: They mention that the perspective of a person expressing the sentiment is an important aspect of contextual polarity and yet they do not consider this fact in their work. No matter what the system says the polarity of a phrase or sentence is, it still depends on the how the person perceives it. So how does this system help then? “From the perspective of Israel, failed to defeat is negative. From the perspective of Hezbollah, failed to defeat is positive.” 49
50. Discussion Were there enough annotators/diversity? Why not use feature selection? Why are negative polarity clues easier to recognize? 50