Recognizing Contextual Polarity (Wilson et. al., 2009)

Recognizing Contextual Polarity (Wilson et. al., 2009)

An Exploration of Features for Phrase-Level Sentiment Analysis

3c02038387302217d6b670ba6dafaecf?s=128

Kalan MacRow

March 22, 2013
Tweet

Transcript

  1. Recognizing Contextual Polarity: An Exploration of Features for Phrase-Level Sentiment

    Analysis Theresa Wilson, Janyce Wiebe, Paul Hoffmann Presented by Kalan MacRow 1
  2. Introduction • Sentiment analysis is a type of subjectivity analysis

    • Focus on identifying positive/negative opinions, emotions • Identify angry messages, classify reviews, answer opinion questions, etc. 2
  3. Introduction • Need to determine positive/negative opinions and sentiments •

    But also need to know when no opinion is expressed • “The elementary school was condemned...” 3
  4. Introduction • Typically start with a lexicon of words annotated

    with polarity • reason, reasonable, trust, good, ... • condemned, pollute, fail, bad, ... 4
  5. Introduction • Call these prior polarities • Can be manually

    compiled, or • learned automatically (various techniques) • But we also have contextual polarity... 5
  6. “...the contextual polarity of the phrase in which a particular

    instance of a word appears may be quite different from the word’s prior polarity. Positive words are used in phrases expressing negative sentiments, or vice versa. Also, quite often words that are positive or negative out of context are neutral in context...” Wilson et. al., 2008 6
  7. Introduction • Words have different senses (e.g., trust) • Polarity

    can be negated • Can be neutral • Often prior == contextual, but not always! 7
  8. Introduction • Focus on contextual polarity • Annotate MPQA corpus

    with contextual polarities • Try to automatically distinguish between prior and contextual polarities 8
  9. Overview • Polarity influencers • Corpora and annotations • Prior

    polarity lexicon • The Gold Standard • Prior polarity classifier • Features • Experiments 9
  10. Polarity Influencers • Sentiment analysis is not simple • Negation

    can be local, longer distance dependency, or even intensify polarity • Word sense, syntactic role, diminishers (little) play into contextual polarity 10
  11. Polarity Influencers • Domain/topic is also an influencer • A

    cool car vs. cool demeanor • fever might be neutral in a medical context • Perspective: USA failed to defeat terrorists... • These will be features in the experiments 11
  12. Overview • Polarity influencers • Corpora and annotations • Prior

    polarity lexicon • The Gold Standard • Prior polarity classifier • Features • Experiments 12
  13. Corpora & Annotations • Need a corpus annotated for sentiment

    expressions & contextual polarity • Add these to the Multi-perspective Question Answering Corpus! • Sentiments are a subset of already annotated private states 13
  14. Corpora & Annotations • Direct subjective frames mark references to

    private states • Each direct subjective frame also has an expression intensity •Subjective expressions: direct subjective frames with non-neutral intensity •Sentiment expressions: subjective expressions tagged positive, negative or both 14
  15. Corpus & Annotations Agreement = 90% K = 0.84 Wilson

    et. al., 2008 Agreement Study 15
  16. Overview • Polarity influencers • Corpora and annotations • Prior

    polarity lexicon • The Gold Standard • Prior polarity classifier • Features • Experiments 16
  17. Prior Polarity Lexicon • 8,000 single-word subjectivity clues • Based

    on the list by Riloff et. al. (2003) • Add some clues from thesaurus, General Inquirer list 17
  18. Prior Polarity Lexicon • Need to tag the clues with

    prior polarity: positive, negative, both or neutral • Positive evokes something positive, both imparts both (bittersweet, brag, etc.) •Sentiment clues: clues marked as positive, negative or both. • Neutrals are good clues that a sentiment is being expressed (feel, look, think, deeply, etc.) 18
  19. Overview • Polarity influencers • Corpora and annotations • Prior

    polarity lexicon • The Gold Standard • Prior polarity classifier • Features • Experiments 19
  20. The Gold Standard • Goal is to classify contextual polarity

    of expressions that contain clues in the lexicon • Each clue instance is labelled individually because identifying expr. boundaries is hard • Define the gold-standard in terms of the manual annotations... (Definition Thereof) 20
  21. The Gold Standard If a clue is... • Not in

    a subjective expression: neutral • In a mixture of positive and neutral expressions: positive • In a mixture of negative and neutral expressions: negative • Positive and negative expressions: both 21
  22. Overview • Polarity influencers • Corpora and annotations • Prior

    polarity lexicon • The Gold Standard • Prior polarity classifier • Features • Experiments 22
  23. Prior Polarity Classifier • How useful is a classifier based

    solely in prior polarities? • Not very: 48% accuracy • Most errors come from polar words appearing in neutral contexts • Very few errors from neutral words appearing with polarity 23
  24. Prior Polarity Classifier • Try a 2-stage classification process •

    First: classify instances as polar or neutral • Second: classify polar instances as positive, negative or both 24
  25. Wilson et. al., 2008 25

  26. Overview • Polarity influencers • Corpora and annotations • Prior

    polarity lexicon • The Gold Standard • Prior polarity classifier • Features • Experiments 26
  27. Features • Motivated by the literature (Polanyi and Zaenan, 2004)

    • Exploration of the contextual polarity annotations in the data • And other features found useful for recognizing subjective sentences 27
  28. Features Neutral-Polar Classification Wilson et. al., 2008 28

  29. Features Neutral-Polar Classification Dependency parse tree supports the modifies/modified by

    features Wilson et. al., 2008 29
  30. Features Polarity Classification Wilson et. al., 2008 30

  31. Overview • Polarity influencers • Corpora and annotations • Prior

    polarity lexicon • The Gold Standard • Prior polarity classifier • Features • Experiments 31
  32. Experiments • Two major goals: • Evaluate the aforementioned features

    for recognizing contextual polarity • Investigate the importance of recognizing neutral instances of clues 32
  33. Experiments • Evaluate features’ performance together and separately • Verify

    features are robust and performance is not an artifact of the algorithm used • Vary the learning algorithm used... 33
  34. Experiments • BoosTexter, AdaBoost.MH (boosting) • Ripper (rule learning) •

    TiMBL, IB1 (memory-based) • SVM-light, SVM-multiclass (support vector) SVM-light for binary classification, multiclass for more than two classes 34
  35. Experiments • 10-fold cross validation over 10,287 sentences • 494

    MPQA corpus documents • Look at accuracy, precision, recall and F-measure 35
  36. Experiments Neutral-Polar Classification Learning algorithm configurations based on exploration of

    development data Wilson et. al., 2008 36
  37. Experiments Neutral-Polar Classification Wilson et. al., 2008 37

  38. • Results suggest highest performance for neutral-polar classification requires a

    wide variety of features working together • Smaller feature sets did not produce significant improvements over the baseline Experiments Neutral-Polar Classification 38
  39. • Two conditions: • Perfect neutral-polar recognition (according to gold

    standard) • Automatic neutral-polar recognition (best classifier from each algorithm) Experiments Polarity Classification 39
  40. Experiments Polarity Classification 40

  41. • Negative polarity is much easier to recognize! (at least

    in this corpus) • Negation features and polarity-modification features result in significant improvements when added, polarity shifters were weak • Word token actually has little effect Experiments Polarity Classification 41
  42. • With automatic neutral-polar classification, results are much lower •

    Still get classifiers that outperform the baseline! • For most algorithms, classifier with all polarity features has highest accuracy Experiments Polarity Classification 42
  43. Conclusions • One-step approaches can perform about as well (or

    better) than two-step • Negation features are the most critical for contextual polarity • Identifying when contextual polarity is neutral is key • Identifying more complex interdependencies between polarity clues will be useful 43
  44. Discussion Q: While this method identifies polarity information, I am

    not sure whether it finds which aspect (e.g., lens, picture quality, battery) of an entity (camera) this polarity is related to? 44
  45. Discussion Q: In section 5, when defining the gold standard

    class of a clue instance, if the instance appears in a mixture of negative (or positive) and neutral subjective expressions, its class is negative (or positive). I am thinking maybe a majority vote approach here would be better, especially for those extreme cases (a mixture of one negative and many neutral). 45
  46. Discussion Q: In the two step method the phrases are

    classified first and then the polar phrases are disambiguated. Why are netural phrases not disambiguated? Can they not contain polar clues still? 46
  47. Discussion Q: The system presented in this paper uses a

    two-stage classification scheme where sentences are first classified as neutral or polar and the second stage classifies polar sentences as positive, negative, both, or neutral. What are the main advantages of splitting classification into two stages like this? Are there any drawbacks? 47
  48. Discussion Q: While finding the subjectivity clues, they only consider

    single-word clues. However, phrases can also contribute as subjectivity clues. Wouldn't it increase the system's accuracy by doing so? 48
  49. Discussion Q: They mention that the perspective of a person

    expressing the sentiment is an important aspect of contextual polarity and yet they do not consider this fact in their work. No matter what the system says the polarity of a phrase or sentence is, it still depends on the how the person perceives it. So how does this system help then? “From the perspective of Israel, failed to defeat is negative. From the perspective of Hezbollah, failed to defeat is positive.” 49
  50. Discussion Were there enough annotators/diversity? Why not use feature selection?

    Why are negative polarity clues easier to recognize? 50