Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Interpretable Representations for Affect and Semantics

Gerard de Melo
July 01, 2021
Technology
0
29

Interpretable Representations for Affect and Semantics

While NLP has advanced considerably in recent years, state-of-the-art models can no longer easily be inspected and understood directly. This talk presents a series of interpretable approaches for affect and semantics-related settings.

The first part of the talk will present interpretable vectors custom-tailored for emotions and their connections to words, fonts, and colors. Subsequently, I will discuss NLP approaches for emojis and their connection to emotion. The final part of the talk will consider structured representations to better model the structure of a sentence or other kinds of knowledge that may be useful in downstream applications.

Gerard de Melo

July 01, 2021
Tweet

More Decks by Gerard de Melo

See All by Gerard de Melo
Curriculum Prompt Learning with Self-Training for Abstractive Dialogue Summarization
gdm
0
76
Appearances Matter: AI Learning from Text and Images
gdm
0
74
Putting AI to Work – How AI is Disrupting the Workplace
gdm
1
170

Other Decks in Technology

See All in Technology
Race Conditions em Microsserviços: Desmistificando um Problema Comum em Arquiteturas Distribuídas
jordihofc
0
320
開幕LT
makamaka
0
360
ジョブキューシステムFireworqのアーキテクチャ設計と運用時のベストプラクティス
tarao
1
820
Agileを始める前に知っておきたい3つの真実
chiroruxx
7
1.1k
Federated Learning 連合学習
regonn
1
490
「はたらく」前に知るべきIT企業5つのヒミツ 〜プロが教えるプロダクト開発最前線〜 / geeksai_2023spring
visional_engineering_and_design
0
260
エンジニアのためのドキュメントライティング / Docs for Developers and Paragraph Writing
nttcom
3
540
計測できるレガシーさを捉え、コード改善に対処する / phperkaigi2023
blue_goheimochi
0
160
世界モデルによる4脚ロボットの歩行学習
robots
1
250
ChatGPTをプロダクトに入れる開発が"毎日がAfter GPT"だった件/ChatGPT LT Link and Motivation
lmi
1
3k
テスト技法を使ったテストケースの表現方法/How to express test cases using test techniques
shimashima35
0
290
SIGNATE開発失敗談
bonjiri_niwa
0
190

Featured

See All Featured
Debugging Ruby Performance
tmm1
67
11k
Art Directing for the Web. Five minutes with CSS Template Areas
malarkey
197
11k
Web Components: a chance to create the future
zenorocha
304
41k
Refactoring Trust on Your Teams (GOTO; Chicago 2020)
rmw
22
1.7k
Designing for humans not robots
tammielis
245
24k
ParisWeb 2013: Learning to Love: Crash Course in Emotional UX Design
dotmariusz
101
6.2k
Building an army of robots
kneath
300
41k
Navigating Team Friction
lara
177
12k
How GitHub (no longer) Works
holman
299
140k
Infographics Made Easy
chrislema
236
17k
JazzCon 2018 Closing Keynote - Leadership for the Reluctant Leader
reverentgeek
176
9.2k
Large-scale JavaScript Application Architecture
addyosmani
499
110k

Transcript

  1. Interpretable Representations
    for Affect and Semantics
    Gerard de Melo
    http://gerard.demelo.org
    Interpretable Representations
    for Affect and Semantics
    Gerard de Melo
    http://gerard.demelo.org

    View Slide

  2. Machine Learning
    Machine Learning
    Examples
    Negative
    Positive

    View Slide

  3. Machine Learning
    Machine Learning
    Classifier
    Model
    Model
    Examples
    Learning
    Learning
    Negative
    Positive

    View Slide

  4. Machine Learning
    Machine Learning
    Classifier
    Model
    Model
    Examples
    Learning
    Learning
    ???
    Negative
    Positive

    View Slide

  5. Machine Learning
    Machine Learning
    Probably
    Positive!
    Prediction
    Prediction
    Classifier
    Model
    Model
    Examples
    Learning
    Learning
    Negative
    Positive

    View Slide

  6. Machine Learning
    Machine Learning
    Probably
    Positive!
    Prediction
    Prediction
    Classifier
    Model
    Model
    Examples
    Learning
    Learning
    Negative
    Positive

    View Slide

  7. Probably
    Incorrect!
    Prediction
    Prediction
    Classifier
    Model
    Model
    Examples
    Learning
    Learning
    Negative
    Positive
    Machine Learning
    Machine Learning
    Image:
    Method Matters Blog

    View Slide

  8. Machine Learning
    Nowadays
    Machine Learning
    Nowadays
    Probably
    Positive!
    Classifier
    Model
    Model
    Examples
    Negative
    Positive
    Prediction
    Prediction
    Learning
    Learning
    Deep
    neural network
    Deep
    neural network

    View Slide

  9. AI as a Black Box
    AI as a Black Box
    Probably
    Positive!
    Prediction
    Prediction
    Classifier
    Model
    Model
    Labelled
    Examples
    Learning
    Learning
    Negative
    Positive

    View Slide

  10. Why Explainable AI (XAI)?
    Why Explainable AI (XAI)?
    Why Explainable AI (XAI)?
    Why Explainable AI (XAI)?
    https://www.idealrole.com/blog/cv-bias

    View Slide

  11. Why Explainable AI (XAI)?
    Why Explainable AI (XAI)?
    Why Explainable AI (XAI)?
    Why Explainable AI (XAI)?

    View Slide

  12. Public Domain Image from https://pixabay.com/en/book-text-read-paper-education-451067/
    Outline
    Outline
    Vectors for Sentiment/Emotion
    Vectors for Emojis
    Trees for Sentences
    Graphs for Knowledge

    View Slide

  13. Public Domain Image from https://pixabay.com/en/book-text-read-paper-education-451067/
    Outline
    Outline
    Vectors for Sentiment/Emotion
    Vectors for Emojis
    Trees for Sentences
    Graphs for Knowledge

    View Slide

  14. Modern AI
    Modern AI
    Probably
    Positive!
    Classifier
    Model
    Model
    Examples
    Negative
    Positive
    Prediction
    Prediction
    Learning
    Learning
    Deep
    neural network
    Deep
    neural network

    View Slide

  15. Modern AI
    Modern AI
    Probably
    Positive!
    Classifier
    Model
    Model
    Examples
    Negative
    Positive
    Prediction
    Prediction
    Learning
    Learning
    Deep
    neural network
    Deep
    neural network

    View Slide

  16. 0.00
    0.23
    0.03
    0.31
    0.01
    0.03
    0.91
    ...
    0.31
    0.50
    Embeddings
    Embeddings
    0.01
    0.23
    0.01
    0.29
    0.00
    0.03
    0.92
    ...
    0.30
    0.51
    “dry” “arid”
    ≈
    x
    x
    x x
    amazing
    awesome
    unbearable
    horrible
    x
    terrible
    xstupendous
    Vector space
    in which similar
    words have
    similar vectors
    Vector space
    in which similar
    words have
    similar vectors

    View Slide

  17. Question
    Question
    Question
    Question
    Image: https://www.synthesio.com/blog/social-sentiment-and-your-business/
    Can we have vector representations
    that are interpretable and
    provide detailed information about
    the sentiment/emotion of a word?
    Can we have vector representations
    that are interpretable and
    provide detailed information about
    the sentiment/emotion of a word?
    0.00
    0.23
    0.03
    0.31
    0.01
    0.03
    0.91
    ...
    0.31
    0.50
    0.01
    0.23
    0.01
    0.29
    0.00
    0.03
    0.92
    ...
    0.30
    0.51
    “stupendous” “amazing”
    ≈
    ??

    View Slide

  18. Multi-Dimensional
    Multi-Dimensional
    Sentiment Embeddings
    Sentiment Embeddings
    Multi-Dimensional
    Multi-Dimensional
    Sentiment Embeddings
    Sentiment Embeddings
    Xin Dong, Gerard de Melo. Cross-Lingual Propagation for Deep Sentiment Analysis. AAAI 2018
    Music
    Laptops
    Image: http://www.thewilliamnyc.com/william-gallery/
    0.97
    -0.30
    ...
    0.90
    0.94
    ...
    “hot” “amazing”

    View Slide

  19. Method: Data-Driven
    Method: Data-Driven
    Sentiment Embeddings
    Sentiment Embeddings
    Method: Data-Driven
    Method: Data-Driven
    Sentiment Embeddings
    Sentiment Embeddings
    Transfer Learning
    using Supervised Linear
    Models
    Train n linear models
    Xin Dong, Gerard de Melo. Cross-Lingual Propagation for Deep Sentiment Analysis. AAAI 2018

    View Slide

  20. Method: Data-Driven
    Method: Data-Driven
    Sentiment Embeddings
    Sentiment Embeddings
    Method: Data-Driven
    Method: Data-Driven
    Sentiment Embeddings
    Sentiment Embeddings
    Transfer Learning
    using Supervised Linear
    Models
    Train n linear models
    Xin Dong, Gerard de Melo. Cross-Lingual Propagation for Deep Sentiment Analysis. AAAI 2018
    f
    Feature Value
    “horrible” 1
    “great” 2
    “inside” 1
    “cat” 3
    ... ...

    View Slide

  21. Method: Data-Driven
    Method: Data-Driven
    Sentiment Embeddings
    Sentiment Embeddings
    Method: Data-Driven
    Method: Data-Driven
    Sentiment Embeddings
    Sentiment Embeddings
    Transfer Learning
    using Supervised Linear
    Models
    Train n linear models
    Xin Dong, Gerard de Melo. Cross-Lingual Propagation for Deep Sentiment Analysis. AAAI 2018
    f
    Feature Value Weight
    “horrible” 1 -5.2
    “great” 2 4.4
    “inside” 1 -0.1
    “cat” 3 1.2
    ... ... ...

    View Slide

  22. Method: Data-Driven
    Method: Data-Driven
    Sentiment Embeddings
    Sentiment Embeddings
    Method: Data-Driven
    Method: Data-Driven
    Sentiment Embeddings
    Sentiment Embeddings
    Xin Dong, Gerard de Melo. Cross-Lingual Propagation for Deep Sentiment Analysis. AAAI 2018
    Train n linear models
    Used 25 different domains from Amazon
    Books, Electronics, Movies, Kitchen, etc.
    Used 25 different domains from Amazon
    Books, Electronics, Movies, Kitchen, etc.

    View Slide

  23. Method: Data-Driven
    Method: Data-Driven
    Sentiment Embeddings
    Sentiment Embeddings
    Method: Data-Driven
    Method: Data-Driven
    Sentiment Embeddings
    Sentiment Embeddings
    Xin Dong, Gerard de Melo. Cross-Lingual Propagation for Deep Sentiment Analysis. AAAI 2018
    0.0
    0.8
    0.2
    ...
    –0.8
    0.1
    hot
    Train n linear models
    For each word,
    turn its linear
    coefficients across
    different n models
    into a single vector
    ...

    View Slide

  24. Emotion Analysis
    Emotion Analysis
    Emotion Analysis
    Emotion Analysis
    “What the hell is going on?!
    I've been waiting for 2 weeks now!!”
    “What the hell is going on?!
    I've been waiting for 2 weeks now!!”

    View Slide

  25. Emotion Models
    Emotion Models
    Emotion Models
    Emotion Models
    Image: Paul Ekman
    Ekman's
    6 Basic
    Emotion
    Ekman's
    6 Basic
    Emotion

    View Slide

  26. Emotion Models
    Emotion Models
    Emotion Models
    Emotion Models
    Image: https://commons.wikimedia.org/wiki/File:Plutchik-wheel.svg
    Plutchik's
    Wheel of Emotions
    Plutchik's
    Wheel of Emotions

    View Slide

  27. Even More Emotions
    Even More Emotions
    Even More Emotions
    Even More Emotions
    Cowen & Keltner
    (2017)
    Are
    8 basic
    emotions
    really
    sufficient?
    Are
    8 basic
    emotions
    really
    sufficient?

    View Slide

  28. AffectVec
    AffectVec
    AffectVec
    AffectVec
    https://www.flickr.com/photos/ill-padrino/6437837857/
    joy
    sadness
    anger
    guilt
    suspense
    0.5
    0.1
    0.4
    ...
    0.3
    0.1
    prank
    Shahab Raji, Gerard de Melo. What Sparks Joy: The AffectVec Emotion Database
    Matej Kren: Idiom. Prague Municipal Library Prank Image: CollegeHumor

    View Slide

  29. AffectVec
    AffectVec
    AffectVec
    AffectVec
    Shahab Raji, Gerard de Melo. What Sparks Joy: The AffectVec Emotion Database

    View Slide

  30. AffectVec
    AffectVec
    AffectVec
    AffectVec
    Shahab Raji, Gerard de Melo. What Sparks Joy: The AffectVec Emotion Database

    View Slide

  31. AffectVec
    AffectVec
    AffectVec
    AffectVec
    Download:
    http://emotionlexicon.org/
    Download:
    http://emotionlexicon.org/
    Shahab Raji, Gerard de Melo. What Sparks Joy: The AffectVec Emotion Database
    Also: Experiments on Unsupervised
    Document-Level Emotion Prediction

    View Slide

  32. Cross-Lingual
    Cross-Lingual
    Emotion Lexicon Induction
    Emotion Lexicon Induction
    Cross-Lingual
    Cross-Lingual
    Emotion Lexicon Induction
    Emotion Lexicon Induction
    Over 300 languages
    http://emotionlexicon.org/
    Over 300 languages
    http://emotionlexicon.org/
    Arun Ramachandran, Gerard de Melo. COLING 2020

    View Slide

  33. Application:
    Application:
    Fonts
    Fonts
    Application:
    Application:
    Fonts
    Fonts
    Tugba Kulahcioglu, Gerard de Melo. Predicting Semantic Signatures of Fonts
    Van Rompay and Pruyn (2011)
    Choice of font
    affects assumed
    brand credibility
    price expectations
    Choice of font
    affects assumed
    brand credibility
    price expectations

    View Slide

  34. Application:
    Application:
    Fonts
    Fonts
    Application:
    Application:
    Fonts
    Fonts
    Tugba Kulahcioglu, Gerard de Melo. Predicting Semantic Signatures of Fonts
    de Sousa et al. (2020). Journal of Sensory Studies
    Choice of font
    affects
    intent to purchase
    and assumed
    acidity, sweetness
    Choice of font
    affects
    intent to purchase
    and assumed
    acidity, sweetness

    View Slide

  35. Application:
    Application:
    Fonts
    Fonts
    Application:
    Application:
    Fonts
    Fonts
    Tugba Kulahcioglu, Gerard de Melo. Predicting Semantic Signatures of Fonts
    Shaikh (2007)
    O’Brien Louch & Stork (2014)
    Choice of font
    affects assumed
    attributes of
    people using them
    Choice of font
    affects assumed
    attributes of
    people using them

    View Slide

  36. Font Selection
    Font Selection
    Font Selection
    Font Selection
    Tugba Kulahcioglu, Gerard de Melo. Predicting Semantic Signatures of Fonts

    View Slide

  37. Learning Representations of Fonts
    Learning Representations of Fonts
    Learning Representations of Fonts
    Learning Representations of Fonts
    Tugba Kulahcioglu, Gerard de Melo. Predicting Semantic Signatures of Fonts

    View Slide

  38. Learning Representations of Fonts
    Learning Representations of Fonts
    Learning Representations of Fonts
    Learning Representations of Fonts
    Tugba Kulahcioglu, Gerard de Melo. Predicting Semantic Signatures of Fonts

    View Slide

  39. Learning Representations of Fonts
    Learning Representations of Fonts
    Learning Representations of Fonts
    Learning Representations of Fonts
    Tugba Kulahcioglu, Gerard de Melo. Predicting Semantic Signatures of Fonts
    fontjoy.com

    View Slide

  40. Learning Representations of Fonts
    Learning Representations of Fonts
    Learning Representations of Fonts
    Learning Representations of Fonts
    http://fontjoy.com/projector/
    Visual
    Representation Space
    Visual
    Representation Space

    View Slide

  41. Learning Representations of Fonts
    Learning Representations of Fonts
    Learning Representations of Fonts
    Learning Representations of Fonts
    Tugba Kulahcioglu, Gerard de Melo. Predicting Semantic Signatures of Fonts
    fontjoy.com
    Infer Attribute-Based
    Representations
    Infer Attribute-Based
    Representations

    View Slide

  42. Learning Representations of Fonts
    Learning Representations of Fonts
    Learning Representations of Fonts
    Learning Representations of Fonts
    Tugba Kulahcioglu, Gerard de Melo. Predicting Semantic Signatures of Fonts

    View Slide

  43. FontLex
    FontLex
    FontLex
    FontLex
    Tugba Kulahcioglu, Gerard de Melo. FontLex: A Typographical Lexicon based on Affective Associations
    Infer font–emotion mapping
    using seed data + word2vec
    Infer font–emotion mapping
    using seed data + word2vec

    View Slide

  44. FontLex
    FontLex
    FontLex
    FontLex
    via Sentiment/Emotions associations of words
    Tugba Kulahcioglu, Gerard de Melo. FontLex: A Typographical Lexicon based on Affective Associations

    View Slide

  45. FontLex:
    FontLex:
    Complex Emotions
    Complex Emotions
    FontLex:
    FontLex:
    Complex Emotions
    Complex Emotions
    Tugba Kulahcioglu, Gerard de Melo. Semantics-Aware Typographical Choices via Affective Associations
    Extension by using more fonts
    for FontLex data

    View Slide

  46. Emotions vs.
    Emotions vs.
    Font and Color
    Font and Color
    Emotions vs.
    Emotions vs.
    Font and Color
    Font and Color
    Tugba Kulahcioglu, Gerard de Melo. Paralinguistic Recommendations for Affective Word Clouds. Proc. IUI
    Fonts: best for serious, trustworthy, disturbing
    Color palettes: best for calm, negative, playful

    View Slide

  47. Emotions vs.
    Emotions vs.
    Font and Color
    Font and Color
    Emotions vs.
    Emotions vs.
    Font and Color
    Font and Color
    Tugba Kulahcioglu, Gerard de Melo. Paralinguistic Recommendations for Affective Word Clouds. Proc. IUI
    United Nations

    View Slide

  48. Emotions vs.
    Emotions vs.
    Font and Color
    Font and Color
    Emotions vs.
    Emotions vs.
    Font and Color
    Font and Color
    Tugba Kulahcioglu, Gerard de Melo. FontLex: A Typographical Lexicon based on Affective Associations /
    Tugba Kulahcioglu, Gerard de Melo. Paralinguistic Recommendations for Affective Word Clouds. Proc. IUI
    The Smurfs Scream

    View Slide

  49. Public Domain Image from https://pixabay.com/en/book-text-read-paper-education-451067/
    Outline
    Outline
    Vectors for Sentiment/Emotion
    Vectors for Emojis
    Trees for Sentences
    Graphs for Knowledge

    View Slide

  50. Emojis are ubiquitous
    Emojis are ubiquitous
    Emojis are ubiquitous
    Emojis are ubiquitous
    Image: https://www.theverge.com/2016/2/24/11105250/facebook-reactions-emoji-how-to
    Very prominent on social media,
    instant messaging
    email subject lines etc.
    Very prominent on social media,
    instant messaging
    email subject lines etc.
    Face with Tears of Joy:
    Oxford Dictionaries
    Word of the Year 2015
    Face with Tears of Joy:
    Oxford Dictionaries
    Word of the Year 2015

    View Slide

  51. Emojis vs. Emotions
    Emojis vs. Emotions
    Emojis vs. Emotions
    Emojis vs. Emotions
    Abu Awal Md Shoeb, Shahab Raji, Gerard de Melo. EmoTag – Towards an Emotion-Based Analysis of Emojis. Proc. RANLP
    Same parts of brain activated
    as when seeing a real face
    Same parts of brain activated
    as when seeing a real face
    “Emoji” is
    Japanese for
    “picture character”
    but clearly
    closely linked
    to emotions
    Image: from Petra Kralj Novak. Sentiment of Emojis

    View Slide

  52. Emoji-Based
    Emoji-Based
    Word Embeddings
    Word Embeddings
    Emoji-Based
    Emoji-Based
    Word Embeddings
    Word Embeddings
    Image: https://www.theverge.com/2016/2/24/11105250/facebook-reactions-emoji-how-to
    0.8
    0.4
    0.0
    ...
    0.0
    0.1
    prank
    Abu Awal Md Shoeb, Shahab Raji, Gerard de Melo. EmoTag – Towards an Emotion-Based Analysis of Emojis. Proc. RANLP

    View Slide

  53. Use Case: Deep Neural Network for
    Use Case: Deep Neural Network for
    Tweet Emotion Classification
    Tweet Emotion Classification
    Use Case: Deep Neural Network for
    Use Case: Deep Neural Network for
    Tweet Emotion Classification
    Tweet Emotion Classification
    Abu Awal Md Shoeb, Shahab Raji, Gerard de Melo. EmoTag – Towards an Emotion-Based Analysis of Emojis. Proc. RANLP
    Methods Anger Fear Joy Sadness Average Dim
    Interpretable
    Affective Tweets 0.65 0.66 0.60 0.69 0.65 n/a
    EmoTag 0.70 0.73 0.69 0.75 0.72 620
    Non-Interpretable
    Random Int. 0.68 0.72 0.66 0.73 0.70 300
    word2vec 0.70 0.72 0.67 0.75 0.71 300
    GloVe 0.70 0.73 0.68 0.76 0.72 300
    GloVe Twitter 0.72 0.74 0.68 0.76 0.73 200

    View Slide

  54. Use Case: Emoji Sentiment
    Use Case: Emoji Sentiment
    Use Case: Emoji Sentiment
    Use Case: Emoji Sentiment
    Abu Awal Md Shoeb, Shahab Raji, Gerard de Melo. EmoTag – Towards an Emotion-Based Analysis of Emojis. Proc. RANLP

    View Slide

  55. Dataset and Study:
    Dataset and Study:
    Emoji Emotions
    Emoji Emotions
    Dataset and Study:
    Dataset and Study:
    Emoji Emotions
    Emoji Emotions
    Abu Shoeb, Gerard de Melo. EmoTag1200 : Understanding the Association between Emojis and Emotions . EMNLP
    👍 😄 😻
    Abu Shoeb, Shahab Raji, Gerard de Melo. EmoTag – Towards an Emotion-Based Analysis of Emojis. Proc. RANLP
    Online:
    emoji.nlproc.org
    Online:
    emoji.nlproc.org

    View Slide

  56. Dataset and Study:
    Dataset and Study:
    Emoji Emotions
    Emoji Emotions
    Dataset and Study:
    Dataset and Study:
    Emoji Emotions
    Emoji Emotions
    Abu Shoeb, Gerard de Melo. EmoTag1200 : Understanding the Association between Emojis and Emotions . EMNLP
    👍 😄 😻
    Abu Shoeb, Shahab Raji, Gerard de Melo. EmoTag – Towards an Emotion-Based Analysis of Emojis. Proc. RANLP

    View Slide

  57. Emojis in NLP:
    Emojis in NLP:
    Sentiment Analysis
    Sentiment Analysis
    Emojis in NLP:
    Emojis in NLP:
    Sentiment Analysis
    Sentiment Analysis
    Abu Shoeb, Gerard de Melo. Assessing Emoji Use in Modern Text Processing Tools . ACL-IJCNLP 2021

    View Slide

  58. Emojis in NLP:
    Emojis in NLP:
    Tokenization
    Tokenization
    Emojis in NLP:
    Emojis in NLP:
    Tokenization
    Tokenization
    Abu Shoeb, Gerard de Melo. Assessing Emoji Use in Modern Text Processing Tools . ACL-IJCNLP 2021

    View Slide

  59. Complex Emojis
    Complex Emojis
    Complex Emojis
    Complex Emojis
    Abu Shoeb, Gerard de Melo. Assessing Emoji Use in Modern Text Processing Tools . ACL-IJCNLP 2021

    View Slide

  60. Emojis in NLP:
    Emojis in NLP:
    POS Tagging
    POS Tagging
    Emojis in NLP:
    Emojis in NLP:
    POS Tagging
    POS Tagging
    Abu Shoeb, Gerard de Melo. Assessing Emoji Use in Modern Text Processing Tools . ACL-IJCNLP 2021

    View Slide

  61. Emojis in NLP:
    Emojis in NLP:
    Dependency Parsing
    Dependency Parsing
    Emojis in NLP:
    Emojis in NLP:
    Dependency Parsing
    Dependency Parsing
    Abu Shoeb, Gerard de Melo. Assessing Emoji Use in Modern Text Processing Tools . ACL-IJCNLP 2021

    View Slide

  62. Public Domain Image from https://pixabay.com/en/book-text-read-paper-education-451067/
    Outline
    Outline
    Vectors for Sentiment/Emotion
    Vectors for Emojis
    Trees for Sentences
    Graphs for Knowledge

    View Slide

  63. What are they capturing?
    What are they capturing?
    What are they capturing?
    What are they capturing?
    Image: Adapted from http://www.cse.unt.edu/people/StudentNewsletters/2013_Apr_StudentEmailNewsletter.html
    you can't cram the meaning of a whole
    ***ing sentence into a single ***ing vector

    View Slide

  64. Evaluation via
    Evaluation via
    Probing
    Probing
    Evaluation via
    Evaluation via
    Probing
    Probing
    Also: Adi et al. ICLR 2016
    These test whether enough information is kept to
    learn something from 100,000 training examples.
    These test whether enough information is kept to
    learn something from 100,000 training examples.

    View Slide

  65. Our Approach:
    Our Approach:
    Inspect Proximity Structure
    Inspect Proximity Structure
    Our Approach:
    Our Approach:
    Inspect Proximity Structure
    Inspect Proximity Structure
    A person is slicing an onion.
    A person is cutting an onion.
    A person is not slicing an onion.
    S0
    S=
    S*
    sim(S0,S=)
    sim(S0,S=)
    Zhu, Li, de Melo. Exploring Semantic Properties of Sentence Embeddings. Proc. ACL 2018

    View Slide

  66. Our Approach:
    Our Approach:
    Inspect Proximity Structure
    Inspect Proximity Structure
    Our Approach:
    Our Approach:
    Inspect Proximity Structure
    Inspect Proximity Structure
    A person is slicing an onion.
    A person is cutting an onion.
    A person is not slicing an onion.
    S0
    S=
    S*
    sim(S0,S=)
    sim(S0,S=)
    sim(S0,S*)
    sim(S0,S*)
    sim(S0,S=) > sim(S0,S*) ?

    View Slide

  67. Negation Detection
    Negation Detection
    Negation Detection
    Negation Detection
    A person is slicing an onion.
    A person is cutting an onion.
    A person is not slicing an onion.
    S0
    S=
    S*
    Zhu, Li, de Melo. Exploring Semantic Properties of Sentence Embeddings. Proc. ACL 2018

    View Slide

  68. Negation Variant
    Negation Variant
    Negation Variant
    Negation Variant
    A man is not standing on his head
    under water.
    There is no man standing on his head
    under water.
    A man is standing on his head
    under water.
    S0
    S=
    S*
    (Negation)
    (Negated
    Existential)
    Zhu, Li, de Melo. Exploring Semantic Properties of Sentence Embeddings. Proc. ACL 2018

    View Slide

  69. Clause Relatedness
    Clause Relatedness
    Clause Relatedness
    Clause Relatedness
    Octel said the purchase was expected.
    The purchase was expected.
    Octel said the purchase was
    not expected.
    S0
    S=
    S*
    Clause Extraction (for suitable head verbs only)
    Zhu, Li, de Melo. Exploring Semantic Properties of Sentence Embeddings. Proc. ACL 2018

    View Slide

  70. Argument Sensitivity
    Argument Sensitivity
    Argument Sensitivity
    Argument Sensitivity
    Francesca teaches Adam to adjust
    the microphone on his stage.
    Adam is taught to adjust
    the microphone on his stage.
    Adam teaches Francesca to adjust
    the microphone on his stage.
    S0
    S=
    S*
    (Passive)
    (Argument
    Inversion)
    Zhu, Li, de Melo. Exploring Semantic Properties of Sentence Embeddings. Proc. ACL 2018

    View Slide

  71. Argument Sensitivity
    Argument Sensitivity
    Argument Sensitivity
    Argument Sensitivity
    Zhu, Li, de Melo. Exploring Semantic Properties of Sentence Embeddings. Proc. ACL 2018
    Who-did-what-
    to-whom not reflected
    in topology
    Who-did-what-
    to-whom not reflected
    in topology

    View Slide

  72. R2D2:
    R2D2:
    Differentiable Tree Language Model
    Differentiable Tree Language Model
    R2D2:
    R2D2:
    Differentiable Tree Language Model
    Differentiable Tree Language Model
    Hu et al. R2D2: Recursive Transformer based on Differentiable Tree for Interpretable Hierarchical Language Modeling

    View Slide

  73. R2D2:
    R2D2:
    Differentiable Tree Language Model
    Differentiable Tree Language Model
    R2D2:
    R2D2:
    Differentiable Tree Language Model
    Differentiable Tree Language Model
    Hu et al. R2D2: Recursive Transformer based on Differentiable Tree for Interpretable Hierarchical Language Modeling
    black
    cat
    black cat
    the
    black
    the black cat
    the
    Which composition
    makes more sense?
    Which composition
    makes more sense?

    View Slide

  74. R2D2:
    R2D2:
    Differentiable Tree Language Model
    Differentiable Tree Language Model
    R2D2:
    R2D2:
    Differentiable Tree Language Model
    Differentiable Tree Language Model
    Hu et al. R2D2: Recursive Transformer based on Differentiable Tree for Interpretable Hierarchical Language Modeling
    the
    black
    cat
    black cat
    jumped
    away
    the
    black cat
    jumped
    the
    black
    the black cat jumped away
    black
    cat
    jumped
    cat
    jumped
    away
    black
    cat
    cat
    jumped
    jumped
    away
    the
    black cat
    jumped
    away
    black
    cat
    black cat
    the
    black
    the black cat
    the

    View Slide

  75. R2D2:
    R2D2:
    Differentiable Tree Language Model
    Differentiable Tree Language Model
    R2D2:
    R2D2:
    Differentiable Tree Language Model
    Differentiable Tree Language Model
    Hu et al. R2D2: Recursive Transformer based on Differentiable Tree for Interpretable Hierarchical Language Modeling
    the
    black
    cat
    black cat
    jumped
    away
    the
    black cat
    jumped
    the
    black
    the black cat jumped away
    black
    cat
    jumped
    cat
    jumped
    away
    black
    cat
    cat
    jumped
    jumped
    away
    the
    black cat
    jumped
    away
    black
    cat
    black cat
    the
    black
    the black cat
    the

    View Slide

  76. R2D2:
    R2D2:
    Differentiable Tree Language Model
    Differentiable Tree Language Model
    R2D2:
    R2D2:
    Differentiable Tree Language Model
    Differentiable Tree Language Model
    Hu et al. R2D2: Recursive Transformer based on Differentiable Tree for Interpretable Hierarchical Language Modeling
    the
    black
    cat
    black cat
    jumped
    away
    the
    black cat
    jumped
    the
    black
    the black cat jumped away
    black
    cat
    jumped
    cat
    jumped
    away
    black
    cat
    cat
    jumped
    jumped
    away
    the
    black cat
    jumped
    away

    View Slide

  77. R2D2:
    R2D2:
    Differentiable Tree Language Model
    Differentiable Tree Language Model
    R2D2:
    R2D2:
    Differentiable Tree Language Model
    Differentiable Tree Language Model
    Hu et al. R2D2: Recursive Transformer based on Differentiable Tree for Interpretable Hierarchical Language Modeling
    the
    black
    cat
    black cat
    jumped
    away
    the
    black cat
    jumped
    the
    black
    the black cat jumped away
    black
    cat
    jumped
    cat
    jumped
    away
    black
    cat
    cat
    jumped
    jumped
    away
    the
    black cat
    jumped
    away

    View Slide

  78. R2D2:
    R2D2:
    Differentiable Tree Language Model
    Differentiable Tree Language Model
    R2D2:
    R2D2:
    Differentiable Tree Language Model
    Differentiable Tree Language Model
    Hu et al. R2D2: Recursive Transformer based on Differentiable Tree for Interpretable Hierarchical Language Modeling
    the
    black
    cat
    black cat
    jumped
    away
    the
    black cat
    jumped
    the
    black
    the black cat jumped away
    black
    cat
    jumped
    cat
    jumped
    away
    black
    cat
    cat
    jumped
    jumped
    away
    the
    black cat
    jumped
    away

    View Slide

  79. R2D2:
    R2D2:
    Differentiable Tree Language Model
    Differentiable Tree Language Model
    R2D2:
    R2D2:
    Differentiable Tree Language Model
    Differentiable Tree Language Model
    Hu et al. R2D2: Recursive Transformer based on Differentiable Tree for Interpretable Hierarchical Language Modeling
    the
    black
    cat
    black cat
    jumped
    away
    the
    black cat
    jumped
    the
    black
    the black cat jumped away
    black
    cat
    jumped
    cat
    jumped
    away
    black
    cat
    cat
    jumped
    jumped
    away
    the
    black cat
    jumped
    away

    View Slide

  80. R2D2:
    R2D2:
    Differentiable Tree Language Model
    Differentiable Tree Language Model
    R2D2:
    R2D2:
    Differentiable Tree Language Model
    Differentiable Tree Language Model
    Hu et al. R2D2: Recursive Transformer based on Differentiable Tree for Interpretable Hierarchical Language Modeling
    Local
    Probability
    Compositional
    Embedding
    For each split point k of T1,3

    View Slide

  81. R2D2:
    R2D2:
    Differentiable Tree Language Model
    Differentiable Tree Language Model
    R2D2:
    R2D2:
    Differentiable Tree Language Model
    Differentiable Tree Language Model
    Hu et al. R2D2: Recursive Transformer based on Differentiable Tree for Interpretable Hierarchical Language Modeling
    Local
    Probability
    Compositional
    Embedding
    For each split point k of T1,3
    Weighted sum over all split points

    View Slide

  82. R2D2:
    R2D2:
    Differentiable Tree Language Model
    Differentiable Tree Language Model
    R2D2:
    R2D2:
    Differentiable Tree Language Model
    Differentiable Tree Language Model
    Hu et al. R2D2: Recursive Transformer based on Differentiable Tree for Interpretable Hierarchical Language Modeling
    Unsupervised
    Language Model
    Pre-Training
    Unsupervised
    Language Model
    Pre-Training

    View Slide

  83. R2D2:
    R2D2:
    Differentiable Tree Language Model
    Differentiable Tree Language Model
    R2D2:
    R2D2:
    Differentiable Tree Language Model
    Differentiable Tree Language Model
    Hu et al. R2D2: Recursive Transformer based on Differentiable Tree for Interpretable Hierarchical Language Modeling

    View Slide

  84. Public Domain Image from https://pixabay.com/en/book-text-read-paper-education-451067/
    Outline
    Outline
    Vectors for Sentiment/Emotion
    Vectors for Emojis
    Trees for Sentences
    Graphs for Knowledge

    View Slide

  85. Universal Wordnet
    Universal Wordnet
    Universal Wordnet
    Universal Wordnet
    cmn: “ ”
    机构
    http://lexvo.org/uwn/
    Gerard de Melo. Towards a Universal Wordnet by Learning from Combined Evidence.

    View Slide

  86. Explainable Inference
    in Graphs
    Explainable Inference
    in Graphs
    Fu et al. Fairness-Aware Explainable Recommendation over Knowledge Graphs. In: Proceedings of SIGIR 2020.
    http://gerard.demelo.org/publications.html

    View Slide

  87. Explainable Inference
    in Graphs
    Explainable Inference
    in Graphs
    Fu et al. Fairness-Aware Explainable Recommendation over Knowledge Graphs. In: Proceedings of SIGIR 2020.
    http://gerard.demelo.org/publications.html

    View Slide

  88. Explainable Inference
    in Graphs
    Explainable Inference
    in Graphs
    Fu et al. Fairness-Aware Explainable Recommendation over Knowledge Graphs. In: Proceedings of SIGIR 2020.
    http://gerard.demelo.org/publications.html
    In reality, there are
    many possible paths.
    Which to choose?
    In reality, there are
    many possible paths.
    Which to choose?

    View Slide

  89. Explainable Inference
    in Graphs
    Explainable Inference
    in Graphs
    Fu et al. Fairness-Aware Explainable Recommendation over Knowledge Graphs. In: Proceedings of SIGIR 2020.
    http://gerard.demelo.org/publications.html
    Reward if you guessed
    correctly!
    Reward if you guessed
    correctly!

    View Slide

  90. Explainable Inference
    in Graphs
    Explainable Inference
    in Graphs
    Xian et al. Reinforcement Knowledge Graph Reasoning for Explainable Recommendation. SIGIR 2019

    View Slide

  91. x
    x
    감사합니다
    gracias
    xthank you
    Summary
    Summary
    Summary
    Summary
    Vectors for Sentiment
    and Emotion
    ►Interpretable Vectors
    ►Font Affect Embeddings
    Emojis
    ►Interpretable Vectors
    ►NLP Analysis
    Trees for Sentences
    ►Analysis
    ►R2D2 Differentiable Tree Model
    Graphs for Knowledge
    ►Reinforcement Learning over
    Knowledge Graphs for
    Explainable AI
    Get in Touch!
    http://gerard.demelo.org
    [email protected]
    Get in Touch!
    http://gerard.demelo.org
    [email protected]
    x 谢谢
    x
    Vielen Dank
    спаси́бо
    x
    x obrigado

    View Slide