Upgrade to Pro — share decks privately, control downloads, hide ads and more …

The Humane Machine: Building AI systems with Emotional Intelligence

The Humane Machine: Building AI systems with Emotional Intelligence

AI based systems have become more and more a part of our everyday lives in the last few years. We now interact routinely with AI based systems like Siri and Alexa. However most such systems today are designed to be functional with little attention paid to the emotional context of these interactions. This is the space of the emerging science of Affective Computing, sometimes referred to as "Emotional AI".

In this session we'll look at the state of the Artificial Emotional Intelligence space as well as the components of modern AEI systems especially in the domain of digital conversational agents. We'll also look at how humans react to such systems and conclude with the risks and rewards of building systems that are capable of understanding and simulating emotions.

Arpit Mathur

August 18, 2019
Tweet

More Decks by Arpit Mathur

Other Decks in Technology

Transcript

  1. CONFIDENTIAL
    The Humane
    Machine
    Building AI systems with emotional intelligence
    Arpit Mathur

    Principal Engineer, Comcast Labs


    View Slide

  2. https://en.wikipedia.org/wiki/File:NASA_Mars_Rover.jpg

    View Slide

  3. “The fact Mars rovers last message before it went silent
    is “my battery is low and its getting dark” literally has
    me in tears like it probably died out there in mars alone
    and afraid hoping nasa would save it”
    Last message was, "my battery is low and its getting
    dark out”. Why tf am i crying in the club over a robot
    Farewell, tiny Martian robot car. One day you’ll be found by our
    descendants, a monument to our primitive 21st century
    technology.
    Y’all I’m tearing up over a robot help me. That “my battery is low and
    its getting dark” got me. You did great, sweetie. You did so great.
    Brb crying over a space robot @NASA #Oppy #MarsRover
    "My battery is low and it's getting dark" is perhaps the saddest
    thing I can imagine a little robot saying. And, thanks to my dumb
    human pack bonding instincts, I can think of robots like little pets
    so I'm crying. Goodnight, #Oppy

    View Slide

  4. We are wired for emotional connection

    View Slide

  5. Scenario
    • User is frustrated with a lag in the
    application/game he is playing. Do you

    • Ignore user’s state?

    • Let the user vent?

    • Acknowledge the user’s frustration?

    View Slide

  6. Scenario
    • User is frustrated with a lag in the
    application/game he is playing. Do you

    • Ignore user’s state?

    • Let the user vent?

    • Acknowledge the user’s frustration?
    Source: 

    This computer responds to user frustration 

    J. Klein ; Y. Moon ; R.W. Picard

    View Slide

  7. Affective Computing
    The study and development of systems and devices that can
    recognize, process, and simulate human affects

    View Slide


  8. Sentiment Emotion
    Emotion vs Sentiment

    View Slide

  9. Recognition

    View Slide

  10. Modalities
    Text
    Voice
    Facial Expressions
    EEG
    Heart Rate
    Skin Conductance

    View Slide

  11. Linguistic Inquiry and Word Count (LIWC)
    • Text Analysis module + Internal Dictionaries

    • Main dictionary composed of almost 6,400
    words, word stems, and selected emoticons

    • Each dictionary entry maps to multiple
    categories

    • Categories are arranged hierarchically

    • 90 different output dimensions
    “I love this restaurant. Best sushi in
    town. And a BYOB for extra win! ”
    Source: http://liwc.wpengine.com

    View Slide

  12. Face Emotion Recognition
    • DIY

    • Source publicly available image datasets
    ( Microsoft FER+ is a great fit )

    • EmoPy or OpenCV or just Github search

    • APIs

    • Microsoft Azure, Amazon Recognition,
    Google
    Source: Microsoft FER+

    View Slide

  13. Processing

    View Slide

  14. Circumplex Model of Emotion

    View Slide

  15. Plutchik’s wheel of Emotions

    View Slide

  16. MFOEM Ontology
    An Emotion Ontology to describe
    affective phenomena, as a branch of the
    broader Mental Functioning ontology
    effort developed by The Swiss Centre
    for Affective Sciences, in collaboration
    with the University at Buffalo
    https://github.com/jannahastings/emotion-ontology

    View Slide

  17. Emotion Understanding as a Service
    IBM Tone Analyzer (text) Microsoft (vision) Amazon Rekognition Emoshape EPU Voicery generation
    Anger TRUE TRUE TRUE TRUE TRUE
    Fear TRUE TRUE
    Joy / Happy TRUE TRUE TRUE TRUE TRUE
    Sadness TRUE TRUE TRUE TRUE TRUE
    Analytical TRUE
    Confident TRUE TRUE
    Tentative TRUE
    Surprized TRUE TRUE
    Calm TRUE TRUE
    Conversational
    Flirty
    Flustered
    Scared TRUE TRUE
    Disgust TRUE
    Indifference TRUE
    Regret TRUE
    Anticipation TRUE
    Trust TRUE
    Desire TRUE
    Contempt TRUE

    View Slide

  18. Simulation

    View Slide

  19. Affective Text
    Cake Chat (Replica.ai) E.L.S.A (MIT Media Lab)

    View Slide

  20. Microsoft Xioice
    Source: 

    The Design and Implementation of XiaoIce, an Empathetic Social Chatbot 

    Li Zhou, Jianfeng Gao, Di Li, Heung-Yeung Shum

    View Slide

  21. Affective Speech
    • Gender

    • Location (Accents)

    • Prosody

    • Wavenet

    • Tacotron

    • DeepVoice

    • Personality

    • Voicery
    Source: Google AI Blog

    Expressive Speech Synthesis with Tacotron

    View Slide

  22. Parting thoughts:

    Designing Affective Experiences Responsibly

    View Slide

  23. References
    • Affective Computing Group @ MIT Media Labs

    • Swiss Centre for Affective Sciences

    • https://en.wikipedia.org/wiki/Affective_computing

    • Replika.ai (Research Github)

    • Why we are Wired to Connect (Scientific American)

    • EmoPy: A machine learning toolkit for emotion expression

    View Slide