The Humane Machine: Building AI systems with Emotional Intelligence

AI based systems have become more and more a part of our everyday lives in the last few years. We now interact routinely with AI based systems like Siri and Alexa. However most such systems today are designed to be functional with little attention paid to the emotional context of these interactions. This is the space of the emerging science of Affective Computing, sometimes referred to as "Emotional AI".

In this session we'll look at the state of the Artificial Emotional Intelligence space as well as the components of modern AEI systems especially in the domain of digital conversational agents. We'll also look at how humans react to such systems and conclude with the risks and rewards of building systems that are capable of understanding and simulating emotions.

Transcript

1. CONFIDENTIAL
   The Humane
   Machine
   Building AI systems with emotional intelligence
   Arpit Mathur

   Principal Engineer, Comcast Labs

   

   View Slide

2. https://en.wikipedia.org/wiki/File:NASA_Mars_Rover.jpg
   

   View Slide

3. “The fact Mars rovers last message before it went silent
   is “my battery is low and its getting dark” literally has
   me in tears like it probably died out there in mars alone
   and afraid hoping nasa would save it”
   Last message was, "my battery is low and its getting
   dark out”. Why tf am i crying in the club over a robot
   Farewell, tiny Martian robot car. One day you’ll be found by our
   descendants, a monument to our primitive 21st century
   technology.
   Y’all I’m tearing up over a robot help me. That “my battery is low and
   its getting dark” got me. You did great, sweetie. You did so great.
   Brb crying over a space robot @NASA #Oppy #MarsRover
   "My battery is low and it's getting dark" is perhaps the saddest
   thing I can imagine a little robot saying. And, thanks to my dumb
   human pack bonding instincts, I can think of robots like little pets
   so I'm crying. Goodnight, #Oppy
   

   View Slide

4. We are wired for emotional connection
   

   View Slide

5. Scenario
   • User is frustrated with a lag in the
   application/game he is playing. Do you
   
   • Ignore user’s state?
   
   • Let the user vent?
   
   • Acknowledge the user’s frustration?
   

   View Slide

6. Scenario
   • User is frustrated with a lag in the
   application/game he is playing. Do you
   
   • Ignore user’s state?
   
   • Let the user vent?
   
   • Acknowledge the user’s frustration?
   Source: 

   This computer responds to user frustration 

   J. Klein ; Y. Moon ; R.W. Picard
   

   View Slide

7. Affective Computing
   The study and development of systems and devices that can
   recognize, process, and simulate human affects
   

   View Slide

8. 
   Sentiment Emotion
   Emotion vs Sentiment
   

   View Slide

9. Recognition
   

   View Slide

10. Modalities
    Text
    Voice
    Facial Expressions
    EEG
    Heart Rate
    Skin Conductance
    

    View Slide

11. Linguistic Inquiry and Word Count (LIWC)
    • Text Analysis module + Internal Dictionaries
    
    • Main dictionary composed of almost 6,400
    words, word stems, and selected emoticons
    
    • Each dictionary entry maps to multiple
    categories
    
    • Categories are arranged hierarchically
    
    • 90 different output dimensions
    “I love this restaurant. Best sushi in
    town. And a BYOB for extra win! ”
    Source: http://liwc.wpengine.com
    

    View Slide

12. Face Emotion Recognition
    • DIY
    
    • Source publicly available image datasets
    ( Microsoft FER+ is a great fit )
    
    • EmoPy or OpenCV or just Github search
    
    • APIs
    
    • Microsoft Azure, Amazon Recognition,
    Google
    Source: Microsoft FER+
    

    View Slide

13. Processing
    

    View Slide

14. Circumplex Model of Emotion
    

    View Slide

15. Plutchik’s wheel of Emotions
    

    View Slide

16. MFOEM Ontology
    An Emotion Ontology to describe
    affective phenomena, as a branch of the
    broader Mental Functioning ontology
    effort developed by The Swiss Centre
    for Affective Sciences, in collaboration
    with the University at Buffalo
    https://github.com/jannahastings/emotion-ontology
    

    View Slide

17. Emotion Understanding as a Service
    IBM Tone Analyzer (text) Microsoft (vision) Amazon Rekognition Emoshape EPU Voicery generation
    Anger TRUE TRUE TRUE TRUE TRUE
    Fear TRUE TRUE
    Joy / Happy TRUE TRUE TRUE TRUE TRUE
    Sadness TRUE TRUE TRUE TRUE TRUE
    Analytical TRUE
    Confident TRUE TRUE
    Tentative TRUE
    Surprized TRUE TRUE
    Calm TRUE TRUE
    Conversational
    Flirty
    Flustered
    Scared TRUE TRUE
    Disgust TRUE
    Indifference TRUE
    Regret TRUE
    Anticipation TRUE
    Trust TRUE
    Desire TRUE
    Contempt TRUE
    

    View Slide

18. Simulation
    

    View Slide

19. Affective Text
    Cake Chat (Replica.ai) E.L.S.A (MIT Media Lab)
    

    View Slide

20. Microsoft Xioice
    Source: 

    The Design and Implementation of XiaoIce, an Empathetic Social Chatbot 

    Li Zhou, Jianfeng Gao, Di Li, Heung-Yeung Shum
    

    View Slide

21. Affective Speech
    • Gender
    
    • Location (Accents)
    
    • Prosody
    
    • Wavenet
    
    • Tacotron
    
    • DeepVoice
    
    • Personality
    
    • Voicery
    Source: Google AI Blog

    Expressive Speech Synthesis with Tacotron
    

    View Slide

22. Parting thoughts:

    Designing Affective Experiences Responsibly
    

    View Slide

23. References
    • Affective Computing Group @ MIT Media Labs
    
    • Swiss Centre for Affective Sciences
    
    • https://en.wikipedia.org/wiki/Affective_computing
    
    • Replika.ai (Research Github)
    
    • Why we are Wired to Connect (Scientific American)
    
    • EmoPy: A machine learning toolkit for emotion expression
    

    View Slide