Slide 1

Slide 1 text

CONFIDENTIAL The Humane Machine Building AI systems with emotional intelligence Arpit Mathur
 Principal Engineer, Comcast Labs

Slide 2

Slide 2 text

Slide 3

Slide 3 text

“The fact Mars rovers last message before it went silent is “my battery is low and its getting dark” literally has me in tears like it probably died out there in mars alone and afraid hoping nasa would save it” Last message was, "my battery is low and its getting dark out”. Why tf am i crying in the club over a robot Farewell, tiny Martian robot car. One day you’ll be found by our descendants, a monument to our primitive 21st century technology. Y’all I’m tearing up over a robot help me. That “my battery is low and its getting dark” got me. You did great, sweetie. You did so great. Brb crying over a space robot @NASA #Oppy #MarsRover "My battery is low and it's getting dark" is perhaps the saddest thing I can imagine a little robot saying. And, thanks to my dumb human pack bonding instincts, I can think of robots like little pets so I'm crying. Goodnight, #Oppy

Slide 4

Slide 4 text

We are wired for emotional connection

Slide 5

Slide 5 text

Scenario • User is frustrated with a lag in the application/game he is playing. Do you • Ignore user’s state? • Let the user vent? • Acknowledge the user’s frustration?

Slide 6

Slide 6 text

Scenario • User is frustrated with a lag in the application/game he is playing. Do you • Ignore user’s state? • Let the user vent? • Acknowledge the user’s frustration? Source: 
 This computer responds to user frustration 
 J. Klein ; Y. Moon ; R.W. Picard

Slide 7

Slide 7 text

Affective Computing The study and development of systems and devices that can recognize, process, and simulate human affects

Slide 8

Slide 8 text

Sentiment Emotion Emotion vs Sentiment

Slide 9

Slide 9 text


Slide 10

Slide 10 text

Modalities Text Voice Facial Expressions EEG Heart Rate Skin Conductance

Slide 11

Slide 11 text

Linguistic Inquiry and Word Count (LIWC) • Text Analysis module + Internal Dictionaries • Main dictionary composed of almost 6,400 words, word stems, and selected emoticons • Each dictionary entry maps to multiple categories • Categories are arranged hierarchically • 90 different output dimensions “I love this restaurant. Best sushi in town. And a BYOB for extra win! ” Source:

Slide 12

Slide 12 text

Face Emotion Recognition • DIY • Source publicly available image datasets ( Microsoft FER+ is a great fit ) • EmoPy or OpenCV or just Github search • APIs • Microsoft Azure, Amazon Recognition, Google Source: Microsoft FER+

Slide 13

Slide 13 text


Slide 14

Slide 14 text

Circumplex Model of Emotion

Slide 15

Slide 15 text

Plutchik’s wheel of Emotions

Slide 16

Slide 16 text

MFOEM Ontology An Emotion Ontology to describe affective phenomena, as a branch of the broader Mental Functioning ontology effort developed by The Swiss Centre for Affective Sciences, in collaboration with the University at Buffalo

Slide 17

Slide 17 text

Emotion Understanding as a Service IBM Tone Analyzer (text) Microsoft (vision) Amazon Rekognition Emoshape EPU Voicery generation Anger TRUE TRUE TRUE TRUE TRUE Fear TRUE TRUE Joy / Happy TRUE TRUE TRUE TRUE TRUE Sadness TRUE TRUE TRUE TRUE TRUE Analytical TRUE Confident TRUE TRUE Tentative TRUE Surprized TRUE TRUE Calm TRUE TRUE Conversational Flirty Flustered Scared TRUE TRUE Disgust TRUE Indifference TRUE Regret TRUE Anticipation TRUE Trust TRUE Desire TRUE Contempt TRUE

Slide 18

Slide 18 text


Slide 19

Slide 19 text

Affective Text Cake Chat ( E.L.S.A (MIT Media Lab)

Slide 20

Slide 20 text

Microsoft Xioice Source: 
 The Design and Implementation of XiaoIce, an Empathetic Social Chatbot 
 Li Zhou, Jianfeng Gao, Di Li, Heung-Yeung Shum

Slide 21

Slide 21 text

Affective Speech • Gender • Location (Accents) • Prosody • Wavenet • Tacotron • DeepVoice • Personality • Voicery Source: Google AI Blog
 Expressive Speech Synthesis with Tacotron

Slide 22

Slide 22 text

Parting thoughts:
 Designing Affective Experiences Responsibly

Slide 23

Slide 23 text

References • Affective Computing Group @ MIT Media Labs • Swiss Centre for Affective Sciences • • (Research Github) • Why we are Wired to Connect (Scientific American) • EmoPy: A machine learning toolkit for emotion expression