CONFIDENTIAL The Humane Machine Building AI systems with emotional intelligence Arpit Mathur
 Principal Engineer, Comcast Labs


https://en.wikipedia.org/wiki/File:NASA_Mars_Rover.jpg

“The fact Mars rovers last message before it went silent is “my battery is low and its getting dark” literally has me in tears like it probably died out there in mars alone and afraid hoping nasa would save it” Last message was, "my battery is low and its getting dark out”. Why tf am i crying in the club over a robot Farewell, tiny Martian robot car. One day you’ll be found by our descendants, a monument to our primitive 21st century technology. Y’all I’m tearing up over a robot help me. That “my battery is low and its getting dark” got me. You did great, sweetie. You did so great. Brb crying over a space robot @NASA #Oppy #MarsRover "My battery is low and it's getting dark" is perhaps the saddest thing I can imagine a little robot saying. And, thanks to my dumb human pack bonding instincts, I can think of robots like little pets so I'm crying. Goodnight, #Oppy

We are wired for emotional connection

Scenario • User is frustrated with a lag in the application/game he is playing. Do you • Ignore user’s state? • Let the user vent? • Acknowledge the user’s frustration?

Scenario • User is frustrated with a lag in the application/game he is playing. Do you • Ignore user’s state? • Let the user vent? • Acknowledge the user’s frustration? Source: 
 This computer responds to user frustration 
 J. Klein ; Y. Moon ; R.W. Picard

Affective Computing The study and development of systems and devices that can recognize, process, and simulate human affects

Sentiment Emotion Emotion vs Sentiment

Recognition

Modalities Text Voice Facial Expressions EEG Heart Rate Skin Conductance

Linguistic Inquiry and Word Count (LIWC) • Text Analysis module + Internal Dictionaries • Main dictionary composed of almost 6,400 words, word stems, and selected emoticons • Each dictionary entry maps to multiple categories • Categories are arranged hierarchically • 90 different output dimensions “I love this restaurant. Best sushi in town. And a BYOB for extra win! ” Source: http://liwc.wpengine.com

Face Emotion Recognition • DIY • Source publicly available image datasets ( Microsoft FER+ is a great fit ) • EmoPy or OpenCV or just Github search • APIs • Microsoft Azure, Amazon Recognition, Google Source: Microsoft FER+

Processing

Circumplex Model of Emotion

Plutchik’s wheel of Emotions

MFOEM Ontology An Emotion Ontology to describe affective phenomena, as a branch of the broader Mental Functioning ontology effort developed by The Swiss Centre for Affective Sciences, in collaboration with the University at Buffalo https://github.com/jannahastings/emotion-ontology

Emotion Understanding as a Service IBM Tone Analyzer (text) Microsoft (vision) Amazon Rekognition Emoshape EPU Voicery generation Anger TRUE TRUE TRUE TRUE TRUE Fear TRUE TRUE Joy / Happy TRUE TRUE TRUE TRUE TRUE Sadness TRUE TRUE TRUE TRUE TRUE Analytical TRUE Confident TRUE TRUE Tentative TRUE Surprized TRUE TRUE Calm TRUE TRUE Conversational Flirty Flustered Scared TRUE TRUE Disgust TRUE Indifference TRUE Regret TRUE Anticipation TRUE Trust TRUE Desire TRUE Contempt TRUE

Simulation

Affective Text Cake Chat (Replica.ai) E.L.S.A (MIT Media Lab)

Microsoft Xioice Source: 
 The Design and Implementation of XiaoIce, an Empathetic Social Chatbot 
 Li Zhou, Jianfeng Gao, Di Li, Heung-Yeung Shum

Affective Speech • Gender • Location (Accents) • Prosody • Wavenet • Tacotron • DeepVoice • Personality • Voicery Source: Google AI Blog
 Expressive Speech Synthesis with Tacotron

Parting thoughts:
 Designing Affective Experiences Responsibly

References • Affective Computing Group @ MIT Media Labs • Swiss Centre for Affective Sciences • https://en.wikipedia.org/wiki/Affective_computing • Replika.ai (Research Github) • Why we are Wired to Connect (Scientific American) • EmoPy: A machine learning toolkit for emotion expression