Emotional AI

Gettysburg College
(ACM Distinguished Speakers Series 2021)

Mount St. Mary's University
(ACM Distinguished Speakers Series 2022)

Transcript

1. !"#$%&'(&)*+&'"#,$-.#/0#1'$213*1/4$
   &'$4/,#$.3''&56#$57
   For additional information, please visit http://dsp.acm.org/
   

   View Slide

2. Thank you
   

   View Slide

3. Thank you
   For additional information, please visit http://dsp.acm.org/
   

   View Slide

4. Javier Gonzalez-Sanchez
   javiergs@asu.edu
   javiergs.com
   Artificial Emotional Intelligence
   Building Empathetic Machines
   

   View Slide

5. Research
   Teaching
   Me
   

   View Slide

6. 6
   Emotions
   Motivation
   signals what humans care about
   is involved in rational
   decision-making and action
   selection.
   

   View Slide

7. 7
   Motivation
   rational
   decision-making
   

   View Slide

8. 8
   Motivation
   

   View Slide

9. 9
   Outline
   Background 1
   § Key Ideas
   § Context and Workflow
   Sensing and Perception 2
   § Data: Brainwaves, Facial Gestures, Eye Tracking, and More
   § Machine Learning Models
   Integration 3
   § Fusion
   § Emotional Models: Ekman and Mehrabian
   Projects 4
   

   View Slide

10. 10
    Key Ideas
    Affect, affective state, emotion, emotional state, 👤
    feelings 🧠, mood ⏱.
    

    View Slide

11. 11
    Key Ideas
    +P+A+D
    Engagement
    +P-A+D
    Meditation
    Concentration
    Thought
    Relaxation
    +P+A-D
    Excitement
    Interest
    Dependence
    +P-A-D
    Starting
    Agreement
    Docility
    -P+A+D
    Disagreement
    Hostility
    -P-A+D
    Disdain
    -P+A-D
    Frustration
    Unsureness
    Anxiety
    -P-A-D
    Boredom
    

    View Slide

12. 12
    Key Ideas
    Many technologies may be improved by the capability
    to recognize human affect and to respond adaptively
    by appropriately modifying their operation
    Empathy is the capacity to understand what another
    person is experiencing
    Emotion AI
    

    View Slide

13. 13
    Context
    Rosalind Picard
    MIT MediaLab
    HCI
    Affective
    Computing
    1997
    

    View Slide

14. 14
    Context
    

    View Slide

15. 15
    Context
    Rosalind Picard
    MIT MediaLab
    Winslow Burleson
    University of Arizona
    HCI
    Affective
    Computing
    1997
    SW
    Engineering
    Self-Adaptive
    Systems
    

    View Slide

16. 16
    Workflow
    

    View Slide

17. 17
    Outline
    Background 1
    § Key Ideas
    § Context and Workflow
    Sensing and Perception 2
    § Data: Brainwaves, Facial Gestures, Eye Tracking, and More
    § Machine Learning Models
    Integration 3
    § Fusion
    § Emotional Models: Ekman and Mehrabian
    Projects 4
    

    View Slide

18. View Slide

19. 19
    Brain
    

    View Slide

20. 20
    Brain
    https://askabiologist.asu.edu/brain-regions
    

    View Slide

21. 21
    Brain
    Timestamp AF3 F7 F3 FC5 T7 P7 O1 O2 P8 T8 FC6 F4 F8 AF4 AccX AccY
    101116112544901 4542.05 4831.79 4247.18 4690.26 4282.56 4395.38 4591.79 4569.23 4360 4570.77 4297.44 4311.28 4282.56 4367.18 1660 2003
    101116112544901 4536.92 4802.05 4243.08 4673.85 4272.31 4393.33 4592.82 4570.26 4354.87 4570.26 4292.31 4309.74 4277.95 4370.77 1658 2002
    101116112545010 4533.33 4798.97 4234.87 4669.74 4301.03 4396.92 4592.31 4570.77 4351.28 4561.03 4281.54 4301.54 4271.28 4363.59 1659 2003
    101116112545010 4549.23 4839.49 4241.03 4691.28 4333.85 4397.95 4596.41 4567.18 4355.9 4556.41 4286.15 4306.15 4277.95 4369.74 1659 2003
    101116112545010 4580 4865.64 4251.79 4710.26 4340 4401.54 4603.59 4572.82 4360 4558.46 4298.97 4324.62 4296.41 4395.9 1657 2004
    101116112545010 4597.44 4860 4252.82 4705.64 4350.26 4412.31 4603.59 4577.44 4357.44 4555.9 4295.38 4329.23 4296.41 4414.36 1656 2005
    101116112545010 4584.62 4847.69 4246.67 4690.26 4360 4409.23 4597.44 4569.74 4351.79 4549.74 4278.97 4316.92 4272.82 4399.49 1656 2006
    101116112545010 4566.15 4842.05 4238.46 4684.1 4322.05 4389.74 4592.82 4566.67 4351.79 4549.74 4274.36 4310.26 4262.05 4370.77 1655 2005
    101116112545010 4563.59 4844.62 4231.79 4687.69 4267.69 4387.69 4594.36 4580 4361.03 4556.41 4278.97 4310.77 4274.36 4370.77 1653 2006
    101116112545010 4567.18 4847.18 4233.33 4688.72 4285.13 4409.23 4602.05 4589.23 4368.21 4560 4280.51 4310.77 4281.54 4390.26 1655 2004
    101116112545010 4562.05 4840.51 4227.18 4673.85 4300 4405.13 4611.28 4601.03 4376.41 4561.54 4280 4303.59 4279.49 4374.87 1652 2000
    

    View Slide

22. 22
    Brain
    14 channels
    128 samples per second
    1,792 values por second
    107,520 values per minute
    6,451,200 values per hour
    

    View Slide

23. 23
    Brain
    Timestamp Short Term Excitement Long Term Excitement Engagement Meditation Frustration
    101116091145065 0.447595 0.54871 0.834476 0.333844 0.536197
    101116091145190 0.447595 0.54871 0.834476 0.333844 0.536197
    101116091145315 0.447595 0.54871 0.834476 0.333844 0.536197
    101116091145440 0.487864 0.546877 0.834146 0.339548 0.54851
    101116091145565 0.487864 0.546877 0.834146 0.339548 0.54851
    101116091145690 0.487864 0.546877 0.834146 0.339548 0.54851
    101116091145815 0.487864 0.546877 0.834146 0.339548 0.54851
    101116091145940 0.521663 0.545609 0.839321 0.348321 0.558228
    101116091146065 0.521663 0.545609 0.839321 0.348321 0.558228
    101116091146190 0.521663 0.545609 0.839321 0.348321 0.558228
    101116091146315 0.521663 0.545609 0.839321 0.348321 0.558228
    101116091146440 0.509297 0.544131 0.84401 0.358717 0.546771
    101116091146565 0.509297 0.544131 0.84401 0.358717 0.546771
    101116091146690 0.509297 0.544131 0.84401 0.358717 0.546771
    101116091146815 0.509297 0.544131 0.84401 0.358717 0.546771
    101116091146941 0.451885 0.541695 0.848087 0.368071 0.533919
    

    View Slide

24. 24
    Brain
    24
    5 samples per second
    5 affective states
    25 values per second
    1,500 values per minute
    90,000 values per hour
    

    View Slide

25. 25
    Brain
    

    View Slide

26. 26
    ML
    • Neural Networks
    • Random Forest
    

    View Slide

27. View Slide

28. 28
    Face
    (Ekman and Friesen 1978) – Facial Action Coding System, 46 actions (plus head movements).
    19 Lip Corner Depressor
    26 Jaw Drop
    27 Mouth Stretch
    

    View Slide

29. 29
    Face
    

    View Slide

30. 30
    Face
    Timestamp Agreement Concentrating Disagreement Interested Thinking Unsure
    101116112838516 0.001836032 0.999917 1.79E-04 0.16485406 0.57114255 0.04595062
    101116112838578 0.001447654 0.9999516 1.29E-04 0.16310683 0.5958921 0.042706452
    101116112838672 5.97E-04 0 1.5E-04 0.44996294 0.45527613 0.00789697
    101116112838766 2.46E-04 0 1.75E-04 0.77445686 0.32144752 0.001418217
    101116112838860 1.01E-04 0 2.04E-04 0.93511915 0.21167138 2.53E-04
    101116112838953 4.18E-05 0 2.38E-04 0.983739 0.13208677 4.52E-05
    101116112839016 1.72E-05 0 2.78E-04 0.9960774 0.07941038 8.07E-06
    101116112839110 7.1E-06 0 3.24E-04 0.99906266 0.046613157 1.44E-06
    101116112839156 2.92E-06 0 3.77E-04 0.99977654 0.026964737 2.57E-07
    101116112839250 1.21E-06 0 4.4E-04 0.9999467 0.015464196 4.58E-08
    101116112839391 4.97E-07 0 5.12E-04 0.9999873 0.008824189 8.18E-09
    101116112839438 2.05E-07 0 5.97E-04 0.999997 0.005020725 1.46E-09
    101116112839547 8.43E-08 0 6.96E-04 0.9999993 0.002851939 2.6E-10
    101116112839578 3.47E-08 0 8.11E-04 0.9999999 0.001618473 4.64E-11
    101116112839688 1.43E-08 0 9.45E-04 0.99999994 9.18E-04 8.29E-12
    101116112839781 5.9E-09 0 0.001101404 1 5.21E-04 1.48E-12
    101116112839828 2.43E-09 0 0.001283521 1 2.95E-04 2.64E-13
    

    View Slide

31. 31
    Face
    31
    30 frames per second
    10 inferences per second
    600 values per minute
    36,000 values per hour
    

    View Slide

32. 32
    Face
    

    View Slide

33. 33
    ML
    • Support Vector Machine
    

    View Slide

34. View Slide

35. 35
    Eye
    

    View Slide

36. 36
    Eye
    Timestamp GPX GPY Pupil Left Validity L Pupil Right Validity R Fixation Event AOI
    101124162405582 636 199 2.759313 0 2.88406 0 48 Content
    101124162405599 641 207 2.684893 0 2.855817 0 48 Content
    101124162405615 659 211 2.624458 0 2.903861 0 48 Content
    101124162405632 644 201 2.636186 0 2.916132 0 48 Content
    101124162405649 644 213 2.690685 0 2.831013 0 48 Content
    101124162405666 628 194 2.651784 0 2.869714 0 48 Content
    101124162405682 614 177 2.829281 0 2.899828 0 48 Content
    101124162405699 701 249 2.780344 0 2.907665 0 49 Content
    101124162405716 906 341 2.853761 0 2.916398 0 49 Content
    101124162405732 947 398 2.829427 0 2.889944 0 49 Content
    101124162405749 941 400 2.826602 0 2.881179 0 49 Content
    101124162405766 938 403 2.78699 0 2.87948 0 49 KeyPress Content
    101124162405782 937 411 2.803387 0 2.821803 0 49 Content
    101124162405799 934 397 2.819166 0 2.871547 0 49 Content
    101124162405816 941 407 2.811687 0 2.817927 0 49 Content
    101124162405832 946 405 2.857419 0 2.857427 0 49 Content
    101124162405849 0 0 -1 4 -1 4 49 Content
    

    View Slide

37. 37
    Eye
    30 o 60 frames per second
    30 o 60 inferences per second
    1,800 o 3,600 values per minute
    108,000 o 216, 000 values per hour
    

    View Slide

38. 38
    Eye
    

    View Slide

39. 39
    ML
    • Just Geometry
    

    View Slide

40. View Slide

41. 41
    Pressure Sensor
    More
    

    View Slide

42. 42
    Galvanic Skin Conductance
    More
    

    View Slide

43. 43
    More
    

    View Slide

44. 44
    More
    Gonzalez-Sanchez et al, 2011
    

    View Slide

45. 45
    ML
    • Random Forest
    • Deep Learning
    

    View Slide

46. 46
    Outline
    Background 1
    § Key Ideas
    § Context and Workflow
    Sensing and Perception 2
    § Data: Brainwaves, Facial Gestures, Eye Tracking, and More
    § Machine Learning Models
    Integration 3
    § Fusion
    § Emotional Models: Ekman and Mehrabian
    Projects 4
    

    View Slide

47. 47
    Sparse Learning
    timestamp fixationIndex gazePointX gazePointY
    mappedFixationPoin
    tX
    mappedFixationPoin
    tY fixationDuration
    Short Term
    Excitement
    Long Term
    Excitement Engagement/Boredom Meditation Frustration Conductance agreement concentrating
    4135755652 0.436697 0.521059 0.550011 0.335825 0.498908
    0.40169062
    8
    4135755659 213 573 408 570 408 216
    4135755668 0.436697 0.521059 0.550011 0.335825 0.498908
    4135755676 213 566 412 570 408 216
    4135755692 213 565 404 570 408 216
    4135755709 213 567 404 570 408 216
    4135755714
    4135755726 213 568 411 570 408 216
    4135755742 213 568 409 570 408 216
    4135755759 213 563 411 570 408 216
    4135755761
    4135755776 213 574 413 570 408 216
    4135755792 213 554 402 570 408 216
    4135755809 214 603 409 696 405 216
    4135755824
    4135755826 214 701 407 696 405 216
    4135755842 214 697 403 696 405 216
    4135755859 214 693 401 696 405 216
    4135755876 214 700 402 696 405 216
    4135755892 214 701 411 696 405 216
    4135755909 214 686 398 696 405 216
    4135755918
    4135755926 214 694 399 696 405 216
    4135755942 214 694 407 696 405 216
    4135755959 214 698 404 696 405 216
    4135755964
    4135756027 0.436697 0.521059 0.550011 0.335825 0.498908 1 1
    

    View Slide

48. 48
    State Machine
    timestamp fixationIndex gazePointX gazePointY
    mappedFixationPoin
    tX
    mappedFixationPoin
    tY fixationDuration
    Short Term
    Excitement
    Long Term
    Excitement Engagement/Boredom Meditation Frustration Conductance agreement
    concentrati
    ng
    4135755652 213 574 414 570 408 216 0.436697 0.521059 0.550011 0.335825 0.498908 0.401690628 1 1
    4135755659 213 573 408 570 408 216 0.436697 0.521059 0.550011 0.335825 0.498908 0.401690628 1 1
    4135755668 213 573 408 570 408 216 0.436697 0.521059 0.550011 0.335825 0.498908 0.401690628 1 1
    4135755676 213 566 412 570 408 216 0.436697 0.521059 0.550011 0.335825 0.498908 0.401690628 1 1
    4135755692 213 565 404 570 408 216 0.436697 0.521059 0.550011 0.335825 0.498908 0.401690628 1 1
    4135755709 213 567 404 570 408 216 0.436697 0.521059 0.550011 0.335825 0.498908 0.401690628 1 1
    4135755714 213 567 404 570 408 216 0.436697 0.521059 0.550011 0.335825 0.498908 0.401690628 1 1
    4135755726 213 568 411 570 408 216 0.436697 0.521059 0.550011 0.335825 0.498908 0.401690628 1 1
    4135755742 213 568 409 570 408 216 0.436697 0.521059 0.550011 0.335825 0.498908 0.401690628 1 1
    4135755759 213 563 411 570 408 216 0.436697 0.521059 0.550011 0.335825 0.498908 0.401690628 1 1
    4135755761 213 563 411 570 408 216 0.436697 0.521059 0.550011 0.335825 0.498908 0.401690628 1 1
    4135755776 213 574 413 570 408 216 0.436697 0.521059 0.550011 0.335825 0.498908 0.401690628 1 1
    4135755792 213 554 402 570 408 216 0.436697 0.521059 0.550011 0.335825 0.498908 0.401690628 1 1
    4135755809 214 603 409 696 405 216 0.436697 0.521059 0.550011 0.335825 0.498908 0.401690628 1 1
    4135756027 215 728 406 804 387 183 0.436697 0.521059 0.550011 0.335825 0.498908 0.401690628 1 1
    

    View Slide

49. 49
    PAD
    

    View Slide

50. 50
    Outline
    Background 1
    § Key Ideas
    § Context and Workflow
    Sensing and Perception 2
    § Data: Brainwaves, Facial Gestures, Eye Tracking, and More
    § Machine Learning Models
    Integration 3
    § Fusion
    § Emotional Models: Ekman and Mehrabian
    Projects 4
    

    View Slide

51. 51
    Affect Recognition
    BCI and Gaze Points engagement
    

    View Slide

52. 52
    Affect Recognition
    BCI and Gaze Points frustration
    

    View Slide

53. 53
    Affect Recognition
    BCI and Gaze Points engagement
    

    View Slide

54. 54
    Affect Recognition
    BCI and Gaze Points frustration
    

    View Slide

55. 55
    Neuromarketing
    Chavez, M., Christopherson, R., Gonzalez-Sanchez, J., Atkinson, R.
    User Experience. 2018
    

    View Slide

56. 56
    Projects
    Gonzalez-Sanchez, J., Chavez, M., Gibson, D., and Atkinson, R. Multimodal Affect
    Recognition in Virtual Worlds. ACII 2013
    

    View Slide

57. 57
    Projects
    Harris, A., Hoch, A., Kral, R., Teposte, M., Villa, A., et. al.
    Including affect-driven adaptation to the Pac-Man video game. ACM ISWC 2014
    

    View Slide

58. 58
    Projects
    Bernays, R., Mone, J., Yau, P., Murcia, M., Gonzalez-Sanchez, J., et al.
    Lost in the dark:
    ACM UIST 2012
    

    View Slide

59. 59
    Projects
    Hang, B., Loucks, S., Patel, P., Wiseman, K.
    Capstone Project
    2021-
    

    View Slide

60. 60
    Projects
    VanLehn, K., Burleson, W., Chavez, M., Gonzalez-Sanchez, J., et al.
    The Affective Meta-Tutoring project
    ITS 2014 - 2018
    

    View Slide

61. 61
    Projects
    Rodriguez, J., Gonzalez-Sanchez, J., Del-Valle, C.
    Affect-Driven Robot-assisted Walking Therapy
    2019-
    

    View Slide

62. 62
    Education Marketing Framework
    Tools
    Vision
    Health
    

    View Slide

63. 63
    Projects
    

    View Slide

64. 64
    Conclusion
    Let us rethink how scientist and engineers design
    future software systems
    

    View Slide

65. 65
    Artificial Emotional
    Intelligence:
    Building Empathetic
    Machines
    Questions
    Javier Gonzalez-Sanchez
    javiergs@asu.edu
    javiergs.com
    

    View Slide

66. Thank you
    

    View Slide

67. Thank you
    For additional information, please visit http://dsp.acm.org/
    

    View Slide

68. !"#$%&'(&)*+&'"#,$-.#/0#1'$213*1/4$
    &'$4/,#$.3''&56#$57
    For additional information, please visit http://dsp.acm.org/
    

    View Slide

69. About ACM
    § ACM, the Association for Computing Machinery (www.acm.org), is the
    premier global community of computing professionals and students
    with nearly 100,000 members in more than 170 countries interacting
    with more than 2 million computing professionals worldwide.
    § OUR MISSION: We help computing professionals to be their best and
    most creative. We connect them to their peers, to what the latest
    developments, and inspire them to advance the profession and make a
    positive impact on society.
    § OUR VISION: We see a world where computing helps solve tomorrow’s
    problems – where we use our knowledge and skills to advance the
    computing profession and make a positive social impact throughout the
    world.
    § I am proud to be an ACM Member.
    

    View Slide