Ulster Research Seminar Series

Ulster University
(School of Computing Research Seminar Series 2022)

Transcript

1. Javier Gonzalez-Sanchez
   javiergs@asu.edu
   javiergs.com
   Artificial Emotional Intelligence
   Building Empathetic Machines
   

   View Slide

2. Thank you
   

   View Slide

3. 3
   Emotions
   Motivation
   signals what humans care about
   is involved in rational
   decision-making and action
   selection.
   

   View Slide

4. 4
   Motivation
   rational
   decision-making
   

   View Slide

5. 5
   Motivation
   

   View Slide

6. 6
   Outline
   Background 1
   § Key Ideas
   § Context and Workflow
   Sensing and Perception 2
   § Data: Brainwaves, Facial Gestures, Eye Tracking, and More
   § Machine Learning Models
   Integration 3
   § Fusion
   § Emotional Models: Ekman and Mehrabian
   Projects 4
   

   View Slide

7. 7
   Key Ideas
   Affect, affective state, emotion, emotional state, 👤
   feelings 🧠, mood ⏱.
   

   View Slide

8. 8
   Key Ideas
   +P+A+D
   Engagement
   +P-A+D
   Meditation
   Concentration
   Thought
   Relaxation
   +P+A-D
   Excitement
   Interest
   Dependence
   +P-A-D
   Starting
   Agreement
   Docility
   -P+A+D
   Disagreement
   Hostility
   -P-A+D
   Disdain
   -P+A-D
   Frustration
   Unsureness
   Anxiety
   -P-A-D
   Boredom
   

   View Slide

9. 9
   Key Ideas
   Many technologies may be improved by the capability
   to recognize human affect and to respond adaptively
   by appropriately modifying their operation
   Empathy is the capacity to understand what another
   person is experiencing
   Emotion AI
   

   View Slide

10. 10
    Context
    Rosalind Picard
    MIT MediaLab
    HCI
    Affective
    Computing
    1997
    

    View Slide

11. 11
    Context
    

    View Slide

12. 12
    Context
    Rosalind Picard
    MIT MediaLab
    Winslow Burleson
    University of Arizona
    HCI
    Affective
    Computing
    1997
    SW
    Engineering
    Self-Adaptive
    Systems
    David Garlan
    CMU
    

    View Slide

13. 13
    Workflow
    

    View Slide

14. 14
    Outline
    Background 1
    § Key Ideas
    § Context and Workflow
    Sensing and Perception 2
    § Data: Brainwaves, Facial Gestures, Eye Tracking, and More
    § Machine Learning Models
    Integration 3
    § Fusion
    § Emotional Models: Ekman and Mehrabian
    Projects 4
    

    View Slide

15. View Slide

16. 16
    Brain
    

    View Slide

17. 17
    Brain
    https://askabiologist.asu.edu/brain-regions
    

    View Slide

18. 18
    Brain
    Timestamp AF3 F7 F3 FC5 T7 P7 O1 O2 P8 T8 FC6 F4 F8 AF4 AccX AccY
    101116112544901 4542.05 4831.79 4247.18 4690.26 4282.56 4395.38 4591.79 4569.23 4360 4570.77 4297.44 4311.28 4282.56 4367.18 1660 2003
    101116112544901 4536.92 4802.05 4243.08 4673.85 4272.31 4393.33 4592.82 4570.26 4354.87 4570.26 4292.31 4309.74 4277.95 4370.77 1658 2002
    101116112545010 4533.33 4798.97 4234.87 4669.74 4301.03 4396.92 4592.31 4570.77 4351.28 4561.03 4281.54 4301.54 4271.28 4363.59 1659 2003
    101116112545010 4549.23 4839.49 4241.03 4691.28 4333.85 4397.95 4596.41 4567.18 4355.9 4556.41 4286.15 4306.15 4277.95 4369.74 1659 2003
    101116112545010 4580 4865.64 4251.79 4710.26 4340 4401.54 4603.59 4572.82 4360 4558.46 4298.97 4324.62 4296.41 4395.9 1657 2004
    101116112545010 4597.44 4860 4252.82 4705.64 4350.26 4412.31 4603.59 4577.44 4357.44 4555.9 4295.38 4329.23 4296.41 4414.36 1656 2005
    101116112545010 4584.62 4847.69 4246.67 4690.26 4360 4409.23 4597.44 4569.74 4351.79 4549.74 4278.97 4316.92 4272.82 4399.49 1656 2006
    101116112545010 4566.15 4842.05 4238.46 4684.1 4322.05 4389.74 4592.82 4566.67 4351.79 4549.74 4274.36 4310.26 4262.05 4370.77 1655 2005
    101116112545010 4563.59 4844.62 4231.79 4687.69 4267.69 4387.69 4594.36 4580 4361.03 4556.41 4278.97 4310.77 4274.36 4370.77 1653 2006
    101116112545010 4567.18 4847.18 4233.33 4688.72 4285.13 4409.23 4602.05 4589.23 4368.21 4560 4280.51 4310.77 4281.54 4390.26 1655 2004
    101116112545010 4562.05 4840.51 4227.18 4673.85 4300 4405.13 4611.28 4601.03 4376.41 4561.54 4280 4303.59 4279.49 4374.87 1652 2000
    

    View Slide

19. 19
    Brain
    14 channels
    128 samples per second
    1,792 values por second
    107,520 values per minute
    6,451,200 values per hour
    

    View Slide

20. 20
    Brain
    Timestamp Short Term Excitement Long Term Excitement Engagement Meditation Frustration
    101116091145065 0.447595 0.54871 0.834476 0.333844 0.536197
    101116091145190 0.447595 0.54871 0.834476 0.333844 0.536197
    101116091145315 0.447595 0.54871 0.834476 0.333844 0.536197
    101116091145440 0.487864 0.546877 0.834146 0.339548 0.54851
    101116091145565 0.487864 0.546877 0.834146 0.339548 0.54851
    101116091145690 0.487864 0.546877 0.834146 0.339548 0.54851
    101116091145815 0.487864 0.546877 0.834146 0.339548 0.54851
    101116091145940 0.521663 0.545609 0.839321 0.348321 0.558228
    101116091146065 0.521663 0.545609 0.839321 0.348321 0.558228
    101116091146190 0.521663 0.545609 0.839321 0.348321 0.558228
    101116091146315 0.521663 0.545609 0.839321 0.348321 0.558228
    101116091146440 0.509297 0.544131 0.84401 0.358717 0.546771
    101116091146565 0.509297 0.544131 0.84401 0.358717 0.546771
    101116091146690 0.509297 0.544131 0.84401 0.358717 0.546771
    101116091146815 0.509297 0.544131 0.84401 0.358717 0.546771
    101116091146941 0.451885 0.541695 0.848087 0.368071 0.533919
    

    View Slide

21. 21
    Brain
    21
    5 samples per second
    5 affective states
    25 values per second
    1,500 values per minute
    90,000 values per hour
    

    View Slide

22. 22
    Brain
    

    View Slide

23. 23
    ML
    • Neural Networks
    • Random Forest
    

    View Slide

24. View Slide

25. 25
    Face
    (Ekman and Friesen 1978) – Facial Action Coding System, 46 actions (plus head movements).
    19 Lip Corner Depressor
    26 Jaw Drop
    27 Mouth Stretch
    

    View Slide

26. 26
    Face
    

    View Slide

27. 27
    Face
    Timestamp Agreement Concentrating Disagreement Interested Thinking Unsure
    101116112838516 0.001836032 0.999917 1.79E-04 0.16485406 0.57114255 0.04595062
    101116112838578 0.001447654 0.9999516 1.29E-04 0.16310683 0.5958921 0.042706452
    101116112838672 5.97E-04 0 1.5E-04 0.44996294 0.45527613 0.00789697
    101116112838766 2.46E-04 0 1.75E-04 0.77445686 0.32144752 0.001418217
    101116112838860 1.01E-04 0 2.04E-04 0.93511915 0.21167138 2.53E-04
    101116112838953 4.18E-05 0 2.38E-04 0.983739 0.13208677 4.52E-05
    101116112839016 1.72E-05 0 2.78E-04 0.9960774 0.07941038 8.07E-06
    101116112839110 7.1E-06 0 3.24E-04 0.99906266 0.046613157 1.44E-06
    101116112839156 2.92E-06 0 3.77E-04 0.99977654 0.026964737 2.57E-07
    101116112839250 1.21E-06 0 4.4E-04 0.9999467 0.015464196 4.58E-08
    101116112839391 4.97E-07 0 5.12E-04 0.9999873 0.008824189 8.18E-09
    101116112839438 2.05E-07 0 5.97E-04 0.999997 0.005020725 1.46E-09
    101116112839547 8.43E-08 0 6.96E-04 0.9999993 0.002851939 2.6E-10
    101116112839578 3.47E-08 0 8.11E-04 0.9999999 0.001618473 4.64E-11
    101116112839688 1.43E-08 0 9.45E-04 0.99999994 9.18E-04 8.29E-12
    101116112839781 5.9E-09 0 0.001101404 1 5.21E-04 1.48E-12
    101116112839828 2.43E-09 0 0.001283521 1 2.95E-04 2.64E-13
    

    View Slide

28. 28
    Face
    28
    30 frames per second
    10 inferences per second
    600 values per minute
    36,000 values per hour
    

    View Slide

29. 29
    Face
    

    View Slide

30. 30
    ML
    • Support Vector Machine
    

    View Slide

31. View Slide

32. 32
    Eye
    

    View Slide

33. 33
    Eye
    Timestamp GPX GPY Pupil Left Validity L Pupil Right Validity R Fixation Event AOI
    101124162405582 636 199 2.759313 0 2.88406 0 48 Content
    101124162405599 641 207 2.684893 0 2.855817 0 48 Content
    101124162405615 659 211 2.624458 0 2.903861 0 48 Content
    101124162405632 644 201 2.636186 0 2.916132 0 48 Content
    101124162405649 644 213 2.690685 0 2.831013 0 48 Content
    101124162405666 628 194 2.651784 0 2.869714 0 48 Content
    101124162405682 614 177 2.829281 0 2.899828 0 48 Content
    101124162405699 701 249 2.780344 0 2.907665 0 49 Content
    101124162405716 906 341 2.853761 0 2.916398 0 49 Content
    101124162405732 947 398 2.829427 0 2.889944 0 49 Content
    101124162405749 941 400 2.826602 0 2.881179 0 49 Content
    101124162405766 938 403 2.78699 0 2.87948 0 49 KeyPress Content
    101124162405782 937 411 2.803387 0 2.821803 0 49 Content
    101124162405799 934 397 2.819166 0 2.871547 0 49 Content
    101124162405816 941 407 2.811687 0 2.817927 0 49 Content
    101124162405832 946 405 2.857419 0 2.857427 0 49 Content
    101124162405849 0 0 -1 4 -1 4 49 Content
    

    View Slide

34. 34
    Eye
    30 o 60 frames per second
    30 o 60 inferences per second
    1,800 o 3,600 values per minute
    108,000 o 216, 000 values per hour
    

    View Slide

35. 35
    Eye
    

    View Slide

36. 36
    ML
    • Just Geometry
    

    View Slide

37. View Slide

38. 38
    Pressure Sensor
    More
    

    View Slide

39. 39
    Galvanic Skin Conductance
    More
    

    View Slide

40. 40
    More
    

    View Slide

41. 41
    More
    Gonzalez-Sanchez et al, 2011
    

    View Slide

42. 42
    ML
    • Random Forest
    • Deep Learning
    

    View Slide

43. 43
    Outline
    Background 1
    § Key Ideas
    § Context and Workflow
    Sensing and Perception 2
    § Data: Brainwaves, Facial Gestures, Eye Tracking, and More
    § Machine Learning Models
    Integration 3
    § Fusion
    § Emotional Models: Ekman and Mehrabian
    Projects 4
    

    View Slide

44. 44
    Sparse Learning
    timestamp fixationIndex gazePointX gazePointY
    mappedFixationPoin
    tX
    mappedFixationPoin
    tY fixationDuration
    Short Term
    Excitement
    Long Term
    Excitement Engagement/Boredom Meditation Frustration Conductance agreement concentrating
    4135755652 0.436697 0.521059 0.550011 0.335825 0.498908
    0.40169062
    8
    4135755659 213 573 408 570 408 216
    4135755668 0.436697 0.521059 0.550011 0.335825 0.498908
    4135755676 213 566 412 570 408 216
    4135755692 213 565 404 570 408 216
    4135755709 213 567 404 570 408 216
    4135755714
    4135755726 213 568 411 570 408 216
    4135755742 213 568 409 570 408 216
    4135755759 213 563 411 570 408 216
    4135755761
    4135755776 213 574 413 570 408 216
    4135755792 213 554 402 570 408 216
    4135755809 214 603 409 696 405 216
    4135755824
    4135755826 214 701 407 696 405 216
    4135755842 214 697 403 696 405 216
    4135755859 214 693 401 696 405 216
    4135755876 214 700 402 696 405 216
    4135755892 214 701 411 696 405 216
    4135755909 214 686 398 696 405 216
    4135755918
    4135755926 214 694 399 696 405 216
    4135755942 214 694 407 696 405 216
    4135755959 214 698 404 696 405 216
    4135755964
    4135756027 0.436697 0.521059 0.550011 0.335825 0.498908 1 1
    

    View Slide

45. 45
    State Machine
    timestamp fixationIndex gazePointX gazePointY
    mappedFixationPoin
    tX
    mappedFixationPoin
    tY fixationDuration
    Short Term
    Excitement
    Long Term
    Excitement Engagement/Boredom Meditation Frustration Conductance agreement
    concentrati
    ng
    4135755652 213 574 414 570 408 216 0.436697 0.521059 0.550011 0.335825 0.498908 0.401690628 1 1
    4135755659 213 573 408 570 408 216 0.436697 0.521059 0.550011 0.335825 0.498908 0.401690628 1 1
    4135755668 213 573 408 570 408 216 0.436697 0.521059 0.550011 0.335825 0.498908 0.401690628 1 1
    4135755676 213 566 412 570 408 216 0.436697 0.521059 0.550011 0.335825 0.498908 0.401690628 1 1
    4135755692 213 565 404 570 408 216 0.436697 0.521059 0.550011 0.335825 0.498908 0.401690628 1 1
    4135755709 213 567 404 570 408 216 0.436697 0.521059 0.550011 0.335825 0.498908 0.401690628 1 1
    4135755714 213 567 404 570 408 216 0.436697 0.521059 0.550011 0.335825 0.498908 0.401690628 1 1
    4135755726 213 568 411 570 408 216 0.436697 0.521059 0.550011 0.335825 0.498908 0.401690628 1 1
    4135755742 213 568 409 570 408 216 0.436697 0.521059 0.550011 0.335825 0.498908 0.401690628 1 1
    4135755759 213 563 411 570 408 216 0.436697 0.521059 0.550011 0.335825 0.498908 0.401690628 1 1
    4135755761 213 563 411 570 408 216 0.436697 0.521059 0.550011 0.335825 0.498908 0.401690628 1 1
    4135755776 213 574 413 570 408 216 0.436697 0.521059 0.550011 0.335825 0.498908 0.401690628 1 1
    4135755792 213 554 402 570 408 216 0.436697 0.521059 0.550011 0.335825 0.498908 0.401690628 1 1
    4135755809 214 603 409 696 405 216 0.436697 0.521059 0.550011 0.335825 0.498908 0.401690628 1 1
    4135756027 215 728 406 804 387 183 0.436697 0.521059 0.550011 0.335825 0.498908 0.401690628 1 1
    

    View Slide

46. 46
    PAD
    

    View Slide

47. 47
    Outline
    Background 1
    § Key Ideas
    § Context and Workflow
    Sensing and Perception 2
    § Data: Brainwaves, Facial Gestures, Eye Tracking, and More
    § Machine Learning Models
    Integration 3
    § Fusion
    § Emotional Models: Ekman and Mehrabian
    Projects 4
    

    View Slide

48. 48
    Affect Recognition
    BCI and Gaze Points engagement
    

    View Slide

49. 49
    Affect Recognition
    BCI and Gaze Points frustration
    

    View Slide

50. 50
    Affect Recognition
    BCI and Gaze Points engagement
    

    View Slide

51. 51
    Affect Recognition
    BCI and Gaze Points frustration
    

    View Slide

52. 52
    Neuromarketing
    Chavez, M., Christopherson, R., Gonzalez-Sanchez, J., Atkinson, R.
    User Experience. 2018
    

    View Slide

53. 53
    Avatar
    Gonzalez-Sanchez, J., Chavez, M., Gibson, D., and Atkinson, R.
    Multimodal Affect Recognition in Virtual Worlds. ACII 2013
    

    View Slide

54. 54
    Projects
    Harris, A., Hoch, A., Kral, R., Teposte, M., Villa, A., et. al.
    Including affect-driven adaptation to the Pac-Man video game. ACM ISWC 2014
    

    View Slide

55. 55
    Projects
    Bernays, R., Mone, J., Yau, P., Murcia, M., Gonzalez-Sanchez, J., et al.
    Lost in the dark. ACM UIST 2012
    

    View Slide

56. 56
    Projects
    Hang, B., Loucks, S., Patel, P., Wiseman, K.
    Capstone Project
    2021-
    

    View Slide

57. 57
    Projects
    Rodriguez, J., Gonzalez-Sanchez, J., Del-Valle, C.
    Affect-Driven Robot-assisted Walking Therapy
    2019-
    

    View Slide

58. 58
    Projects
    VanLehn, K., Burleson, W., Chavez, M., Gonzalez-Sanchez, J., et al.
    The Affective Meta-Tutoring project
    ITS 2014 - 2018
    

    View Slide

59. 59
    Education Marketing Framework
    Tools
    Vision
    Health
    

    View Slide

60. 60
    Projects
    

    View Slide

61. 61
    Conclusion
    Let us rethink how scientist and engineers design
    future software systems
    

    View Slide

62. 62
    Artificial Emotional
    Intelligence:
    Building Empathetic
    Machines
    Questions
    Javier Gonzalez-Sanchez
    javiergs@asu.edu
    javiergs.com
    

    View Slide

63. Thank you
    

    View Slide

64. !"#$%&'(&)*+&'"#,$-.#/0#1'$213*1/4$
    &'$4/,#$.3''&56#$57
    For additional information, please visit http://dsp.acm.org/
    

    View Slide