Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Features
Speaker Deck
PRO
Sign in
Sign up for free
Search
Search
Ulster Research Seminar Series
Search
Javier Gonzalez-Sanchez
PRO
March 23, 2022
Research
0
310
Ulster Research Seminar Series
Ulster University
(School of Computing Research Seminar Series 2022)
Javier Gonzalez-Sanchez
PRO
March 23, 2022
Tweet
Share
More Decks by Javier Gonzalez-Sanchez
See All by Javier Gonzalez-Sanchez
CSC307 Lecture 15
javiergs
PRO
0
84
CSC307 Lecture 14
javiergs
PRO
0
220
CSC307 Lecture 13
javiergs
PRO
0
150
CSC307 Lecture 12
javiergs
PRO
0
220
CSC307 Lecture 11
javiergs
PRO
0
240
CSC307 Lecture 10
javiergs
PRO
0
310
CSC307 Lecture 09
javiergs
PRO
1
500
CSC307 Lecture 08
javiergs
PRO
0
330
CSC307 Lecture 07
javiergs
PRO
0
220
Other Decks in Research
See All in Research
アジャイルコミュニティが、宗教ポイと云われるのは何故なのか?
fujiihideo
0
260
【ICASSP2024】音声変換に関する全論文まとめ【Parakeet株式会社】
supikiti
0
600
機械学習と最適化の融合動的ロットサイズ決定問題を例として
mickey_kubo
2
360
WikipediaやYouTubeにおける論文参照 / joss2024
corgies
1
210
"多様な推薦"はユーザーの目にどう映るか
kuri8ive
3
260
論文紹介 AST: Audio Spectrogram Transformer
kazu07
0
190
デジタルツインによる ネイチャーポジティブへの挑戦
fullfull
0
210
新入生向けチュートリアル:文献のサーベイv2
a1da4
9
7.8k
[第55回 NLPコロキウム] コンピュータビジョン分野での評価設計と分析の研究について
otani_mayu
0
130
Mathematical Optimization +Artificial Intelligence =MOAI
mickey_kubo
1
230
点群処理の基礎: 平面の検出と、その上下の点の取り出しについて
kentaitakura
0
330
SSII2024 [SS1] 拡散モデルの今 〜 2024年の研究動向 〜
ssii
PRO
2
1.9k
Featured
See All Featured
Done Done
chrislema
179
15k
Build The Right Thing And Hit Your Dates
maggiecrowley
28
2.2k
Testing 201, or: Great Expectations
jmmastey
33
6.9k
Navigating Team Friction
lara
181
13k
Java REST API Framework Comparison - PWX 2021
mraible
PRO
20
7.2k
Unsuck your backbone
ammeep
666
57k
CSS Pre-Processors: Stylus, Less & Sass
bermonpainter
353
29k
Mobile First: as difficult as doing things right
swwweet
219
8.8k
4 Signs Your Business is Dying
shpigford
178
21k
WebSockets: Embracing the real-time Web
robhawkes
59
7.2k
Designing for Performance
lara
604
67k
Clear Off the Table
cherdarchuk
89
320k
Transcript
Javier Gonzalez-Sanchez
[email protected]
javiergs.com Artificial Emotional Intelligence Building Empathetic Machines
Thank you
3 Emotions Motivation signals what humans care about is involved
in rational decision-making and action selection.
4 Motivation rational decision-making
5 Motivation
6 Outline Background 1 § Key Ideas § Context and
Workflow Sensing and Perception 2 § Data: Brainwaves, Facial Gestures, Eye Tracking, and More § Machine Learning Models Integration 3 § Fusion § Emotional Models: Ekman and Mehrabian Projects 4
7 Key Ideas Affect, affective state, emotion, emotional state, 👤
feelings 🧠, mood ⏱.
8 Key Ideas +P+A+D Engagement +P-A+D Meditation Concentration Thought Relaxation
+P+A-D Excitement Interest Dependence +P-A-D Starting Agreement Docility -P+A+D Disagreement Hostility -P-A+D Disdain -P+A-D Frustration Unsureness Anxiety -P-A-D Boredom
9 Key Ideas Many technologies may be improved by the
capability to recognize human affect and to respond adaptively by appropriately modifying their operation Empathy is the capacity to understand what another person is experiencing Emotion AI
10 Context Rosalind Picard MIT MediaLab HCI Affective Computing 1997
11 Context
12 Context Rosalind Picard MIT MediaLab Winslow Burleson University of
Arizona HCI Affective Computing 1997 SW Engineering Self-Adaptive Systems David Garlan CMU
13 Workflow
14 Outline Background 1 § Key Ideas § Context and
Workflow Sensing and Perception 2 § Data: Brainwaves, Facial Gestures, Eye Tracking, and More § Machine Learning Models Integration 3 § Fusion § Emotional Models: Ekman and Mehrabian Projects 4
15 1
16 Brain
17 Brain https://askabiologist.asu.edu/brain-regions
18 Brain Timestamp AF3 F7 F3 FC5 T7 P7 O1
O2 P8 T8 FC6 F4 F8 AF4 AccX AccY 101116112544901 4542.05 4831.79 4247.18 4690.26 4282.56 4395.38 4591.79 4569.23 4360 4570.77 4297.44 4311.28 4282.56 4367.18 1660 2003 101116112544901 4536.92 4802.05 4243.08 4673.85 4272.31 4393.33 4592.82 4570.26 4354.87 4570.26 4292.31 4309.74 4277.95 4370.77 1658 2002 101116112545010 4533.33 4798.97 4234.87 4669.74 4301.03 4396.92 4592.31 4570.77 4351.28 4561.03 4281.54 4301.54 4271.28 4363.59 1659 2003 101116112545010 4549.23 4839.49 4241.03 4691.28 4333.85 4397.95 4596.41 4567.18 4355.9 4556.41 4286.15 4306.15 4277.95 4369.74 1659 2003 101116112545010 4580 4865.64 4251.79 4710.26 4340 4401.54 4603.59 4572.82 4360 4558.46 4298.97 4324.62 4296.41 4395.9 1657 2004 101116112545010 4597.44 4860 4252.82 4705.64 4350.26 4412.31 4603.59 4577.44 4357.44 4555.9 4295.38 4329.23 4296.41 4414.36 1656 2005 101116112545010 4584.62 4847.69 4246.67 4690.26 4360 4409.23 4597.44 4569.74 4351.79 4549.74 4278.97 4316.92 4272.82 4399.49 1656 2006 101116112545010 4566.15 4842.05 4238.46 4684.1 4322.05 4389.74 4592.82 4566.67 4351.79 4549.74 4274.36 4310.26 4262.05 4370.77 1655 2005 101116112545010 4563.59 4844.62 4231.79 4687.69 4267.69 4387.69 4594.36 4580 4361.03 4556.41 4278.97 4310.77 4274.36 4370.77 1653 2006 101116112545010 4567.18 4847.18 4233.33 4688.72 4285.13 4409.23 4602.05 4589.23 4368.21 4560 4280.51 4310.77 4281.54 4390.26 1655 2004 101116112545010 4562.05 4840.51 4227.18 4673.85 4300 4405.13 4611.28 4601.03 4376.41 4561.54 4280 4303.59 4279.49 4374.87 1652 2000
19 Brain 14 channels 128 samples per second 1,792 values
por second 107,520 values per minute 6,451,200 values per hour
20 Brain Timestamp Short Term Excitement Long Term Excitement Engagement
Meditation Frustration 101116091145065 0.447595 0.54871 0.834476 0.333844 0.536197 101116091145190 0.447595 0.54871 0.834476 0.333844 0.536197 101116091145315 0.447595 0.54871 0.834476 0.333844 0.536197 101116091145440 0.487864 0.546877 0.834146 0.339548 0.54851 101116091145565 0.487864 0.546877 0.834146 0.339548 0.54851 101116091145690 0.487864 0.546877 0.834146 0.339548 0.54851 101116091145815 0.487864 0.546877 0.834146 0.339548 0.54851 101116091145940 0.521663 0.545609 0.839321 0.348321 0.558228 101116091146065 0.521663 0.545609 0.839321 0.348321 0.558228 101116091146190 0.521663 0.545609 0.839321 0.348321 0.558228 101116091146315 0.521663 0.545609 0.839321 0.348321 0.558228 101116091146440 0.509297 0.544131 0.84401 0.358717 0.546771 101116091146565 0.509297 0.544131 0.84401 0.358717 0.546771 101116091146690 0.509297 0.544131 0.84401 0.358717 0.546771 101116091146815 0.509297 0.544131 0.84401 0.358717 0.546771 101116091146941 0.451885 0.541695 0.848087 0.368071 0.533919
21 Brain 21 5 samples per second 5 affective states
25 values per second 1,500 values per minute 90,000 values per hour
22 Brain
23 ML • Neural Networks • Random Forest
2
25 Face (Ekman and Friesen 1978) – Facial Action Coding
System, 46 actions (plus head movements). 19 Lip Corner Depressor 26 Jaw Drop 27 Mouth Stretch
26 Face
27 Face Timestamp Agreement Concentrating Disagreement Interested Thinking Unsure 101116112838516
0.001836032 0.999917 1.79E-04 0.16485406 0.57114255 0.04595062 101116112838578 0.001447654 0.9999516 1.29E-04 0.16310683 0.5958921 0.042706452 101116112838672 5.97E-04 0 1.5E-04 0.44996294 0.45527613 0.00789697 101116112838766 2.46E-04 0 1.75E-04 0.77445686 0.32144752 0.001418217 101116112838860 1.01E-04 0 2.04E-04 0.93511915 0.21167138 2.53E-04 101116112838953 4.18E-05 0 2.38E-04 0.983739 0.13208677 4.52E-05 101116112839016 1.72E-05 0 2.78E-04 0.9960774 0.07941038 8.07E-06 101116112839110 7.1E-06 0 3.24E-04 0.99906266 0.046613157 1.44E-06 101116112839156 2.92E-06 0 3.77E-04 0.99977654 0.026964737 2.57E-07 101116112839250 1.21E-06 0 4.4E-04 0.9999467 0.015464196 4.58E-08 101116112839391 4.97E-07 0 5.12E-04 0.9999873 0.008824189 8.18E-09 101116112839438 2.05E-07 0 5.97E-04 0.999997 0.005020725 1.46E-09 101116112839547 8.43E-08 0 6.96E-04 0.9999993 0.002851939 2.6E-10 101116112839578 3.47E-08 0 8.11E-04 0.9999999 0.001618473 4.64E-11 101116112839688 1.43E-08 0 9.45E-04 0.99999994 9.18E-04 8.29E-12 101116112839781 5.9E-09 0 0.001101404 1 5.21E-04 1.48E-12 101116112839828 2.43E-09 0 0.001283521 1 2.95E-04 2.64E-13
28 Face 28 30 frames per second 10 inferences per
second 600 values per minute 36,000 values per hour
29 Face
30 ML • Support Vector Machine
3
32 Eye
33 Eye Timestamp GPX GPY Pupil Left Validity L Pupil
Right Validity R Fixation Event AOI 101124162405582 636 199 2.759313 0 2.88406 0 48 Content 101124162405599 641 207 2.684893 0 2.855817 0 48 Content 101124162405615 659 211 2.624458 0 2.903861 0 48 Content 101124162405632 644 201 2.636186 0 2.916132 0 48 Content 101124162405649 644 213 2.690685 0 2.831013 0 48 Content 101124162405666 628 194 2.651784 0 2.869714 0 48 Content 101124162405682 614 177 2.829281 0 2.899828 0 48 Content 101124162405699 701 249 2.780344 0 2.907665 0 49 Content 101124162405716 906 341 2.853761 0 2.916398 0 49 Content 101124162405732 947 398 2.829427 0 2.889944 0 49 Content 101124162405749 941 400 2.826602 0 2.881179 0 49 Content 101124162405766 938 403 2.78699 0 2.87948 0 49 KeyPress Content 101124162405782 937 411 2.803387 0 2.821803 0 49 Content 101124162405799 934 397 2.819166 0 2.871547 0 49 Content 101124162405816 941 407 2.811687 0 2.817927 0 49 Content 101124162405832 946 405 2.857419 0 2.857427 0 49 Content 101124162405849 0 0 -1 4 -1 4 49 Content
34 Eye 30 o 60 frames per second 30 o
60 inferences per second 1,800 o 3,600 values per minute 108,000 o 216, 000 values per hour
35 Eye
36 ML • Just Geometry
4
38 Pressure Sensor More
39 Galvanic Skin Conductance More
40 More
41 More Gonzalez-Sanchez et al, 2011
42 ML • Random Forest • Deep Learning
43 Outline Background 1 § Key Ideas § Context and
Workflow Sensing and Perception 2 § Data: Brainwaves, Facial Gestures, Eye Tracking, and More § Machine Learning Models Integration 3 § Fusion § Emotional Models: Ekman and Mehrabian Projects 4
44 Sparse Learning timestamp fixationIndex gazePointX gazePointY mappedFixationPoin tX mappedFixationPoin
tY fixationDuration Short Term Excitement Long Term Excitement Engagement/Boredom Meditation Frustration Conductance agreement concentrating 4135755652 0.436697 0.521059 0.550011 0.335825 0.498908 0.40169062 8 4135755659 213 573 408 570 408 216 4135755668 0.436697 0.521059 0.550011 0.335825 0.498908 4135755676 213 566 412 570 408 216 4135755692 213 565 404 570 408 216 4135755709 213 567 404 570 408 216 4135755714 4135755726 213 568 411 570 408 216 4135755742 213 568 409 570 408 216 4135755759 213 563 411 570 408 216 4135755761 4135755776 213 574 413 570 408 216 4135755792 213 554 402 570 408 216 4135755809 214 603 409 696 405 216 4135755824 4135755826 214 701 407 696 405 216 4135755842 214 697 403 696 405 216 4135755859 214 693 401 696 405 216 4135755876 214 700 402 696 405 216 4135755892 214 701 411 696 405 216 4135755909 214 686 398 696 405 216 4135755918 4135755926 214 694 399 696 405 216 4135755942 214 694 407 696 405 216 4135755959 214 698 404 696 405 216 4135755964 4135756027 0.436697 0.521059 0.550011 0.335825 0.498908 1 1
45 State Machine timestamp fixationIndex gazePointX gazePointY mappedFixationPoin tX mappedFixationPoin
tY fixationDuration Short Term Excitement Long Term Excitement Engagement/Boredom Meditation Frustration Conductance agreement concentrati ng 4135755652 213 574 414 570 408 216 0.436697 0.521059 0.550011 0.335825 0.498908 0.401690628 1 1 4135755659 213 573 408 570 408 216 0.436697 0.521059 0.550011 0.335825 0.498908 0.401690628 1 1 4135755668 213 573 408 570 408 216 0.436697 0.521059 0.550011 0.335825 0.498908 0.401690628 1 1 4135755676 213 566 412 570 408 216 0.436697 0.521059 0.550011 0.335825 0.498908 0.401690628 1 1 4135755692 213 565 404 570 408 216 0.436697 0.521059 0.550011 0.335825 0.498908 0.401690628 1 1 4135755709 213 567 404 570 408 216 0.436697 0.521059 0.550011 0.335825 0.498908 0.401690628 1 1 4135755714 213 567 404 570 408 216 0.436697 0.521059 0.550011 0.335825 0.498908 0.401690628 1 1 4135755726 213 568 411 570 408 216 0.436697 0.521059 0.550011 0.335825 0.498908 0.401690628 1 1 4135755742 213 568 409 570 408 216 0.436697 0.521059 0.550011 0.335825 0.498908 0.401690628 1 1 4135755759 213 563 411 570 408 216 0.436697 0.521059 0.550011 0.335825 0.498908 0.401690628 1 1 4135755761 213 563 411 570 408 216 0.436697 0.521059 0.550011 0.335825 0.498908 0.401690628 1 1 4135755776 213 574 413 570 408 216 0.436697 0.521059 0.550011 0.335825 0.498908 0.401690628 1 1 4135755792 213 554 402 570 408 216 0.436697 0.521059 0.550011 0.335825 0.498908 0.401690628 1 1 4135755809 214 603 409 696 405 216 0.436697 0.521059 0.550011 0.335825 0.498908 0.401690628 1 1 4135756027 215 728 406 804 387 183 0.436697 0.521059 0.550011 0.335825 0.498908 0.401690628 1 1
46 PAD
47 Outline Background 1 § Key Ideas § Context and
Workflow Sensing and Perception 2 § Data: Brainwaves, Facial Gestures, Eye Tracking, and More § Machine Learning Models Integration 3 § Fusion § Emotional Models: Ekman and Mehrabian Projects 4
48 Affect Recognition BCI and Gaze Points engagement
49 Affect Recognition BCI and Gaze Points frustration
50 Affect Recognition BCI and Gaze Points engagement
51 Affect Recognition BCI and Gaze Points frustration
52 Neuromarketing Chavez, M., Christopherson, R., Gonzalez-Sanchez, J., Atkinson, R.
User Experience. 2018
53 Avatar Gonzalez-Sanchez, J., Chavez, M., Gibson, D., and Atkinson,
R. Multimodal Affect Recognition in Virtual Worlds. ACII 2013
54 Projects Harris, A., Hoch, A., Kral, R., Teposte, M.,
Villa, A., et. al. Including affect-driven adaptation to the Pac-Man video game. ACM ISWC 2014
55 Projects Bernays, R., Mone, J., Yau, P., Murcia, M.,
Gonzalez-Sanchez, J., et al. Lost in the dark. ACM UIST 2012
56 Projects Hang, B., Loucks, S., Patel, P., Wiseman, K.
Capstone Project 2021-
57 Projects Rodriguez, J., Gonzalez-Sanchez, J., Del-Valle, C. Affect-Driven Robot-assisted
Walking Therapy 2019-
58 Projects VanLehn, K., Burleson, W., Chavez, M., Gonzalez-Sanchez, J.,
et al. The Affective Meta-Tutoring project ITS 2014 - 2018
59 Education Marketing Framework Tools Vision Health
60 Projects
61 Conclusion Let us rethink how scientist and engineers design
future software systems
62 Artificial Emotional Intelligence: Building Empathetic Machines Questions Javier Gonzalez-Sanchez
[email protected]
javiergs.com
Thank you
!"#$%&'(&)*+&'"#,$-.#/0#1'$213*1/4$ &'$4/,#$.3''&56#$57 For additional information, please visit http://dsp.acm.org/