OpenTalks.AI - Ольга Перепелкина, Human-centered AI: распознавание эмоций и физиологических сигналов по видео

Transcript

1. !1 HUMAN-CENTERED AI 2020 Olga Perepelkina, COO & Scientific Director
2. !2 ONE BILLION $ QUESTIONS What do people really think and feel? How do people make decisions, for example, about purchases? What do people want?
3. !3
4. !4 IMAGES ID Sex Age Emotions?
5. «READING» THE EMOTIONS
6. SOFTWARE • Affectiva: Affdex • CrowdEmotion: FaceVideo • Emotient: Facet • Microsoft: Cognitive Services • MorphCast: EmotionalTracking • Neurodatalab: EmotionRecognition (facial NN) • VicarVision: FaceReader • VisageTechnologies: FaceAnalysis DATABASES • BU-4DFE: 468 videos, posed • UT-Dallas: 470 videos, spontaneous https://psyarxiv.com/kzhds/
7. Classifiers VisageTechnologies (FaceAnalysis) VicarVision (FaceReader) Neurodatalab (EmotionRecognition) MorphCast (EmotionalTracking) Microsoft (Cognitive Services) Emotient (Facet) CrowdEmotion (Face Video) Affectiva (Affdex) Human Observers Accuracy, % 0 25 50 75
8. Classifiers VisageTechnologies (FaceAnalysis) VicarVision (FaceReader) Neurodatalab (EmotionRecognition) MorphCast (EmotionalTracking) Microsoft (Cognitive Services) Emotient (Facet) CrowdEmotion (Face Video) Affectiva (Affdex) Human Observers Accuracy, % 0 25 50 75
9. «READING» THE EMOTIONS ?
10. !10 WIN OR LOSE? Aviezer, H., Trope, Y. and Todorov, A. 2012
11. !11 LOSE WIN OR LOSE? LOSE WIN LOSE WIN WIN Aviezer, H., Trope, Y. and Todorov, A. 2012
12. !12 WIN OR LOSE? Aviezer, H., Trope, Y. and Todorov, A. 2012
13. !13 WIN OR LOSE? Aviezer, H., Trope, Y. and Todorov, A. 2012 LOSE WIN
14. !14 EMOTION != FACIAL EXPRESSION
 OTHER MODALITIES NEED TO BE CONSIDERED
15. !15 VIDEOS ID Sex Age Emotions
16. MULTIMODAL APPROACH !16
17. !17 A: audio, V: video, T: text, B: body 7 classes: Happy, Sad, Angry, Surprised, Anxious, Disgusted, Neutral
18. !18 VIDEOS ID Sex Age Emotions What else?
19. BEYOND 
 HUMAN PERCEPTION… !19
20. Lorem ipsum dolor sit amet consectetur !20 1
21. Lorem ipsum dolor sit amet consectetur !21 2
22. !22 VIDEOS ID Sex Age Emotions Heart rate
23. !23
24. !24 Video frames Diff frames 3D Attention CNN Heart Rate
25. !25
26. !26 Video frames Diff frames 3D Attention CNN Heart Rate Pretrain on synthetic data
27. !27 MAE: 6.71 RMSE: 10.63 MAE: 4.93 RMSE: 9.41
28. !28 Wearable MAE (Device) MAE (NDL) Honor Band 4 8.69 0.97 Amazfit Bip 3.51 3.92 Xiaomi Mi Band 3 5.07 3.71 Apple Watch 2 3.81 2.61 Garmin 2.03 2.46 Samsung Gear S3 2.03 3.28 Mean ± SD 3.6 ± 4.9 2.4 ± 4.5 Device - heart rate detected by wearables NDL - heart rate detected from video by Neurodata Lab
29. !29 VIDEOS ID Sex Age Emotions Heart rate What else?
30. !30
31. Lorem ipsum dolor sit amet consectetur !31
32. !32 MAE: 0.7 ECG-based (contact) Video-based (contactless) Breathing Rate
33. !33 1 2
34. !34 VIDEOS ID Sex Age Emotions Heart rate Breathing rate Eye Tracking
35. !35
36. !36 APPLICATIONS
37. !37 Male 33 years
38. !38 Male 33 years VALUES Offline targeting Higher conversion +5-125% Test faster & find out what works best Better optimization
41. !41 https://api.neurodatalab.dev/
42. !42 THANK YOU 2020 Olga Perepelkina o.perepelkina@neurodatala