CSC570 Lecture 09

jgs CSC 570 Current Topics in Computer Science Applied Affective
Computing Lecture 09: Facial Emotion Recognition Dr. Javier Gonzalez-Sanchez javiergs@calpoly.edu www.javiergs.com Building 14 -227 Office Hours: By appointment

jgs Assignment 03

jgs Javier Gonzalez-Sanchez | CSC 309 | Winter 2023 |
3 Towards This Scenario

4 Architecture Bluetooth EMOTIV PRO Your App WebSocket (server) WebSocket (client) Your App WebGazer

5 Architecture EMOTIV PRO Your App WebSocket (server) WebSocket (client) Your App WebGazer EMOTIV Launcher

6 Simulator

7 Power On

8 Connect

9 Configuration

10 Data

11 WebSocket (in Java) https://mscjaviergs.medium.com WebSocket Programming with Java

jgs Face Emotion Recognition

13 Face

14 MindReader SDK

15 Face https://doi.org/10.1371/journal.pone.0251057

16 Breaking down facial expressions into individual muscle movements, called "action units" that can be combined to create a wide range of facial expressions. Facial Action Coding System, 46 actions (plus head movements). Facial Action Coding System (FACS)

17 Single Action Units

18 Less Precisely Defined AU 40 Sniff

19 § AU 1 (Inner Brow Raiser) involves the raising of the inner portion of the eyebrows, closest to the nose. This movement produces a vertical furrow between the eyebrows and is associated with emotions such as sadness, concern, and concentration. § AU 2 (Outer Brow Raiser) involves the raising of the outer portion of the eyebrows, further away from the nose. This movement produces a diagonal furrow along the forehead and is associated with emotions such as surprise, fear, and anger. § AU 3 (Brow Lowerer) involves the lowering of the eyebrows in the center of the forehead. This movement produces a vertical furrow between the eyebrows and is associated with emotions such as worry, frustration, and anger.

20 Information Given in FACS for Each Action Unit

21 Facial Action Coding System (FACS) Cheat Sheet+

22 Face (Ekman and Friesen 1978) Facial Action Coding System, 46 actions (plus head movements). (update 2022) 19 Lip Corner Depressor 26 Jaw Drop 27 Mouth Stretch

23 AU15 Lip Corner Depressor AU17 Chin Raiser

24

25 AU10 Upper Lip Raiser AU10 Upper Lip Raiser AU15 Lip Corner Depressor

26

27 AU10 Upper Lip Raiser AU17 Chin Raiser AU15 Lip Corner Depressor AU17 Chin Raiser

28 AU17

29 Action Units AU15 AU10 AU10 AU17 AU10 AU15 AU17 AU17 AU10 AU15 AU15 AU17

30 Face 30 30 frames per second 10 inferences per second 600 values per minute 36,000 values per hour

31

32 Emotion Recognition Ali Raza Shahid, Sheheryar Khan, Hong Yan. Contour and region harmonic features for sub-local facial expression recognition. Journal of Visual Communication and Image Representation. Volume 73. 2020. doi.org/10.1016/j.jvcir.2020.102949.

33 Emotions (Ekman)

34 MindReader SDK

35 MindReader SDK

36 MindReader SDK

37 MindReader SDK

38 MindReader SDK

39 Face Timestamp Agreement Concentrating Disagreement Interested Thinking Unsure 101116112838516 0.001836032 0.999917 1.79E-04 0.16485406 0.57114255 0.04595062 101116112838578 0.001447654 0.9999516 1.29E-04 0.16310683 0.5958921 0.042706452 101116112838672 5.97E-04 0 1.5E-04 0.44996294 0.45527613 0.00789697 101116112838766 2.46E-04 0 1.75E-04 0.77445686 0.32144752 0.001418217 101116112838860 1.01E-04 0 2.04E-04 0.93511915 0.21167138 2.53E-04 101116112838953 4.18E-05 0 2.38E-04 0.983739 0.13208677 4.52E-05 101116112839016 1.72E-05 0 2.78E-04 0.9960774 0.07941038 8.07E-06 101116112839110 7.1E-06 0 3.24E-04 0.99906266 0.046613157 1.44E-06 101116112839156 2.92E-06 0 3.77E-04 0.99977654 0.026964737 2.57E-07 101116112839250 1.21E-06 0 4.4E-04 0.9999467 0.015464196 4.58E-08 101116112839391 4.97E-07 0 5.12E-04 0.9999873 0.008824189 8.18E-09 101116112839438 2.05E-07 0 5.97E-04 0.999997 0.005020725 1.46E-09 101116112839547 8.43E-08 0 6.96E-04 0.9999993 0.002851939 2.6E-10 101116112839578 3.47E-08 0 8.11E-04 0.9999999 0.001618473 4.64E-11 101116112839688 1.43E-08 0 9.45E-04 0.99999994 9.18E-04 8.29E-12 101116112839781 5.9E-09 0 0.001101404 1 5.21E-04 1.48E-12 101116112839828 2.43E-09 0 0.001283521 1 2.95E-04 2.64E-13

jgs Programming

41 WebSocket (in Java) OpenCV?

42 Eye Tracking

43 § The Haar Cascade classifiers are pre-trained classifiers in OpenCV that are used for object detection. § OpenCV provides pre-trained Haar Cascade classifiers for a variety of objects, including faces, eyes, noses, and full bodies. § The name "Haar Cascade" comes from the Haar Wavelets used to encode image features that are used by the classifier. § Haar Wavelets are a type of mathematical function Definition

44 Let’s Work

45 Questions

46 Office Hours Tuesday and Thursday 3 - 5 pm But an appointment required Sent me an email – javiergs@calpoly.edu

jgs CSC 570 Applied Affective Computing Javier Gonzalez-Sanchez, Ph.D. javiergs@calpoly.edu
Spring 2023 Copyright. These slides can only be used as study material for the class CSC308 at Cal Poly. They cannot be distributed or used for another purpose.

CSC570 Lecture 09

CSC570 Lecture 09

More Decks by Javier Gonzalez-Sanchez

Other Decks in Research

Featured

Transcript