Upgrade to Pro — share decks privately, control downloads, hide ads and more …

CSC570 Lecture 09

CSC570 Lecture 09

Applied Affective Computing
Face-Based Emotion Recognition
(202305)

Javier Gonzalez-Sanchez
PRO

May 02, 2023
Tweet

More Decks by Javier Gonzalez-Sanchez

Other Decks in Research

Transcript

  1. jgs
    CSC 570
    Current Topics in Computer Science
    Applied Affective Computing
    Lecture 09:
    Facial Emotion Recognition
    Dr. Javier Gonzalez-Sanchez
    [email protected]
    www.javiergs.com
    Building 14 -227
    Office Hours: By appointment

    View Slide

  2. jgs
    Assignment 03

    View Slide

  3. jgs
    Javier Gonzalez-Sanchez | CSC 309 | Winter 2023 | 3
    Towards This Scenario

    View Slide

  4. jgs
    Javier Gonzalez-Sanchez | CSC 309 | Winter 2023 | 4
    Architecture
    Bluetooth
    EMOTIV PRO Your App
    WebSocket
    (server)
    WebSocket
    (client)
    Your App
    WebGazer

    View Slide

  5. jgs
    Javier Gonzalez-Sanchez | CSC 309 | Winter 2023 | 5
    Architecture
    EMOTIV PRO Your App
    WebSocket
    (server)
    WebSocket
    (client)
    Your App
    WebGazer
    EMOTIV Launcher

    View Slide

  6. jgs
    Javier Gonzalez-Sanchez | CSC 309 | Winter 2023 | 6
    Simulator

    View Slide

  7. jgs
    Javier Gonzalez-Sanchez | CSC 309 | Winter 2023 | 7
    Power On

    View Slide

  8. jgs
    Javier Gonzalez-Sanchez | CSC 309 | Winter 2023 | 8
    Connect

    View Slide

  9. jgs
    Javier Gonzalez-Sanchez | CSC 309 | Winter 2023 | 9
    Configuration

    View Slide

  10. jgs
    Javier Gonzalez-Sanchez | CSC 309 | Winter 2023 | 10
    Data

    View Slide

  11. jgs
    Javier Gonzalez-Sanchez | CSC 309 | Winter 2023 | 11
    WebSocket (in Java)
    https://mscjaviergs.medium.com
    WebSocket Programming with Java

    View Slide

  12. jgs
    Face Emotion Recognition

    View Slide

  13. jgs
    Javier Gonzalez-Sanchez | CSC 309 | Winter 2023 | 13
    Face

    View Slide

  14. jgs
    Javier Gonzalez-Sanchez | CSC 309 | Winter 2023 | 14
    MindReader SDK

    View Slide

  15. jgs
    Javier Gonzalez-Sanchez | CSC 309 | Winter 2023 | 15
    Face
    https://doi.org/10.1371/journal.pone.0251057

    View Slide

  16. jgs
    Javier Gonzalez-Sanchez | CSC 309 | Winter 2023 | 16
    Breaking down facial expressions into individual muscle movements,
    called "action units"
    that can be combined to create a wide range of facial expressions.
    Facial Action Coding System,
    46 actions (plus head movements).
    Facial Action Coding System (FACS)

    View Slide

  17. jgs
    Javier Gonzalez-Sanchez | CSC 309 | Winter 2023 | 17
    Single Action Units

    View Slide

  18. jgs
    Javier Gonzalez-Sanchez | CSC 309 | Winter 2023 | 18
    Less Precisely Defined AU
    40 Sniff

    View Slide

  19. jgs
    Javier Gonzalez-Sanchez | CSC 309 | Winter 2023 | 19
    § AU 1 (Inner Brow Raiser) involves the raising of the inner portion of the
    eyebrows, closest to the nose. This movement produces a vertical furrow
    between the eyebrows and is associated with emotions such as sadness,
    concern, and concentration.
    § AU 2 (Outer Brow Raiser) involves the raising of the outer portion of the
    eyebrows, further away from the nose. This movement produces a diagonal
    furrow along the forehead and is associated with emotions such as surprise,
    fear, and anger.
    § AU 3 (Brow Lowerer) involves the lowering of the eyebrows in the center of
    the forehead. This movement produces a vertical furrow between the
    eyebrows and is associated with emotions such as worry, frustration, and
    anger.

    View Slide

  20. jgs
    Javier Gonzalez-Sanchez | CSC 309 | Winter 2023 | 20
    Information Given in FACS for Each Action Unit

    View Slide

  21. jgs
    Javier Gonzalez-Sanchez | CSC 309 | Winter 2023 | 21
    Facial Action Coding System (FACS) Cheat Sheet+

    View Slide

  22. jgs
    Javier Gonzalez-Sanchez | CSC 309 | Winter 2023 | 22
    Face
    (Ekman and Friesen 1978)
    Facial Action Coding System,
    46 actions (plus head movements).
    (update 2022)
    19 Lip Corner
    Depressor
    26 Jaw Drop
    27 Mouth Stretch

    View Slide

  23. jgs
    Javier Gonzalez-Sanchez | CSC 309 | Winter 2023 | 23
    AU15 Lip Corner Depressor
    AU17 Chin Raiser

    View Slide

  24. jgs
    Javier Gonzalez-Sanchez | CSC 309 | Winter 2023 | 24

    View Slide

  25. jgs
    Javier Gonzalez-Sanchez | CSC 309 | Winter 2023 | 25
    AU10 Upper Lip Raiser
    AU10 Upper Lip Raiser
    AU15 Lip Corner Depressor

    View Slide

  26. jgs
    Javier Gonzalez-Sanchez | CSC 309 | Winter 2023 | 26

    View Slide

  27. jgs
    Javier Gonzalez-Sanchez | CSC 309 | Winter 2023 | 27
    AU10 Upper Lip Raiser
    AU17 Chin Raiser
    AU15 Lip Corner Depressor
    AU17 Chin Raiser

    View Slide

  28. jgs
    Javier Gonzalez-Sanchez | CSC 309 | Winter 2023 | 28
    AU17

    View Slide

  29. jgs
    Javier Gonzalez-Sanchez | CSC 309 | Winter 2023 | 29
    Action Units
    AU15
    AU10
    AU10
    AU17
    AU10
    AU15
    AU17
    AU17
    AU10
    AU15
    AU15
    AU17

    View Slide

  30. jgs
    Javier Gonzalez-Sanchez | CSC 309 | Winter 2023 | 30
    Face
    30
    30 frames per second
    10 inferences per second
    600 values per minute
    36,000 values per hour

    View Slide

  31. jgs
    Javier Gonzalez-Sanchez | CSC 309 | Winter 2023 | 31

    View Slide

  32. jgs
    Javier Gonzalez-Sanchez | CSC 309 | Winter 2023 | 32
    Emotion Recognition
    Ali Raza Shahid, Sheheryar Khan, Hong Yan. Contour and region harmonic features
    for sub-local facial expression recognition. Journal of Visual Communication and
    Image Representation. Volume 73. 2020. doi.org/10.1016/j.jvcir.2020.102949.

    View Slide

  33. jgs
    Javier Gonzalez-Sanchez | CSC 309 | Winter 2023 | 33
    Emotions (Ekman)

    View Slide

  34. jgs
    Javier Gonzalez-Sanchez | CSC 309 | Winter 2023 | 34
    MindReader SDK

    View Slide

  35. jgs
    Javier Gonzalez-Sanchez | CSC 309 | Winter 2023 | 35
    MindReader SDK

    View Slide

  36. jgs
    Javier Gonzalez-Sanchez | CSC 309 | Winter 2023 | 36
    MindReader SDK

    View Slide

  37. jgs
    Javier Gonzalez-Sanchez | CSC 309 | Winter 2023 | 37
    MindReader SDK

    View Slide

  38. jgs
    Javier Gonzalez-Sanchez | CSC 309 | Winter 2023 | 38
    MindReader SDK

    View Slide

  39. jgs
    Javier Gonzalez-Sanchez | CSC 309 | Winter 2023 | 39
    Face
    Timestamp Agreement Concentrating Disagreement Interested Thinking Unsure
    101116112838516 0.001836032 0.999917 1.79E-04 0.16485406 0.57114255 0.04595062
    101116112838578 0.001447654 0.9999516 1.29E-04 0.16310683 0.5958921 0.042706452
    101116112838672 5.97E-04 0 1.5E-04 0.44996294 0.45527613 0.00789697
    101116112838766 2.46E-04 0 1.75E-04 0.77445686 0.32144752 0.001418217
    101116112838860 1.01E-04 0 2.04E-04 0.93511915 0.21167138 2.53E-04
    101116112838953 4.18E-05 0 2.38E-04 0.983739 0.13208677 4.52E-05
    101116112839016 1.72E-05 0 2.78E-04 0.9960774 0.07941038 8.07E-06
    101116112839110 7.1E-06 0 3.24E-04 0.99906266 0.046613157 1.44E-06
    101116112839156 2.92E-06 0 3.77E-04 0.99977654 0.026964737 2.57E-07
    101116112839250 1.21E-06 0 4.4E-04 0.9999467 0.015464196 4.58E-08
    101116112839391 4.97E-07 0 5.12E-04 0.9999873 0.008824189 8.18E-09
    101116112839438 2.05E-07 0 5.97E-04 0.999997 0.005020725 1.46E-09
    101116112839547 8.43E-08 0 6.96E-04 0.9999993 0.002851939 2.6E-10
    101116112839578 3.47E-08 0 8.11E-04 0.9999999 0.001618473 4.64E-11
    101116112839688 1.43E-08 0 9.45E-04 0.99999994 9.18E-04 8.29E-12
    101116112839781 5.9E-09 0 0.001101404 1 5.21E-04 1.48E-12
    101116112839828 2.43E-09 0 0.001283521 1 2.95E-04 2.64E-13

    View Slide

  40. jgs
    Programming

    View Slide

  41. jgs
    Javier Gonzalez-Sanchez | CSC 309 | Winter 2023 | 41
    WebSocket (in Java)
    OpenCV?

    View Slide

  42. jgs
    Javier Gonzalez-Sanchez | CSC 309 | Winter 2023 | 42
    Eye Tracking

    View Slide

  43. jgs
    Javier Gonzalez-Sanchez | CSC 309 | Winter 2023 | 43
    § The Haar Cascade classifiers are pre-trained classifiers in OpenCV that are used for
    object detection.
    § OpenCV provides pre-trained Haar Cascade classifiers for a variety of objects,
    including faces, eyes, noses, and full bodies.
    § The name "Haar Cascade" comes from the Haar Wavelets used to encode image
    features that are used by the classifier.
    § Haar Wavelets are a type of mathematical function
    Definition

    View Slide

  44. jgs
    Javier Gonzalez-Sanchez | CSC 309 | Winter 2023 | 44
    Let’s Work

    View Slide

  45. jgs
    Javier Gonzalez-Sanchez | CSC 309 | Winter 2023 | 45
    Questions

    View Slide

  46. jgs
    Javier Gonzalez-Sanchez | CSC 309 | Winter 2023 | 46
    Office Hours
    Tuesday and Thursday 3 - 5 pm
    But an appointment required
    Sent me an email – [email protected]

    View Slide

  47. jgs
    CSC 570 Applied Affective Computing
    Javier Gonzalez-Sanchez, Ph.D.
    [email protected]
    Spring 2023
    Copyright. These slides can only be used as study material for the class CSC308 at Cal Poly.
    They cannot be distributed or used for another purpose.

    View Slide