Leveraging Data Science to Advance the Scientific Study of Smiling

Leveraging Data Science to Advance the Scientific Study of Smiling

Brownbag (Chalk) Talk Slides from Jeffrey M. Girard, PhD (Carnegie Mellon University)

D5982f56a0e8f345a32780c546e90a87?s=128

Jeffrey M Girard

January 28, 2020
Tweet

Transcript

  1. Leveraging Data Science to Advance the Scientific Study of Smiling

    bit.ly/girard2020b
  2. Data Science in Psychology • • • • • •

    • • • •
  3. Data Science in Psychology Solutions (Hammers) • • • •

    • • • • • • • • • Challenges (Nails) • • • • • • • • • • • • •
  4. Presentation Roadmap

  5. Study 1: Web-Scraped Celebrity Smiles

  6. Web-Scraped Celebrity Smiles: Design • • • • • •

  7. Web-Scraped Celebrity Smiles: Collect/Process http://bit.ly/celebs_by_nation

  8. Web-Scraped Celebrity Smiles: Collect/Process

  9. Web-Scraped Celebrity Smiles: Process

  10. Web-Scraped Celebrity Smiles: Validate Measure Algorithm Positive Smile Positive Rating

    0.79 Smile Rating 0.78 0.94 Expert FACS 0.87 0.97 0.94 Measure ICC(A,5) 95% CI Positive Rating 0.90 [0.88, 0.92] Smile Rating 0.90 [0.88, 0.92]
  11. Web-Scraped Celebrity Smiles: Model • • • • • •

    • • Zero-Inflated Beta Regression ZI Beta = 0 ∈ (0, 1)
  12. Web-Scraped Celebrity Smiles: Model/Visualize

  13. Web-Scraped Celebrity Smiles: Visualize

  14. Study 2: Spontaneous Emotion Database

  15. Spontaneous Emotion Database: Design • • • •

  16. Spontaneous Emotion Database: Design • • • •

  17. Spontaneous Emotion Database: Design/Collect Emotion Rating Scale Afraid/Scared 0 1

    2 3 4 5 Angry/Upset 0 1 2 3 4 5 Disgusted 0 1 2 3 4 5 … … Other (write in) 0 1 2 3 4 5
  18. Spontaneous Emotion Database: Collect/Visualize

  19. Spontaneous Emotion Database: Validate/Visualize

  20. Spontaneous Emotion Database: Collect AU 1 + 10 + 12

    + 15 + 17 + 64 Action Unit Occurrence Did the action unit happen? {Absent = 0, 1 = Present} Action Unit Intensity How strong was the action unit? {Trace = 1, 2, 3, 4, 5 = Maximum}
  21. Spontaneous Emotion Database: Collect/Process AU12 AU12 AU06 2 2 3

    4 4 4 2 4 2 2 2 2 2 2 3 3 3 3 3 0 0 1 1 0 0 0 1 1 1 1 1 1 0 0 1 1 1 1 1 1 1 1 1 1 1 Action Unit Occurrence: {0=Absent, 1=Present} Action Unit Intensity: {1=Trace, 2=Slight, 3=Pronounced, 4=Extreme, 5=Maximum}
  22. Spontaneous Emotion Database: Collect

  23. Spontaneous Emotion Database: Validate agreement github.com/jmgirard/agreement mreliability.jmgirard.com 0 1 1

    1 0 0 1 1 0 1 0 1
  24. Study 3: Algorithmic Smile Interpretation

  25. Algorithmic Smile Interpretation: Design • • • •

  26. Algorithmic Smile Interpretation: Process Example Smile Event Video Clips Fr

    AU12 AU6 1 3 0 2 3 3 3 4 2 4 0 0 5 2 0 6 2 0 Frs AU12i AU6o AU6i 1 to 3 4 1 3 5 to 6 2 0 0
  27. Algorithmic Smile Interpretation: Model Smile Appearance Reported Emotion Algorithm

  28. Algorithmic Smile Interpretation: Model Support Vector Machine Random Forest Multilayer

    Perceptron
  29. Algorithmic Smile Interpretation: Validate

  30. Algorithmic Smile Interpretation: Visualize

  31. Discussion Questions • • •