Dr. Javier Gonzalez-Sanchez [email protected] www.javiergs.info o ffi ce: 14 -227 CSC 570 Current Topics in Computer Science Applied Affective Computing Lecture 09. Gestures and Posture 

Facial Gestures 

Face 3 

MindReader SDK 4 

Face 5 https://doi.org/10.1371/journal.pone.0251057 

Facial Action Coding System (FACS) Bre a king down f a ci a l expressions into individu a l muscle movements, c a lled " a ction units" th a t c a n be combined to cre a te a wide r a nge of f a ci a l expressions. Facial Ac ti on Coding System, 46 ac ti ons (plus head movements). 6 

No content

40 Sniff 

Information Given in FACS for Each Action Unit 9 

F a ci a l Action Coding System (FACS) Che a t Sheet+ 10 

Face 11 (Ekman and Friesen 1978) Facial Action Coding System, 46 actions (plus head movements). (update 2022) 19 Lip Corner Depressor 26 Jaw Drop 27 Mouth Stretch 

12 AU15 Lip Corner Depressor AU17 Chin Raiser 

13 

14 AU10 Upper Lip Raiser AU10 Upper Lip Raiser AU15 Lip Corner Depressor 

15 

16 AU10 Upper Lip Raiser AU17 Chin Raiser AU15 Lip Corner Depressor AU17 Chin Raiser 

AU17 17 

AU15 AU10 AU10 AU17 AU10 AU15 AU17 AU17 AU10 AU15 AU15 AU17 

Face 19 30 frames per second 10 inferences per second 600 values per minute 36,000 values per hour 

No content

Emotion Recognition 21 Ali Raza Shahid, Sheheryar Khan, Hong Yan. Contour and region harmonic features for sub-local facial expression recognition. Journal of Visual Communication and Image Representation. Volume 73. 2020. doi.org/10.1016/j.jvcir.2020.102949. 

Emotions (Ekman) 22 

MindReader SDK 23 

MindReader SDK 24 

MindReader SDK 25 

MindReader SDK 26 

MindReader SDK 27 

Face 28 Timestamp Agreement Concentrating Disagreement Interested Thinking Unsure 101116112838516 0.001836032 0.999917 1.79E-04 0.16485406 0.57114255 0.04595062 101116112838578 0.001447654 0.9999516 1.29E-04 0.16310683 0.5958921 0.042706452 101116112838672 5.97E-04 0 1.5E-04 0.44996294 0.45527613 0.00789697 101116112838766 2.46E-04 0 1.75E-04 0.77445686 0.32144752 0.001418217 101116112838860 1.01E-04 0 2.04E-04 0.93511915 0.21167138 2.53E-04 101116112838953 4.18E-05 0 2.38E-04 0.983739 0.13208677 4.52E-05 101116112839016 1.72E-05 0 2.78E-04 0.9960774 0.07941038 8.07E-06 101116112839110 7.1E-06 0 3.24E-04 0.99906266 0.046613157 1.44E-06 101116112839156 2.92E-06 0 3.77E-04 0.99977654 0.026964737 2.57E-07 101116112839250 1.21E-06 0 4.4E-04 0.9999467 0.015464196 4.58E-08 101116112839391 4.97E-07 0 5.12E-04 0.9999873 0.008824189 8.18E-09 101116112839438 2.05E-07 0 5.97E-04 0.999997 0.005020725 1.46E-09 101116112839547 8.43E-08 0 6.96E-04 0.9999993 0.002851939 2.6E-10 101116112839578 3.47E-08 0 8.11E-04 0.9999999 0.001618473 4.64E-11 101116112839688 1.43E-08 0 9.45E-04 0.99999994 9.18E-04 8.29E-12 101116112839781 5.9E-09 0 0.001101404 1 5.21E-04 1.48E-12 101116112839828 2.43E-09 0 0.001283521 1 2.95E-04 2.64E-13 

Body Postures 

Embodiment • Inter a ction th a t involves the whole body a s a medium for eng a gement with digit a l environments. • Theoretic a l b a sis: physic a l sp a ce, a nd soci a l context in sh a ping hum a n inter a ctions • Immersive Computing, (VR/AR) - users feel fully present • A ff ective Computing - Physical actions c a n gener a te emotion a l st a tes; f a ci a l expressions can enh a nce soci a l a nd emotion a l eng a gement also • The Sensorimotor Loop: body’s movement provides feedback that shapes perception and decision-making. 30 

Body Tracking (Estimation) • Met a Quest devices do not h a ve built-in full-body tr a cking, but Met a introduced AI- b a sed body tr a cking solutions. • Upper-Body Estim a tion: Using h a nd tr a cking + he a d movement, the system infers the position of shoulders, elbows, a nd torso. • Inverse Kinem a tics (IK): AI predicts the position of hidden body p a rts (like elbows) b a sed on h a nd a nd he a d movement p a tterns. • VR a pplic a tions use IK models to simul a te full-body motion with limited tr a cking points. 31 

Body Tracking (Estimation) • No direct leg tr a cking (lower-body movements a re not n a tively c a ptured). • Uses AI inference to a pproxim a te w a lking a nd sitting poses. • Some VR a pps require extern a l tr a ckers (like Vive tr a ckers) or Kinect-like c a mer a s for full-body motion. • Extern a l a ccessories: w a ist a nd foot sensors for more precise tr a cking. 32 

Your Java Desktop Application 33 

MQTT Data { “leftEye”:{"x":-0.4216550588607788,"y":0.8787311911582947,"z":-0.00456150621175766}, “rightEye":{"x":-0.3755757808685303,"y":0.8756504058837891,"z":0.04438880831003189}, “leftEyeGaze":{"x":0.050619591027498248,"y":-0.0809454470872879,"z":0.9954323172569275}, “rightEyeGaze":{"x":0.050619591027498248,"y":-0.0809454470872879,"z":0.9954323172569275}, “eyeFixationPoint":{"x":0.11886614561080933,"y":-0.13097167015075684,"z":2.974684476852417}, “leftHand”:{"x":0.0,"y":0.0,"z":0.0}, "rightHand":{"x":0.0,"y":0.0,"z":0.0}, “cube":{"x":-0.5114021897315979,"y":1.5798050165176392,"z":0.024640535935759546}, “head":{"x":-0.7167978286743164,"y":0.8024232983589172,"z":0.17002606391906739}, “torso":{"x":-0.6404322385787964,"y":0.5270168781280518,"z":0.035430606454610828}, “leftFoot":{"x":-0.8061407804489136,"y":-0.16039752960205079,"z":0.25339341163635256}, “rightFoot":{"x":-0.5946151614189148,"y":-0.15849697589874268,"z":0.33175137639045718}, “hips":{"x":-0.6485552787780762,"y":0.33673161268234255,"z":0.0795457512140274}, “leftArmUp":{"x":-0.8079588413238525,"y":0.7046946287155151,"z":0.0354776531457901}, “lefArmLow":{"x":-0.6874216794967651,"y":0.5375530123710632,"z":-0.05098365247249603}, “rightArmUp":{"x":-0.5440698266029358,"y":0.7054383754730225,"z":0.16330549120903016}, “rightArmLow":{"x":-0.6227755546569824,"y":0.5135259032249451,"z":0.2464602291584015}, “leftWrist":{"x":-0.5440698266029358,"y":0.7054383754730225,"z":0.16330549120903016}, “rightWrist":{"x":-0.6227755546569824,"y":0.5135259032249451,"z":0.2464602291584015} } 34 

Body Input Action on a Java Swing Application 35 •Look left Move the circle left • Look right Move the circle right • Look up Move the circle up • Look down Move the circle down • Raise left hand Change circle color to red (e.g., “select”) • Raise right hand Change circle color to blue (e.g., “highlight”) • Lean forward (bend down) Shrink the circle (closer interaction) • Stand up straight Expand the circle (broader interaction) 

Questions 36 

Homework 37 Let us play with Postures and Body Movements 

CSC 570 Applied Affective Computing Javier Gonzalez-Sanchez, Ph.D. [email protected] Spring 2025 Copyright. These slides can only be used as study material for the class CSC 570 at Cal Poly. They cannot be distributed or used for another purpose.