Slide 1

Slide 1 text

SER594 Human-Computer Interaction Lecture 02 Human-Computer Interfaces Javier Gonzalez-Sanchez, PhD [email protected] javiergs.engineering.asu.edu Office Hours: By appointment

Slide 2

Slide 2 text

Humans in HCI

Slide 3

Slide 3 text

Humans in HCI

Slide 4

Slide 4 text

Interface § Physical and functional connection between two independent devices or systems.

Slide 5

Slide 5 text

Multimodality

Slide 6

Slide 6 text

Brain – Computer Interfaces 1

Slide 7

Slide 7 text

Brain-Computer Interfaces (BCI) • It is a particular type of a physiological instrument that uses brainwaves as information sources (electrical activity along the scalp produced by the firing of neurons within the brain). • Emotiv © EPOC headset [5] device will be used to show how to collect and work with this kind of data. Emotiv - Brain Computer Interface Technology. Retrieved April 26, 2011, from http://www.emotiv.com

Slide 8

Slide 8 text

BCI • Wireless Emotiv® EEG Headset. • The device reports data with intervals of 125 ms (8 Hz). • The raw data output includes 14 values (7 channels on each brain hemisphere: AF3, F7, F3, FC5, T7, P7, O1, O2, P8, T8, FC6, F4, F8, and AF4) and two values of the acceleration of the head when leaning (gyrox and gyroy). • The affectiv suite reports 5 emotions: engagement, boredom, excitement, frustration, and meditation. • And the expressiv suite reports facial gestures: blink, wink (left and right), look (left and right), raise brow, furrow brow, smile, clench, smirk (left and right), and laugh.

Slide 9

Slide 9 text

BCI • Electrodes are situated and labeled according with the CMS/DRL configuration [6][7] [6] Sharbrough F, Chatrian G-E, Lesser RP, Lüders H, Nuwer M, Picton TW. American Electroencephalographic Society Guidelines for Standard Electrode Position Nomenclature. J. Clin. Neurophysiol 8: 200-2. [7] Electroencephalography. Retrieved November 14th, 2010, from Electric and Magnetic Measurement of the Electric Activity of Neural Tissue: www.bem.fi/book/13/13.htm

Slide 10

Slide 10 text

Emotiv Systems $299 emotions EEG data facial gestures BCI

Slide 11

Slide 11 text

Demo Wireless Emotiv® EEG Headset BCI

Slide 12

Slide 12 text

Affectiv Suite

Slide 13

Slide 13 text

Expressiv Suite

Slide 14

Slide 14 text

BCI Data Field Description Values Timestamp It is the timestamp (date and time) of the computer running the system. It could be used to synchronize the data with other sensors. Format "yymmddhhmmssSSS" (y - year, m - month, d - day, h - hour, m - minutes, s - seconds, S - milliseconds). UserID Identifies the user. An integer value. Wireless Signal Status Shows the strength of the signal. The value is from 0 to 4, being 4 the best one. Blink, Wink Left and Right, Look Left and Right, Raise Brow, Furrow, Smile, Clench, Smirk Left and Right, Laugh Part of the expressive suite. Values between 0 and 1, being 1 the value that represents the highest power/probability for this emotion. Short Term and Long Term Excitement, Engagement / Boredom, Meditation, Frustration Part of the affective suite. Values between 0 and 1, being 1 the value that represents the highest power/probability for this emotion. AF3, F7, F3, FC5, T7, P7, O1, O2, P8, T8, FC6, F4, F8, AF4. Raw data coming from each of the 14 channels. The name of these fields where defined according with the CMS/DRL configuration [XXX][XXXX]. Values from 4000 and higher. GyroX and GyroY Information about how the head moves/accelerates according with X and Y axis accordingly.

Slide 15

Slide 15 text

Raw Data Timestamp AF3 F7 F3 FC5 T7 P7 O1 O2 P8 T8 FC6 F4 F8 AF4 AccX AccY 101116112544901 4542.05 4831.79 4247.18 4690.26 4282.56 4395.38 4591.79 4569.23 4360 4570.77 4297.44 4311.28 4282.56 4367.18 1660 2003 101116112544901 4536.92 4802.05 4243.08 4673.85 4272.31 4393.33 4592.82 4570.26 4354.87 4570.26 4292.31 4309.74 4277.95 4370.77 1658 2002 101116112545010 4533.33 4798.97 4234.87 4669.74 4301.03 4396.92 4592.31 4570.77 4351.28 4561.03 4281.54 4301.54 4271.28 4363.59 1659 2003 101116112545010 4549.23 4839.49 4241.03 4691.28 4333.85 4397.95 4596.41 4567.18 4355.9 4556.41 4286.15 4306.15 4277.95 4369.74 1659 2003 101116112545010 4580 4865.64 4251.79 4710.26 4340 4401.54 4603.59 4572.82 4360 4558.46 4298.97 4324.62 4296.41 4395.9 1657 2004 101116112545010 4597.44 4860 4252.82 4705.64 4350.26 4412.31 4603.59 4577.44 4357.44 4555.9 4295.38 4329.23 4296.41 4414.36 1656 2005 101116112545010 4584.62 4847.69 4246.67 4690.26 4360 4409.23 4597.44 4569.74 4351.79 4549.74 4278.97 4316.92 4272.82 4399.49 1656 2006 101116112545010 4566.15 4842.05 4238.46 4684.1 4322.05 4389.74 4592.82 4566.67 4351.79 4549.74 4274.36 4310.26 4262.05 4370.77 1655 2005 101116112545010 4563.59 4844.62 4231.79 4687.69 4267.69 4387.69 4594.36 4580 4361.03 4556.41 4278.97 4310.77 4274.36 4370.77 1653 2006 101116112545010 4567.18 4847.18 4233.33 4688.72 4285.13 4409.23 4602.05 4589.23 4368.21 4560 4280.51 4310.77 4281.54 4390.26 1655 2004 101116112545010 4570.26 4846.67 4234.87 4683.08 4323.08 4415.9 4604.1 4585.64 4366.67 4557.44 4277.95 4310.26 4273.33 4384.1 1652 2005 101116112545010 4569.23 4842.56 4233.85 4678.46 4310.77 4402.56 4598.97 4583.08 4364.1 4553.85 4277.44 4310.26 4271.28 4372.31 1654 2005 101116112545010 4558.46 4832.82 4234.87 4676.92 4301.03 4389.74 4595.38 4590.26 4368.72 4556.92 4280 4310.26 4276.92 4380 1653 2004 101116112545010 4555.9 4831.79 4233.33 4679.49 4314.36 4390.26 4597.95 4598.97 4374.87 4562.56 4280.51 4311.28 4280 4386.15 1653 2004 101116112545010 4569.74 4842.56 4232.82 4684.1 4303.59 4405.64 4609.74 4600 4378.46 4567.18 4278.97 4313.33 4280 4382.05 1653 2002 101116112545010 4574.36 4846.67 4235.38 4683.08 4293.33 4416.41 4619.49 4604.1 4382.56 4570.77 4280.51 4310.77 4282.05 4382.05 1652 2002 101116112545010 4562.05 4840.51 4227.18 4673.85 4300 4405.13 4611.28 4601.03 4376.41 4561.54 4280 4303.59 4279.49 4374.87 1652 2000

Slide 16

Slide 16 text

Expressiv Data Timestamp ID Signal Blink Wink L Wink R Look L Look R Eyebrow Furrow Smile Clench Smirk L Smirk R Laugh 101116091145065 0 2 0 0 0 1 0 0 0 0 0 0 0.988563 0 101116091145190 0 2 0 0 0 1 0 0 0 0.45465 0 0 0 0 101116091145315 0 2 0 0 0 1 0 0 0 0.467005 0 0 0 0 101116091145440 0 2 0 0 0 1 0 0 0 0.401006 0 0 0 0 101116091145565 0 2 0 0 0 1 0 0 0 0.248671 0 0 0 0 101116091145690 0 2 0 0 0 1 0 0 0 0.173023 0 0 0 0 101116091145815 0 2 0 0 0 1 0 0 0 0.162788 0 0 0 0 101116091145940 0 2 0 0 0 1 0 0 0 0.156485 0 0 0 0 101116091146065 0 2 0 0 0 1 0 0 0 0 0 0.776925 0 0 101116091146190 0 2 0 0 0 1 0 0 0 0 0 0 0.608679 0 101116091146315 0 2 0 0 0 1 0 0 0 0 0 0 0.342364 0 101116091146440 0 2 0 0 0 1 0 0 0 0 0 0 0.149695 0 101116091146565 0 2 0 0 0 1 0 0 0 0 0 0 0.0864399 0 101116091146690 0 2 0 0 0 1 0 0 0 0 0 0 0.0733481 0 101116091146815 0 2 0 0 0 1 0 0 0 0 0 0 0.118965 0 101116091146941 0 2 0 0 0 1 0 0 0 0 0 0 0.259171 0

Slide 17

Slide 17 text

Affectiv Data Timestamp Short Term Excitement Long Term Excitement Engagement Meditation Frustration 101116091145065 0.447595 0.54871 0.834476 0.333844 0.536197 101116091145190 0.447595 0.54871 0.834476 0.333844 0.536197 101116091145315 0.447595 0.54871 0.834476 0.333844 0.536197 101116091145440 0.487864 0.546877 0.834146 0.339548 0.54851 101116091145565 0.487864 0.546877 0.834146 0.339548 0.54851 101116091145690 0.487864 0.546877 0.834146 0.339548 0.54851 101116091145815 0.487864 0.546877 0.834146 0.339548 0.54851 101116091145940 0.521663 0.545609 0.839321 0.348321 0.558228 101116091146065 0.521663 0.545609 0.839321 0.348321 0.558228 101116091146190 0.521663 0.545609 0.839321 0.348321 0.558228 101116091146315 0.521663 0.545609 0.839321 0.348321 0.558228 101116091146440 0.509297 0.544131 0.84401 0.358717 0.546771 101116091146565 0.509297 0.544131 0.84401 0.358717 0.546771 101116091146690 0.509297 0.544131 0.84401 0.358717 0.546771 101116091146815 0.509297 0.544131 0.84401 0.358717 0.546771 101116091146941 0.451885 0.541695 0.848087 0.368071 0.533919

Slide 18

Slide 18 text

Face-Based Emotion Recognition 2

Slide 19

Slide 19 text

Face-Based Recognition • Face-based emotion recognition systems • These systems infer affective states by capturing images of the users’ facial expressions and head movements. • We are going to show the capabilities of face-based emotion recognition systems using a simple 30 fps USB webcam and software from MIT Media Lab [8]. [8] R. E. Kaliouby and P. Robinson, “Real-Time Inference of Complex Mental States from Facial Expressions and Head Gestures,” Proc. Conference on Computer Vision and Pattern Recognition Workshop (CVPRW ‘04), IEEE Computer Society, June 2004, Volume 10, p. 154.

Slide 20

Slide 20 text

Face-Based Recognition § MindReader API that enables the real time analysis, tagging and inference of cognitive affective mental states from facial video in real-time. § This framework combines vision-based processing of the face with predictions of mental state models to interpret the meaning underlying head and facial signals overtime. § It provides results at intervals of 100 ms approximately (10 Hz). § With this system it is possible to infer emotions such as: agreeing, concentrating, disagreeing, interested, thinking, and unsure. (Ekman and Friesen 1978) – Facial Action Coding System, 46 actions (plus head movements).

Slide 21

Slide 21 text

Face-Based Recognition

Slide 22

Slide 22 text

Mind Reader

Slide 23

Slide 23 text

Mind Reader Field Description Values Timestamp It is the timestamp (date and time) of the computer running the system. It could be used to synchronize the data with other sensors. Format "yymmddhhmmssSSS" (y - year, m - month, d - day, h - hour, m - minutes, s - seconds, S - milliseconds). Agreement, Concentrating, Disagreement, Interested, Thinking, Unsure This value shows the probability of this emotion being present on the user at a particular time (frame). This value is between 0 to 1. If the value is -1 it means it was not possible to define an emotion. This happen the user's face is out of the camera focus.

Slide 24

Slide 24 text

Mind Reader Timestamp Agreement Concentrating Disagreement Interested Thinking Unsure 101116112838516 0.001836032 0.999917 1.79E-04 0.16485406 0.57114255 0.04595062 101116112838578 0.001447654 0.9999516 1.29E-04 0.16310683 0.5958921 0.042706452 101116112838672 5.97E-04 0 1.5E-04 0.44996294 0.45527613 0.00789697 101116112838766 2.46E-04 0 1.75E-04 0.77445686 0.32144752 0.001418217 101116112838860 1.01E-04 0 2.04E-04 0.93511915 0.21167138 2.53E-04 101116112838953 4.18E-05 0 2.38E-04 0.983739 0.13208677 4.52E-05 101116112839016 1.72E-05 0 2.78E-04 0.9960774 0.07941038 8.07E-06 101116112839110 7.1E-06 0 3.24E-04 0.99906266 0.046613157 1.44E-06 101116112839156 2.92E-06 0 3.77E-04 0.99977654 0.026964737 2.57E-07 101116112839250 1.21E-06 0 4.4E-04 0.9999467 0.015464196 4.58E-08 101116112839391 4.97E-07 0 5.12E-04 0.9999873 0.008824189 8.18E-09 101116112839438 2.05E-07 0 5.97E-04 0.999997 0.005020725 1.46E-09 101116112839547 8.43E-08 0 6.96E-04 0.9999993 0.002851939 2.6E-10 101116112839578 3.47E-08 0 8.11E-04 0.9999999 0.001618473 4.64E-11 101116112839688 1.43E-08 0 9.45E-04 0.99999994 9.18E-04 8.29E-12 101116112839781 5.9E-09 0 0.001101404 1 5.21E-04 1.48E-12 101116112839828 2.43E-09 0 0.001283521 1 2.95E-04 2.64E-13

Slide 25

Slide 25 text

Eye Tracking Systems 3 2 5

Slide 26

Slide 26 text

Eye Tracking System • Eye-tracking systems • These are instruments that measure eyes position and eyes movement in order to detect zones in which the user has particular interest in a specific time and moment. • Datasets from Tobii © Eye-tracking system [9] data will be shown. [9] Tobii Technology - Eye Tracking and Eye Control. Retrieved April 26, 2011, from http://www.tobii.com.

Slide 27

Slide 27 text

Eye Tracking System • Tobii® Eye Tracker. • The device reports data with intervals of 100 ms (10Hz). • The output provides data concerning attention direction (gaze-x, gaze-y), duration of fixation, and pupil dilation.

Slide 28

Slide 28 text

Eye Tracking System Demo Tobii® Eye Tracker

Slide 29

Slide 29 text

Eye Tracking System Field Description Values Timestamp It is the timestamp (date and time) of the computer running the system. It could be used to synchronize the data with other sensors. Format "yymmddhhmmssSSS" (y - year, m - month, d - day, h - hour, m - minutes, s - seconds, S - milliseconds). GazePoint X The horizontal screen position for either eye or the average for both eyes. This value is also used for the fixation definition. 0 is the left edge, the maximum value of the horizontal screen resolution is the right edge. GazePoint Y The vertical screen position for either eye or the average for both eyes. This value is also used for the fixation definition. 0 is the bottom edge, the maximum value of the vertical screen resolution is the right edge. Pupil Left Pupil size (left eye) in mm. Variates Validity Left Validity of the gaze data. 0 to 4. 0 if the eye is found and the tracking quality good. If the eye cannot be found by the eye tracker, the validity code will be 4. Pupil Right Pupil size (left eye) in mm. Variates Validity Right Validity of the gaze data. 0 to 4. 0 if the eye is found and the tracking quality good. If the eye cannot be found by the eye tracker, the validity code will be 4. FixationDuration Fixation duration. The time in milliseconds that a fixation lasts. Variates. Event Events, automatic and logged, will show up under Event. Variates. AOI Areas Of Interests if fixations on multiple AOIs are to be written on the same row. Variates.

Slide 30

Slide 30 text

Eye Tracking System Timestamp GPX GPY Pupil Left Validity L Pupil Right Validity R Fixation Event AOI 101124162405582 636 199 2.759313 0 2.88406 0 48 Content 101124162405599 641 207 2.684893 0 2.855817 0 48 Content 101124162405615 659 211 2.624458 0 2.903861 0 48 Content 101124162405632 644 201 2.636186 0 2.916132 0 48 Content 101124162405649 644 213 2.690685 0 2.831013 0 48 Content 101124162405666 628 194 2.651784 0 2.869714 0 48 Content 101124162405682 614 177 2.829281 0 2.899828 0 48 Content 101124162405699 701 249 2.780344 0 2.907665 0 49 Content 101124162405716 906 341 2.853761 0 2.916398 0 49 Content 101124162405732 947 398 2.829427 0 2.889944 0 49 Content 101124162405749 941 400 2.826602 0 2.881179 0 49 Content 101124162405766 938 403 2.78699 0 2.87948 0 49 KeyPress Content 101124162405782 937 411 2.803387 0 2.821803 0 49 Content 101124162405799 934 397 2.819166 0 2.871547 0 49 Content 101124162405816 941 407 2.811687 0 2.817927 0 49 Content 101124162405832 946 405 2.857419 0 2.857427 0 49 Content 101124162405849 0 0 -1 4 -1 4 49 Content

Slide 31

Slide 31 text

Visualization BCI and Gaze Points engagement This figure shows the engagement fixation points of expert player playing in expert-mode. The size of the circle represents the duration of the fixation in that point, while the level of shading represents the intensity of the emotion.

Slide 32

Slide 32 text

Visualization BCI and Gaze Points frustration This figure shows the frustration fixation points of expert player playing in expert-mode. The size of the circle represents the duration of the fixation in that point, while the level of shading represents the intensity of the emotion.

Slide 33

Slide 33 text

Visualization BCI and Gaze Points boredom This figure shows the boredom fixation points of expert player playing in expert-mode. The size of the circle represents the duration of the fixation in that point, while the level of shading represents the intensity of the emotion.

Slide 34

Slide 34 text

Visualization BCI and Gaze Points engagement This figures shows the engagement gaze points (above a threshold of 0.6) of a user reading a material with seductive details (i.e. cartoons). For this user the text on the bottom part of the first column was engaging.

Slide 35

Slide 35 text

Visualization BCI and Gaze Points frustration This figure shows the frustration gaze points (above a threshold of 0.6) of a user reading a material with seductive details (i.e. cartoons). Looking the cartoon is related with high frustration level.

Slide 36

Slide 36 text

Visualization BCI and Gaze Points boredom This figure shows the boredom gaze points (above a threshold of 0.5) of a user while reading a material with seductive details (i.e. cartoons). Notice that the text in the middle part of the second column of that page was boring.

Slide 37

Slide 37 text

Galvanic Skin Conductance Sensor 4

Slide 38

Slide 38 text

Galvanic Skin Conductance

Slide 39

Slide 39 text

Galvanic Skin Conductance • Provide information on the activity of physiological functions of an individual. • Arousal detection. Measures the electrical conductance of the skin, which varies with its moisture level that depends on the sweat glands, which are controlled by the sympathetic, and parasympathetic nervous systems. [10] • Hardware designed by MIT Media Lab. • It is a Wireless Bluetooth device that reports data in intervals of 500 ms approximately (2Hz) [10] M. Strauss, C. Reynolds, S. Hughes, K. Park, G. McDarby, and R.W. Picard, “The HandWave Bluetooth Skin Conductance Sensor,” Proc. First International Conference on Affective Computing and Intelligent Interaction (ACII 05), Springer-Verlang, Oct. 2005, pp. 699-706, doi:10.1007/11573548_90.

Slide 40

Slide 40 text

Galvanic Skin Conductance

Slide 41

Slide 41 text

Galvanic Skin Conductance Field Description Values Timestamp It is the timestamp (date and time) of the computer running the system. It could be used to synchronize the data with other sensors. Format "yymmddhhmmssSSS" (y - year, m - month, d - day, h - hour, m - minutes, s - seconds, S - milliseconds). Battery Voltage Level of the battery voltage. 0 - 3 Volts Conductance Level of arousal. 0 - 3 Volts

Slide 42

Slide 42 text

Galvanic Skin Conductance Timestamp Voltage Conductance 101116101332262 2.482352941 1.030696176 101116101332762 2.482352941 1.023404165 101116101333262 2.482352941 1.019813274 101116101333762 2.482352941 1.041657802 101116101334247 2.482352941 0.998280273 101116101334747 2.482352941 0.991181142 101116101335247 2.482352941 0.980592229 101116101335747 2.482352941 0.998280273 101116101336247 2.482352941 1.012586294 101116101336762 2.482352941 1.012586294 101116101337231 2.482352941 1.012586294 101116101337747 2.482352941 1.009008251 101116101338247 2.482352941 0.998280273 101116101338747 2.482352941 0.991181142 101116101339247 2.482352941 0.987628521 101116101339731 2.482352941 0.987628521 101116101340231 2.482352941 0.980592229

Slide 43

Slide 43 text

Homework Read the Papers posted on Blackboard about examples of Affect-Driven Adaptive Systems

Slide 44

Slide 44 text

SER594 – Human Computer Interaction Javier Gonzalez-Sanchez [email protected] Spring 2019 Disclaimer. These slides can only be used as study material for the SER594 course at ASU. They cannot be distributed or used for another purpose.