$30 off During Our Annual Pro Sale. View Details »

SER594 Lecture 02

SER594 Lecture 02

Human Computer Interaction
Interfaces I
(201901)

Javier Gonzalez-Sanchez
PRO

April 18, 2019
Tweet

More Decks by Javier Gonzalez-Sanchez

Other Decks in Programming

Transcript

  1. SER594
    Human-Computer Interaction
    Lecture 02
    Human-Computer Interfaces
    Javier Gonzalez-Sanchez, PhD
    [email protected]
    javiergs.engineering.asu.edu
    Office Hours: By appointment

    View Slide

  2. Humans in HCI

    View Slide

  3. Humans in HCI

    View Slide

  4. Interface
    § Physical and functional connection between two independent devices or
    systems.

    View Slide

  5. Multimodality

    View Slide

  6. Brain – Computer Interfaces
    1

    View Slide

  7. Brain-Computer Interfaces (BCI)
    • It is a particular type of a physiological instrument that uses brainwaves
    as information sources (electrical activity along the scalp produced by the
    firing of neurons within the brain).
    • Emotiv © EPOC headset [5] device will be used to show how to collect
    and work with this kind of data.
    Emotiv - Brain Computer Interface Technology. Retrieved April 26, 2011, from http://www.emotiv.com

    View Slide

  8. BCI
    • Wireless Emotiv® EEG Headset.
    • The device reports data with intervals of 125 ms (8 Hz).
    • The raw data output includes 14 values (7 channels on each brain hemisphere: AF3,
    F7, F3, FC5, T7, P7, O1, O2, P8, T8, FC6, F4, F8, and AF4) and two values of the
    acceleration of the head when leaning (gyrox and gyroy).
    • The affectiv suite reports 5 emotions: engagement, boredom, excitement, frustration,
    and meditation.
    • And the expressiv suite reports facial gestures: blink, wink (left and right), look (left
    and right), raise brow, furrow brow, smile, clench, smirk (left and right), and laugh.

    View Slide

  9. BCI
    • Electrodes are situated and labeled according with the CMS/DRL configuration [6][7]
    [6] Sharbrough F, Chatrian G-E, Lesser RP, Lüders H, Nuwer M, Picton TW. American Electroencephalographic Society Guidelines for Standard Electrode Position
    Nomenclature. J. Clin. Neurophysiol 8: 200-2.
    [7] Electroencephalography. Retrieved November 14th, 2010, from Electric and Magnetic Measurement of the Electric Activity of Neural Tissue:
    www.bem.fi/book/13/13.htm

    View Slide

  10. Emotiv Systems
    $299
    emotions EEG data facial gestures
    BCI

    View Slide

  11. Demo
    Wireless Emotiv® EEG Headset
    BCI

    View Slide

  12. Affectiv Suite

    View Slide

  13. Expressiv Suite

    View Slide

  14. BCI Data
    Field Description Values
    Timestamp
    It is the timestamp (date and time) of the computer running the
    system. It could be used to synchronize the data with other
    sensors.
    Format "yymmddhhmmssSSS"
    (y - year, m - month, d - day, h - hour, m - minutes, s -
    seconds, S - milliseconds).
    UserID Identifies the user. An integer value.
    Wireless Signal Status Shows the strength of the signal. The value is from 0 to 4, being 4 the best one.
    Blink, Wink Left and Right,
    Look Left and Right, Raise
    Brow, Furrow, Smile,
    Clench, Smirk Left and
    Right, Laugh
    Part of the expressive suite.
    Values between 0 and 1, being 1 the value that represents
    the highest power/probability for this emotion.
    Short Term and Long Term
    Excitement, Engagement /
    Boredom, Meditation,
    Frustration
    Part of the affective suite.
    Values between 0 and 1, being 1 the value that represents
    the highest power/probability for this emotion.
    AF3, F7, F3, FC5, T7, P7,
    O1, O2, P8, T8, FC6, F4, F8,
    AF4.
    Raw data coming from each of the 14 channels. The name of these
    fields where defined according with the CMS/DRL configuration
    [XXX][XXXX].
    Values from 4000 and higher.
    GyroX and GyroY
    Information about how the head moves/accelerates according with
    X and Y axis accordingly.

    View Slide

  15. Raw Data
    Timestamp AF3 F7 F3 FC5 T7 P7 O1 O2 P8 T8 FC6 F4 F8 AF4 AccX AccY
    101116112544901 4542.05 4831.79 4247.18 4690.26 4282.56 4395.38 4591.79 4569.23 4360 4570.77 4297.44 4311.28 4282.56 4367.18 1660 2003
    101116112544901 4536.92 4802.05 4243.08 4673.85 4272.31 4393.33 4592.82 4570.26 4354.87 4570.26 4292.31 4309.74 4277.95 4370.77 1658 2002
    101116112545010 4533.33 4798.97 4234.87 4669.74 4301.03 4396.92 4592.31 4570.77 4351.28 4561.03 4281.54 4301.54 4271.28 4363.59 1659 2003
    101116112545010 4549.23 4839.49 4241.03 4691.28 4333.85 4397.95 4596.41 4567.18 4355.9 4556.41 4286.15 4306.15 4277.95 4369.74 1659 2003
    101116112545010 4580 4865.64 4251.79 4710.26 4340 4401.54 4603.59 4572.82 4360 4558.46 4298.97 4324.62 4296.41 4395.9 1657 2004
    101116112545010 4597.44 4860 4252.82 4705.64 4350.26 4412.31 4603.59 4577.44 4357.44 4555.9 4295.38 4329.23 4296.41 4414.36 1656 2005
    101116112545010 4584.62 4847.69 4246.67 4690.26 4360 4409.23 4597.44 4569.74 4351.79 4549.74 4278.97 4316.92 4272.82 4399.49 1656 2006
    101116112545010 4566.15 4842.05 4238.46 4684.1 4322.05 4389.74 4592.82 4566.67 4351.79 4549.74 4274.36 4310.26 4262.05 4370.77 1655 2005
    101116112545010 4563.59 4844.62 4231.79 4687.69 4267.69 4387.69 4594.36 4580 4361.03 4556.41 4278.97 4310.77 4274.36 4370.77 1653 2006
    101116112545010 4567.18 4847.18 4233.33 4688.72 4285.13 4409.23 4602.05 4589.23 4368.21 4560 4280.51 4310.77 4281.54 4390.26 1655 2004
    101116112545010 4570.26 4846.67 4234.87 4683.08 4323.08 4415.9 4604.1 4585.64 4366.67 4557.44 4277.95 4310.26 4273.33 4384.1 1652 2005
    101116112545010 4569.23 4842.56 4233.85 4678.46 4310.77 4402.56 4598.97 4583.08 4364.1 4553.85 4277.44 4310.26 4271.28 4372.31 1654 2005
    101116112545010 4558.46 4832.82 4234.87 4676.92 4301.03 4389.74 4595.38 4590.26 4368.72 4556.92 4280 4310.26 4276.92 4380 1653 2004
    101116112545010 4555.9 4831.79 4233.33 4679.49 4314.36 4390.26 4597.95 4598.97 4374.87 4562.56 4280.51 4311.28 4280 4386.15 1653 2004
    101116112545010 4569.74 4842.56 4232.82 4684.1 4303.59 4405.64 4609.74 4600 4378.46 4567.18 4278.97 4313.33 4280 4382.05 1653 2002
    101116112545010 4574.36 4846.67 4235.38 4683.08 4293.33 4416.41 4619.49 4604.1 4382.56 4570.77 4280.51 4310.77 4282.05 4382.05 1652 2002
    101116112545010 4562.05 4840.51 4227.18 4673.85 4300 4405.13 4611.28 4601.03 4376.41 4561.54 4280 4303.59 4279.49 4374.87 1652 2000

    View Slide

  16. Expressiv Data
    Timestamp ID Signal Blink Wink L Wink R Look L Look R Eyebrow Furrow Smile Clench Smirk L Smirk R Laugh
    101116091145065 0 2 0 0 0 1 0 0 0 0 0 0 0.988563 0
    101116091145190 0 2 0 0 0 1 0 0 0 0.45465 0 0 0 0
    101116091145315 0 2 0 0 0 1 0 0 0 0.467005 0 0 0 0
    101116091145440 0 2 0 0 0 1 0 0 0 0.401006 0 0 0 0
    101116091145565 0 2 0 0 0 1 0 0 0 0.248671 0 0 0 0
    101116091145690 0 2 0 0 0 1 0 0 0 0.173023 0 0 0 0
    101116091145815 0 2 0 0 0 1 0 0 0 0.162788 0 0 0 0
    101116091145940 0 2 0 0 0 1 0 0 0 0.156485 0 0 0 0
    101116091146065 0 2 0 0 0 1 0 0 0 0 0 0.776925 0 0
    101116091146190 0 2 0 0 0 1 0 0 0 0 0 0 0.608679 0
    101116091146315 0 2 0 0 0 1 0 0 0 0 0 0 0.342364 0
    101116091146440 0 2 0 0 0 1 0 0 0 0 0 0 0.149695 0
    101116091146565 0 2 0 0 0 1 0 0 0 0 0 0 0.0864399 0
    101116091146690 0 2 0 0 0 1 0 0 0 0 0 0 0.0733481 0
    101116091146815 0 2 0 0 0 1 0 0 0 0 0 0 0.118965 0
    101116091146941 0 2 0 0 0 1 0 0 0 0 0 0 0.259171 0

    View Slide

  17. Affectiv Data
    Timestamp Short Term Excitement Long Term Excitement Engagement Meditation Frustration
    101116091145065 0.447595 0.54871 0.834476 0.333844 0.536197
    101116091145190 0.447595 0.54871 0.834476 0.333844 0.536197
    101116091145315 0.447595 0.54871 0.834476 0.333844 0.536197
    101116091145440 0.487864 0.546877 0.834146 0.339548 0.54851
    101116091145565 0.487864 0.546877 0.834146 0.339548 0.54851
    101116091145690 0.487864 0.546877 0.834146 0.339548 0.54851
    101116091145815 0.487864 0.546877 0.834146 0.339548 0.54851
    101116091145940 0.521663 0.545609 0.839321 0.348321 0.558228
    101116091146065 0.521663 0.545609 0.839321 0.348321 0.558228
    101116091146190 0.521663 0.545609 0.839321 0.348321 0.558228
    101116091146315 0.521663 0.545609 0.839321 0.348321 0.558228
    101116091146440 0.509297 0.544131 0.84401 0.358717 0.546771
    101116091146565 0.509297 0.544131 0.84401 0.358717 0.546771
    101116091146690 0.509297 0.544131 0.84401 0.358717 0.546771
    101116091146815 0.509297 0.544131 0.84401 0.358717 0.546771
    101116091146941 0.451885 0.541695 0.848087 0.368071 0.533919

    View Slide

  18. Face-Based Emotion Recognition
    2

    View Slide

  19. Face-Based Recognition
    • Face-based emotion recognition systems
    • These systems infer affective states by capturing images of the users’
    facial expressions and head movements.
    • We are going to show the capabilities of face-based emotion recognition
    systems using a simple 30 fps USB webcam and software from MIT
    Media Lab [8].
    [8] R. E. Kaliouby and P. Robinson, “Real-Time Inference of Complex Mental States from Facial Expressions and Head Gestures,” Proc. Conference on
    Computer Vision and Pattern Recognition Workshop (CVPRW ‘04), IEEE Computer Society, June 2004, Volume 10, p. 154.

    View Slide

  20. Face-Based Recognition
    § MindReader API that enables the real time analysis, tagging and inference
    of cognitive affective mental states from facial video in real-time.
    § This framework combines vision-based processing of the face with
    predictions of mental state models to interpret the meaning underlying
    head and facial signals overtime.
    § It provides results at intervals of 100 ms approximately (10 Hz).
    § With this system it is possible to infer emotions such as: agreeing,
    concentrating, disagreeing, interested, thinking, and unsure.
    (Ekman and Friesen 1978) – Facial Action Coding System, 46 actions (plus head movements).

    View Slide

  21. Face-Based Recognition

    View Slide

  22. Mind Reader

    View Slide

  23. Mind Reader
    Field Description Values
    Timestamp
    It is the timestamp (date and time) of the computer running the
    system. It could be used to synchronize the data with other
    sensors.
    Format "yymmddhhmmssSSS"
    (y - year, m - month, d - day, h - hour, m - minutes, s -
    seconds, S - milliseconds).
    Agreement,
    Concentrating,
    Disagreement, Interested,
    Thinking, Unsure
    This value shows the probability of this emotion being present on
    the user at a particular time (frame).
    This value is between 0 to 1.
    If the value is -1 it means it was not possible to define an
    emotion.
    This happen the user's face is out of the camera focus.

    View Slide

  24. Mind Reader
    Timestamp Agreement Concentrating Disagreement Interested Thinking Unsure
    101116112838516 0.001836032 0.999917 1.79E-04 0.16485406 0.57114255 0.04595062
    101116112838578 0.001447654 0.9999516 1.29E-04 0.16310683 0.5958921 0.042706452
    101116112838672 5.97E-04 0 1.5E-04 0.44996294 0.45527613 0.00789697
    101116112838766 2.46E-04 0 1.75E-04 0.77445686 0.32144752 0.001418217
    101116112838860 1.01E-04 0 2.04E-04 0.93511915 0.21167138 2.53E-04
    101116112838953 4.18E-05 0 2.38E-04 0.983739 0.13208677 4.52E-05
    101116112839016 1.72E-05 0 2.78E-04 0.9960774 0.07941038 8.07E-06
    101116112839110 7.1E-06 0 3.24E-04 0.99906266 0.046613157 1.44E-06
    101116112839156 2.92E-06 0 3.77E-04 0.99977654 0.026964737 2.57E-07
    101116112839250 1.21E-06 0 4.4E-04 0.9999467 0.015464196 4.58E-08
    101116112839391 4.97E-07 0 5.12E-04 0.9999873 0.008824189 8.18E-09
    101116112839438 2.05E-07 0 5.97E-04 0.999997 0.005020725 1.46E-09
    101116112839547 8.43E-08 0 6.96E-04 0.9999993 0.002851939 2.6E-10
    101116112839578 3.47E-08 0 8.11E-04 0.9999999 0.001618473 4.64E-11
    101116112839688 1.43E-08 0 9.45E-04 0.99999994 9.18E-04 8.29E-12
    101116112839781 5.9E-09 0 0.001101404 1 5.21E-04 1.48E-12
    101116112839828 2.43E-09 0 0.001283521 1 2.95E-04 2.64E-13

    View Slide

  25. Eye Tracking Systems
    3
    2
    5

    View Slide

  26. Eye Tracking System
    • Eye-tracking systems
    • These are instruments that measure eyes position and eyes movement in
    order to detect zones in which the user has particular interest in a specific
    time and moment.
    • Datasets from Tobii © Eye-tracking system [9] data will be shown.
    [9] Tobii Technology - Eye Tracking and Eye Control. Retrieved April 26, 2011, from http://www.tobii.com.

    View Slide

  27. Eye Tracking System
    • Tobii® Eye Tracker.
    • The device reports data with intervals of 100 ms (10Hz).
    • The output provides data concerning attention direction (gaze-x, gaze-y),
    duration of fixation, and pupil dilation.

    View Slide

  28. Eye Tracking System
    Demo
    Tobii® Eye Tracker

    View Slide

  29. Eye Tracking System
    Field Description Values
    Timestamp
    It is the timestamp (date and time) of the computer running the
    system. It could be used to synchronize the data with other
    sensors.
    Format "yymmddhhmmssSSS"
    (y - year, m - month, d - day, h - hour, m - minutes, s -
    seconds, S - milliseconds).
    GazePoint X
    The horizontal screen position for either eye or the average for both
    eyes. This value is also used for the fixation definition.
    0 is the left edge, the maximum value of the horizontal
    screen resolution is the right edge.
    GazePoint Y
    The vertical screen position for either eye or the average for both
    eyes. This value is also used for the fixation definition.
    0 is the bottom edge, the maximum value of the vertical
    screen resolution is the right edge.
    Pupil Left Pupil size (left eye) in mm. Variates
    Validity Left Validity of the gaze data.
    0 to 4. 0 if the eye is found and the tracking quality good. If
    the eye cannot be found by the eye tracker, the validity code
    will be 4.
    Pupil Right Pupil size (left eye) in mm. Variates
    Validity Right Validity of the gaze data.
    0 to 4. 0 if the eye is found and the tracking quality good. If
    the eye cannot be found by the eye tracker, the validity code
    will be 4.
    FixationDuration Fixation duration. The time in milliseconds that a fixation lasts. Variates.
    Event Events, automatic and logged, will show up under Event. Variates.
    AOI
    Areas Of Interests if fixations on multiple AOIs are to be written on
    the same row.
    Variates.

    View Slide

  30. Eye Tracking System
    Timestamp GPX GPY Pupil Left Validity L Pupil Right Validity R Fixation Event AOI
    101124162405582 636 199 2.759313 0 2.88406 0 48 Content
    101124162405599 641 207 2.684893 0 2.855817 0 48 Content
    101124162405615 659 211 2.624458 0 2.903861 0 48 Content
    101124162405632 644 201 2.636186 0 2.916132 0 48 Content
    101124162405649 644 213 2.690685 0 2.831013 0 48 Content
    101124162405666 628 194 2.651784 0 2.869714 0 48 Content
    101124162405682 614 177 2.829281 0 2.899828 0 48 Content
    101124162405699 701 249 2.780344 0 2.907665 0 49 Content
    101124162405716 906 341 2.853761 0 2.916398 0 49 Content
    101124162405732 947 398 2.829427 0 2.889944 0 49 Content
    101124162405749 941 400 2.826602 0 2.881179 0 49 Content
    101124162405766 938 403 2.78699 0 2.87948 0 49 KeyPress Content
    101124162405782 937 411 2.803387 0 2.821803 0 49 Content
    101124162405799 934 397 2.819166 0 2.871547 0 49 Content
    101124162405816 941 407 2.811687 0 2.817927 0 49 Content
    101124162405832 946 405 2.857419 0 2.857427 0 49 Content
    101124162405849 0 0 -1 4 -1 4 49 Content

    View Slide

  31. Visualization
    BCI and Gaze Points engagement
    This figure shows the engagement fixation points of expert player playing in expert-mode. The size of the circle represents the duration of the fixation in that point, while the level of shading
    represents the intensity of the emotion.

    View Slide

  32. Visualization
    BCI and Gaze Points frustration
    This figure shows the frustration fixation points of expert player playing in expert-mode. The size of the circle represents the duration of the fixation in that point, while the level of shading
    represents the intensity of the emotion.

    View Slide

  33. Visualization
    BCI and Gaze Points boredom
    This figure shows the boredom fixation points of expert player playing in expert-mode. The size of the circle represents the duration of the fixation in that point, while the level of shading
    represents the intensity of the emotion.

    View Slide

  34. Visualization
    BCI and Gaze Points engagement
    This figures shows the engagement gaze points (above a threshold of 0.6) of a user reading a material with seductive details (i.e. cartoons).
    For this user the text on the bottom part of the first column was engaging.

    View Slide

  35. Visualization
    BCI and Gaze Points frustration
    This figure shows the frustration gaze points (above a threshold of 0.6) of a user reading a material with seductive details (i.e. cartoons).
    Looking the cartoon is related with high frustration level.

    View Slide

  36. Visualization
    BCI and Gaze Points boredom
    This figure shows the boredom gaze points (above a threshold of 0.5) of a user while reading a material with seductive details (i.e. cartoons).
    Notice that the text in the middle part of the second column of that page was boring.

    View Slide

  37. Galvanic Skin Conductance Sensor
    4

    View Slide

  38. Galvanic Skin Conductance

    View Slide

  39. Galvanic Skin Conductance
    • Provide information on the activity of physiological functions of an
    individual.
    • Arousal detection. Measures the electrical conductance of the skin, which
    varies with its moisture level that depends on the sweat glands, which are
    controlled by the sympathetic, and parasympathetic nervous systems.
    [10]
    • Hardware designed by MIT Media Lab.
    • It is a Wireless Bluetooth device that reports data in intervals of 500 ms
    approximately (2Hz)
    [10] M. Strauss, C. Reynolds, S. Hughes, K. Park, G. McDarby, and R.W. Picard, “The HandWave Bluetooth Skin Conductance Sensor,” Proc. First
    International Conference on Affective Computing and Intelligent Interaction (ACII 05), Springer-Verlang, Oct. 2005, pp. 699-706,
    doi:10.1007/11573548_90.

    View Slide

  40. Galvanic Skin Conductance

    View Slide

  41. Galvanic Skin Conductance
    Field Description Values
    Timestamp
    It is the timestamp (date and time) of the computer running the
    system. It could be used to synchronize the data with other
    sensors.
    Format "yymmddhhmmssSSS"
    (y - year, m - month, d - day, h - hour, m - minutes, s -
    seconds, S - milliseconds).
    Battery Voltage Level of the battery voltage. 0 - 3 Volts
    Conductance Level of arousal. 0 - 3 Volts

    View Slide

  42. Galvanic Skin Conductance
    Timestamp Voltage Conductance
    101116101332262 2.482352941 1.030696176
    101116101332762 2.482352941 1.023404165
    101116101333262 2.482352941 1.019813274
    101116101333762 2.482352941 1.041657802
    101116101334247 2.482352941 0.998280273
    101116101334747 2.482352941 0.991181142
    101116101335247 2.482352941 0.980592229
    101116101335747 2.482352941 0.998280273
    101116101336247 2.482352941 1.012586294
    101116101336762 2.482352941 1.012586294
    101116101337231 2.482352941 1.012586294
    101116101337747 2.482352941 1.009008251
    101116101338247 2.482352941 0.998280273
    101116101338747 2.482352941 0.991181142
    101116101339247 2.482352941 0.987628521
    101116101339731 2.482352941 0.987628521
    101116101340231 2.482352941 0.980592229

    View Slide

  43. Homework
    Read the Papers posted on Blackboard about
    examples of Affect-Driven Adaptive Systems

    View Slide

  44. SER594 – Human Computer Interaction
    Javier Gonzalez-Sanchez
    [email protected]
    Spring 2019
    Disclaimer. These slides can only be used as study material for the SER594 course at ASU.
    They cannot be distributed or used for another purpose.

    View Slide