Save 37% off PRO during our Black Friday Sale! »

SER594 Lecture 02

SER594 Lecture 02

Human Computer Interaction
Interfaces I
(201901)

B546a9b97d993392e4b22b74b99b91fe?s=128

Javier Gonzalez
PRO

April 18, 2019
Tweet

Transcript

  1. SER594 Human-Computer Interaction Lecture 02 Human-Computer Interfaces Javier Gonzalez-Sanchez, PhD

    javiergs@asu.edu javiergs.engineering.asu.edu Office Hours: By appointment
  2. Humans in HCI

  3. Humans in HCI

  4. Interface § Physical and functional connection between two independent devices

    or systems.
  5. Multimodality

  6. Brain – Computer Interfaces 1

  7. Brain-Computer Interfaces (BCI) • It is a particular type of

    a physiological instrument that uses brainwaves as information sources (electrical activity along the scalp produced by the firing of neurons within the brain). • Emotiv © EPOC headset [5] device will be used to show how to collect and work with this kind of data. Emotiv - Brain Computer Interface Technology. Retrieved April 26, 2011, from http://www.emotiv.com
  8. BCI • Wireless Emotiv® EEG Headset. • The device reports

    data with intervals of 125 ms (8 Hz). • The raw data output includes 14 values (7 channels on each brain hemisphere: AF3, F7, F3, FC5, T7, P7, O1, O2, P8, T8, FC6, F4, F8, and AF4) and two values of the acceleration of the head when leaning (gyrox and gyroy). • The affectiv suite reports 5 emotions: engagement, boredom, excitement, frustration, and meditation. • And the expressiv suite reports facial gestures: blink, wink (left and right), look (left and right), raise brow, furrow brow, smile, clench, smirk (left and right), and laugh.
  9. BCI • Electrodes are situated and labeled according with the

    CMS/DRL configuration [6][7] [6] Sharbrough F, Chatrian G-E, Lesser RP, Lüders H, Nuwer M, Picton TW. American Electroencephalographic Society Guidelines for Standard Electrode Position Nomenclature. J. Clin. Neurophysiol 8: 200-2. [7] Electroencephalography. Retrieved November 14th, 2010, from Electric and Magnetic Measurement of the Electric Activity of Neural Tissue: www.bem.fi/book/13/13.htm
  10. Emotiv Systems $299 emotions EEG data facial gestures BCI

  11. Demo Wireless Emotiv® EEG Headset BCI

  12. Affectiv Suite

  13. Expressiv Suite

  14. BCI Data Field Description Values Timestamp It is the timestamp

    (date and time) of the computer running the system. It could be used to synchronize the data with other sensors. Format "yymmddhhmmssSSS" (y - year, m - month, d - day, h - hour, m - minutes, s - seconds, S - milliseconds). UserID Identifies the user. An integer value. Wireless Signal Status Shows the strength of the signal. The value is from 0 to 4, being 4 the best one. Blink, Wink Left and Right, Look Left and Right, Raise Brow, Furrow, Smile, Clench, Smirk Left and Right, Laugh Part of the expressive suite. Values between 0 and 1, being 1 the value that represents the highest power/probability for this emotion. Short Term and Long Term Excitement, Engagement / Boredom, Meditation, Frustration Part of the affective suite. Values between 0 and 1, being 1 the value that represents the highest power/probability for this emotion. AF3, F7, F3, FC5, T7, P7, O1, O2, P8, T8, FC6, F4, F8, AF4. Raw data coming from each of the 14 channels. The name of these fields where defined according with the CMS/DRL configuration [XXX][XXXX]. Values from 4000 and higher. GyroX and GyroY Information about how the head moves/accelerates according with X and Y axis accordingly.
  15. Raw Data Timestamp AF3 F7 F3 FC5 T7 P7 O1

    O2 P8 T8 FC6 F4 F8 AF4 AccX AccY 101116112544901 4542.05 4831.79 4247.18 4690.26 4282.56 4395.38 4591.79 4569.23 4360 4570.77 4297.44 4311.28 4282.56 4367.18 1660 2003 101116112544901 4536.92 4802.05 4243.08 4673.85 4272.31 4393.33 4592.82 4570.26 4354.87 4570.26 4292.31 4309.74 4277.95 4370.77 1658 2002 101116112545010 4533.33 4798.97 4234.87 4669.74 4301.03 4396.92 4592.31 4570.77 4351.28 4561.03 4281.54 4301.54 4271.28 4363.59 1659 2003 101116112545010 4549.23 4839.49 4241.03 4691.28 4333.85 4397.95 4596.41 4567.18 4355.9 4556.41 4286.15 4306.15 4277.95 4369.74 1659 2003 101116112545010 4580 4865.64 4251.79 4710.26 4340 4401.54 4603.59 4572.82 4360 4558.46 4298.97 4324.62 4296.41 4395.9 1657 2004 101116112545010 4597.44 4860 4252.82 4705.64 4350.26 4412.31 4603.59 4577.44 4357.44 4555.9 4295.38 4329.23 4296.41 4414.36 1656 2005 101116112545010 4584.62 4847.69 4246.67 4690.26 4360 4409.23 4597.44 4569.74 4351.79 4549.74 4278.97 4316.92 4272.82 4399.49 1656 2006 101116112545010 4566.15 4842.05 4238.46 4684.1 4322.05 4389.74 4592.82 4566.67 4351.79 4549.74 4274.36 4310.26 4262.05 4370.77 1655 2005 101116112545010 4563.59 4844.62 4231.79 4687.69 4267.69 4387.69 4594.36 4580 4361.03 4556.41 4278.97 4310.77 4274.36 4370.77 1653 2006 101116112545010 4567.18 4847.18 4233.33 4688.72 4285.13 4409.23 4602.05 4589.23 4368.21 4560 4280.51 4310.77 4281.54 4390.26 1655 2004 101116112545010 4570.26 4846.67 4234.87 4683.08 4323.08 4415.9 4604.1 4585.64 4366.67 4557.44 4277.95 4310.26 4273.33 4384.1 1652 2005 101116112545010 4569.23 4842.56 4233.85 4678.46 4310.77 4402.56 4598.97 4583.08 4364.1 4553.85 4277.44 4310.26 4271.28 4372.31 1654 2005 101116112545010 4558.46 4832.82 4234.87 4676.92 4301.03 4389.74 4595.38 4590.26 4368.72 4556.92 4280 4310.26 4276.92 4380 1653 2004 101116112545010 4555.9 4831.79 4233.33 4679.49 4314.36 4390.26 4597.95 4598.97 4374.87 4562.56 4280.51 4311.28 4280 4386.15 1653 2004 101116112545010 4569.74 4842.56 4232.82 4684.1 4303.59 4405.64 4609.74 4600 4378.46 4567.18 4278.97 4313.33 4280 4382.05 1653 2002 101116112545010 4574.36 4846.67 4235.38 4683.08 4293.33 4416.41 4619.49 4604.1 4382.56 4570.77 4280.51 4310.77 4282.05 4382.05 1652 2002 101116112545010 4562.05 4840.51 4227.18 4673.85 4300 4405.13 4611.28 4601.03 4376.41 4561.54 4280 4303.59 4279.49 4374.87 1652 2000
  16. Expressiv Data Timestamp ID Signal Blink Wink L Wink R

    Look L Look R Eyebrow Furrow Smile Clench Smirk L Smirk R Laugh 101116091145065 0 2 0 0 0 1 0 0 0 0 0 0 0.988563 0 101116091145190 0 2 0 0 0 1 0 0 0 0.45465 0 0 0 0 101116091145315 0 2 0 0 0 1 0 0 0 0.467005 0 0 0 0 101116091145440 0 2 0 0 0 1 0 0 0 0.401006 0 0 0 0 101116091145565 0 2 0 0 0 1 0 0 0 0.248671 0 0 0 0 101116091145690 0 2 0 0 0 1 0 0 0 0.173023 0 0 0 0 101116091145815 0 2 0 0 0 1 0 0 0 0.162788 0 0 0 0 101116091145940 0 2 0 0 0 1 0 0 0 0.156485 0 0 0 0 101116091146065 0 2 0 0 0 1 0 0 0 0 0 0.776925 0 0 101116091146190 0 2 0 0 0 1 0 0 0 0 0 0 0.608679 0 101116091146315 0 2 0 0 0 1 0 0 0 0 0 0 0.342364 0 101116091146440 0 2 0 0 0 1 0 0 0 0 0 0 0.149695 0 101116091146565 0 2 0 0 0 1 0 0 0 0 0 0 0.0864399 0 101116091146690 0 2 0 0 0 1 0 0 0 0 0 0 0.0733481 0 101116091146815 0 2 0 0 0 1 0 0 0 0 0 0 0.118965 0 101116091146941 0 2 0 0 0 1 0 0 0 0 0 0 0.259171 0
  17. Affectiv Data Timestamp Short Term Excitement Long Term Excitement Engagement

    Meditation Frustration 101116091145065 0.447595 0.54871 0.834476 0.333844 0.536197 101116091145190 0.447595 0.54871 0.834476 0.333844 0.536197 101116091145315 0.447595 0.54871 0.834476 0.333844 0.536197 101116091145440 0.487864 0.546877 0.834146 0.339548 0.54851 101116091145565 0.487864 0.546877 0.834146 0.339548 0.54851 101116091145690 0.487864 0.546877 0.834146 0.339548 0.54851 101116091145815 0.487864 0.546877 0.834146 0.339548 0.54851 101116091145940 0.521663 0.545609 0.839321 0.348321 0.558228 101116091146065 0.521663 0.545609 0.839321 0.348321 0.558228 101116091146190 0.521663 0.545609 0.839321 0.348321 0.558228 101116091146315 0.521663 0.545609 0.839321 0.348321 0.558228 101116091146440 0.509297 0.544131 0.84401 0.358717 0.546771 101116091146565 0.509297 0.544131 0.84401 0.358717 0.546771 101116091146690 0.509297 0.544131 0.84401 0.358717 0.546771 101116091146815 0.509297 0.544131 0.84401 0.358717 0.546771 101116091146941 0.451885 0.541695 0.848087 0.368071 0.533919
  18. Face-Based Emotion Recognition 2

  19. Face-Based Recognition • Face-based emotion recognition systems • These systems

    infer affective states by capturing images of the users’ facial expressions and head movements. • We are going to show the capabilities of face-based emotion recognition systems using a simple 30 fps USB webcam and software from MIT Media Lab [8]. [8] R. E. Kaliouby and P. Robinson, “Real-Time Inference of Complex Mental States from Facial Expressions and Head Gestures,” Proc. Conference on Computer Vision and Pattern Recognition Workshop (CVPRW ‘04), IEEE Computer Society, June 2004, Volume 10, p. 154.
  20. Face-Based Recognition § MindReader API that enables the real time

    analysis, tagging and inference of cognitive affective mental states from facial video in real-time. § This framework combines vision-based processing of the face with predictions of mental state models to interpret the meaning underlying head and facial signals overtime. § It provides results at intervals of 100 ms approximately (10 Hz). § With this system it is possible to infer emotions such as: agreeing, concentrating, disagreeing, interested, thinking, and unsure. (Ekman and Friesen 1978) – Facial Action Coding System, 46 actions (plus head movements).
  21. Face-Based Recognition

  22. Mind Reader

  23. Mind Reader Field Description Values Timestamp It is the timestamp

    (date and time) of the computer running the system. It could be used to synchronize the data with other sensors. Format "yymmddhhmmssSSS" (y - year, m - month, d - day, h - hour, m - minutes, s - seconds, S - milliseconds). Agreement, Concentrating, Disagreement, Interested, Thinking, Unsure This value shows the probability of this emotion being present on the user at a particular time (frame). This value is between 0 to 1. If the value is -1 it means it was not possible to define an emotion. This happen the user's face is out of the camera focus.
  24. Mind Reader Timestamp Agreement Concentrating Disagreement Interested Thinking Unsure 101116112838516

    0.001836032 0.999917 1.79E-04 0.16485406 0.57114255 0.04595062 101116112838578 0.001447654 0.9999516 1.29E-04 0.16310683 0.5958921 0.042706452 101116112838672 5.97E-04 0 1.5E-04 0.44996294 0.45527613 0.00789697 101116112838766 2.46E-04 0 1.75E-04 0.77445686 0.32144752 0.001418217 101116112838860 1.01E-04 0 2.04E-04 0.93511915 0.21167138 2.53E-04 101116112838953 4.18E-05 0 2.38E-04 0.983739 0.13208677 4.52E-05 101116112839016 1.72E-05 0 2.78E-04 0.9960774 0.07941038 8.07E-06 101116112839110 7.1E-06 0 3.24E-04 0.99906266 0.046613157 1.44E-06 101116112839156 2.92E-06 0 3.77E-04 0.99977654 0.026964737 2.57E-07 101116112839250 1.21E-06 0 4.4E-04 0.9999467 0.015464196 4.58E-08 101116112839391 4.97E-07 0 5.12E-04 0.9999873 0.008824189 8.18E-09 101116112839438 2.05E-07 0 5.97E-04 0.999997 0.005020725 1.46E-09 101116112839547 8.43E-08 0 6.96E-04 0.9999993 0.002851939 2.6E-10 101116112839578 3.47E-08 0 8.11E-04 0.9999999 0.001618473 4.64E-11 101116112839688 1.43E-08 0 9.45E-04 0.99999994 9.18E-04 8.29E-12 101116112839781 5.9E-09 0 0.001101404 1 5.21E-04 1.48E-12 101116112839828 2.43E-09 0 0.001283521 1 2.95E-04 2.64E-13
  25. Eye Tracking Systems 3 2 5

  26. Eye Tracking System • Eye-tracking systems • These are instruments

    that measure eyes position and eyes movement in order to detect zones in which the user has particular interest in a specific time and moment. • Datasets from Tobii © Eye-tracking system [9] data will be shown. [9] Tobii Technology - Eye Tracking and Eye Control. Retrieved April 26, 2011, from http://www.tobii.com.
  27. Eye Tracking System • Tobii® Eye Tracker. • The device

    reports data with intervals of 100 ms (10Hz). • The output provides data concerning attention direction (gaze-x, gaze-y), duration of fixation, and pupil dilation.
  28. Eye Tracking System Demo Tobii® Eye Tracker

  29. Eye Tracking System Field Description Values Timestamp It is the

    timestamp (date and time) of the computer running the system. It could be used to synchronize the data with other sensors. Format "yymmddhhmmssSSS" (y - year, m - month, d - day, h - hour, m - minutes, s - seconds, S - milliseconds). GazePoint X The horizontal screen position for either eye or the average for both eyes. This value is also used for the fixation definition. 0 is the left edge, the maximum value of the horizontal screen resolution is the right edge. GazePoint Y The vertical screen position for either eye or the average for both eyes. This value is also used for the fixation definition. 0 is the bottom edge, the maximum value of the vertical screen resolution is the right edge. Pupil Left Pupil size (left eye) in mm. Variates Validity Left Validity of the gaze data. 0 to 4. 0 if the eye is found and the tracking quality good. If the eye cannot be found by the eye tracker, the validity code will be 4. Pupil Right Pupil size (left eye) in mm. Variates Validity Right Validity of the gaze data. 0 to 4. 0 if the eye is found and the tracking quality good. If the eye cannot be found by the eye tracker, the validity code will be 4. FixationDuration Fixation duration. The time in milliseconds that a fixation lasts. Variates. Event Events, automatic and logged, will show up under Event. Variates. AOI Areas Of Interests if fixations on multiple AOIs are to be written on the same row. Variates.
  30. Eye Tracking System Timestamp GPX GPY Pupil Left Validity L

    Pupil Right Validity R Fixation Event AOI 101124162405582 636 199 2.759313 0 2.88406 0 48 Content 101124162405599 641 207 2.684893 0 2.855817 0 48 Content 101124162405615 659 211 2.624458 0 2.903861 0 48 Content 101124162405632 644 201 2.636186 0 2.916132 0 48 Content 101124162405649 644 213 2.690685 0 2.831013 0 48 Content 101124162405666 628 194 2.651784 0 2.869714 0 48 Content 101124162405682 614 177 2.829281 0 2.899828 0 48 Content 101124162405699 701 249 2.780344 0 2.907665 0 49 Content 101124162405716 906 341 2.853761 0 2.916398 0 49 Content 101124162405732 947 398 2.829427 0 2.889944 0 49 Content 101124162405749 941 400 2.826602 0 2.881179 0 49 Content 101124162405766 938 403 2.78699 0 2.87948 0 49 KeyPress Content 101124162405782 937 411 2.803387 0 2.821803 0 49 Content 101124162405799 934 397 2.819166 0 2.871547 0 49 Content 101124162405816 941 407 2.811687 0 2.817927 0 49 Content 101124162405832 946 405 2.857419 0 2.857427 0 49 Content 101124162405849 0 0 -1 4 -1 4 49 Content
  31. Visualization BCI and Gaze Points engagement This figure shows the

    engagement fixation points of expert player playing in expert-mode. The size of the circle represents the duration of the fixation in that point, while the level of shading represents the intensity of the emotion.
  32. Visualization BCI and Gaze Points frustration This figure shows the

    frustration fixation points of expert player playing in expert-mode. The size of the circle represents the duration of the fixation in that point, while the level of shading represents the intensity of the emotion.
  33. Visualization BCI and Gaze Points boredom This figure shows the

    boredom fixation points of expert player playing in expert-mode. The size of the circle represents the duration of the fixation in that point, while the level of shading represents the intensity of the emotion.
  34. Visualization BCI and Gaze Points engagement This figures shows the

    engagement gaze points (above a threshold of 0.6) of a user reading a material with seductive details (i.e. cartoons). For this user the text on the bottom part of the first column was engaging.
  35. Visualization BCI and Gaze Points frustration This figure shows the

    frustration gaze points (above a threshold of 0.6) of a user reading a material with seductive details (i.e. cartoons). Looking the cartoon is related with high frustration level.
  36. Visualization BCI and Gaze Points boredom This figure shows the

    boredom gaze points (above a threshold of 0.5) of a user while reading a material with seductive details (i.e. cartoons). Notice that the text in the middle part of the second column of that page was boring.
  37. Galvanic Skin Conductance Sensor 4

  38. Galvanic Skin Conductance

  39. Galvanic Skin Conductance • Provide information on the activity of

    physiological functions of an individual. • Arousal detection. Measures the electrical conductance of the skin, which varies with its moisture level that depends on the sweat glands, which are controlled by the sympathetic, and parasympathetic nervous systems. [10] • Hardware designed by MIT Media Lab. • It is a Wireless Bluetooth device that reports data in intervals of 500 ms approximately (2Hz) [10] M. Strauss, C. Reynolds, S. Hughes, K. Park, G. McDarby, and R.W. Picard, “The HandWave Bluetooth Skin Conductance Sensor,” Proc. First International Conference on Affective Computing and Intelligent Interaction (ACII 05), Springer-Verlang, Oct. 2005, pp. 699-706, doi:10.1007/11573548_90.
  40. Galvanic Skin Conductance

  41. Galvanic Skin Conductance Field Description Values Timestamp It is the

    timestamp (date and time) of the computer running the system. It could be used to synchronize the data with other sensors. Format "yymmddhhmmssSSS" (y - year, m - month, d - day, h - hour, m - minutes, s - seconds, S - milliseconds). Battery Voltage Level of the battery voltage. 0 - 3 Volts Conductance Level of arousal. 0 - 3 Volts
  42. Galvanic Skin Conductance Timestamp Voltage Conductance 101116101332262 2.482352941 1.030696176 101116101332762

    2.482352941 1.023404165 101116101333262 2.482352941 1.019813274 101116101333762 2.482352941 1.041657802 101116101334247 2.482352941 0.998280273 101116101334747 2.482352941 0.991181142 101116101335247 2.482352941 0.980592229 101116101335747 2.482352941 0.998280273 101116101336247 2.482352941 1.012586294 101116101336762 2.482352941 1.012586294 101116101337231 2.482352941 1.012586294 101116101337747 2.482352941 1.009008251 101116101338247 2.482352941 0.998280273 101116101338747 2.482352941 0.991181142 101116101339247 2.482352941 0.987628521 101116101339731 2.482352941 0.987628521 101116101340231 2.482352941 0.980592229
  43. Homework Read the Papers posted on Blackboard about examples of

    Affect-Driven Adaptive Systems
  44. SER594 – Human Computer Interaction Javier Gonzalez-Sanchez javiergs@asu.edu Spring 2019

    Disclaimer. These slides can only be used as study material for the SER594 course at ASU. They cannot be distributed or used for another purpose.