Design and Implementation of Modules for the Extraction of Biometric Parameters in an Augmented BCI Framework - Presentation - (UniPa 20170302 - OsakaU 20170522)

The interaction between human beings and robotic agents, and the interest towards such topics, have been exponentially growing in the recent years. The purpose of this thesis project is to identify a relation between the behaviours of a humanoid robot placed in a social context, and the emotional responses of a subject interacting with it. In particular, through the use of Brain-Computer Interface (BCI) and gaze tracking technologies, it has been investigated on the relation between the trust towards a robotic agent and the effects it has on the brain signals. In order to evaluate this relation, the framework makes use of the acquired brain signals to extract biometric features, such as attention, stress, and mental workload, along with the visual focus. In order to investigate towards this direction, an interactive game session has been set up for the human-robot interaction. In particular, an instance of the well-known Rock-Paper-Scissors game has been used. The experimental results have been shown a correlation between the behaviours of a robotic agent and the effect of trust on the brain signals of the human user. In particular, the emotional response varies depending the type of behaviours expressed by the robotic agent.

Transcript

1. UNIVERSITY OF PALERMO POLYTECHNIC SCHOOL Department of Industrial and Digital Innovation (DIID) Computer Science Engineering for Intelligent Systems Design and Implementation of Modules for the Extraction of Biometric Parameters in an Augmented BCI Framework Master Degree Thesis of: Salvatore La Bua WWW.SLBLABS.COM March, 2017
2. Introduction ▪What ◦ Investigate the effects of the interaction with a robotic agent on the mental status of the human player through brain signal analysis ◦ Acceptance of a robotic agent by the user ◦ Performance improvements over a classical BCI system ▪How ◦ Rock-Paper-Scissors game integration ◦ UniPA BCI Framework based on the P300 paradigm ◦ Augmented by ◦ Eye gaze coordinate acquisition ◦ Biometric feature extraction DESIGN AND IMPLEMENTATION OF MODULES FOR THE EXTRACTION OF BIOMETRIC PARAMETERS IN AN AUGMENTED BCI FRAMEWORK S. La Bua 2
3. Introduction Human-Robot Interaction (HRI) ▪HRI as a multidisciplinary research topic ◦ Artificial Intelligence ◦ Human-Computer Interaction ◦ Natural Language Processing ◦ Social Sciences ◦ Design ▪Model of the user’s expectation towards a robotic agent in a human-robot interaction DESIGN AND IMPLEMENTATION OF MODULES FOR THE EXTRACTION OF BIOMETRIC PARAMETERS IN AN AUGMENTED BCI FRAMEWORK S. La Bua 3
4. Introduction Brain-Computer Interfaces (BCI) ▪Direct communication between brain and external devices ◦ Non-Invasive ◦ Partially-Invasive ◦ Invasive ▪Brain Lobes ◦ Frontal: emotions, social behaviour ◦ Temporal: speech, hearing recognition ◦ Parietal: sensory recognition ◦ Occipital: visual processing ▪Extraction of biometric features from brain signals DESIGN AND IMPLEMENTATION OF MODULES FOR THE EXTRACTION OF BIOMETRIC PARAMETERS IN AN AUGMENTED BCI FRAMEWORK S. La Bua 4
5. Introduction Visual Focus ▪Importance of eye gaze for direct interaction in a social environment ▪Interfaces dedicated to people affected by degenerative pathologies ▪Entertainment applications, such as games ▪Better advertisement placement DESIGN AND IMPLEMENTATION OF MODULES FOR THE EXTRACTION OF BIOMETRIC PARAMETERS IN AN AUGMENTED BCI FRAMEWORK S. La Bua 5
6. Methodology Background Information ▪Problem ◦ Effects of the behaviour of a robotic agent on the brain signals ◦ Trust context in Human-Robot Interaction ▪Feature Extraction ◦ Entropy: as a stress indicator ◦ Energy: as a concentration indicator ◦ Mental Workload: as an index of engagement in the task ▪Brain waves types ◦ δ Delta: Hz 0.5÷3 related to instinct, deep sleep ◦ θ Theta: Hz 3÷8 related to emotions ◦ α Alpha: Hz 8÷12 related to consciousness ◦ β Beta: Hz 12÷38 related to concentration, stress ◦ γ Gamma: Hz 38÷42 related to information processing DESIGN AND IMPLEMENTATION OF MODULES FOR THE EXTRACTION OF BIOMETRIC PARAMETERS IN AN AUGMENTED BCI FRAMEWORK S. La Bua 6
7. Methodology The math behind Entropy: = 2log ( 2); = − σ Energy: = ෍ =−∞ ∞ () 2 Mental Workload: + DESIGN AND IMPLEMENTATION OF MODULES FOR THE EXTRACTION OF BIOMETRIC PARAMETERS IN AN AUGMENTED BCI FRAMEWORK S. La Bua 7
8. The Proposed Solution Architecture Structure ▪Action Selection ◦ Direct interface with the user ◦ Acquisition of bio-signals ◦ Acquisition of eye gaze coordinates ◦ Selection of the Base action ▪Feature Extraction and Analysis ◦ Bio-signals analysis ◦ Features extraction ◦ Features analysis ◦ Computation of Intention, Attention, Stress indices ▪Response Modulation ◦ Threshold of the Base action by means of the Intention index ◦ Modulation of the resulting action by means of Attention and Stress indices DESIGN AND IMPLEMENTATION OF MODULES FOR THE EXTRACTION OF BIOMETRIC PARAMETERS IN AN AUGMENTED BCI FRAMEWORK S. La Bua 8
9. The Proposed Solution Class Diagram DESIGN AND IMPLEMENTATION OF MODULES FOR THE EXTRACTION OF BIOMETRIC PARAMETERS IN AN AUGMENTED BCI FRAMEWORK S. La Bua 9
10. The Proposed Solution Functional Blocks DESIGN AND IMPLEMENTATION OF MODULES FOR THE EXTRACTION OF BIOMETRIC PARAMETERS IN AN AUGMENTED BCI FRAMEWORK Action Selection ◦ Eye-Tracking module ◦ Screen coordinates acquisition ◦ Weighing module ◦ Weighing of the BCI classifier response precision and the Eye-Tracking module response precision, by means of the user’s skill level ◦ ID Selection module ◦ Action selection by means of the weighted BCI classifier and Eye-Tracking module precisions S. La Bua 10
11. The Proposed Solution Functional Blocks Feature Extraction and Analysis ◦ It makes use of external calls to the MATLAB engine ◦ Features extracted and analysed ◦ Correlation Factor: related to the Intention index ◦ Energy: related to the Attention index ◦ Entropy: related to the Stress index DESIGN AND IMPLEMENTATION OF MODULES FOR THE EXTRACTION OF BIOMETRIC PARAMETERS IN AN AUGMENTED BCI FRAMEWORK S. La Bua 11
12. The Proposed Solution Functional Blocks Response Modulation ◦ Threshold module ◦ ID Selection validation by means of Intention index thresholding ◦ Modulation module ◦ In the case the selected ID has passed the validation step, the resulting action is modulated by means of the Attention and Stress indices DESIGN AND IMPLEMENTATION OF MODULES FOR THE EXTRACTION OF BIOMETRIC PARAMETERS IN AN AUGMENTED BCI FRAMEWORK S. La Bua 12
13. The Proposed Solution Robotic Controller DESIGN AND IMPLEMENTATION OF MODULES FOR THE EXTRACTION OF BIOMETRIC PARAMETERS IN AN AUGMENTED BCI FRAMEWORK S. La Bua 13
14. The Proposed Solution Utilisation Modes Basic Mode ◦ Simplest mode ◦ Minimal number of modules involved ◦ Classical BCI approach ◦ P300 paradigm classification ◦ Direct Behaviour DESIGN AND IMPLEMENTATION OF MODULES FOR THE EXTRACTION OF BIOMETRIC PARAMETERS IN AN AUGMENTED BCI FRAMEWORK S. La Bua 14
15. The Proposed Solution Utilisation Modes Hybrid Mode ◦ Advanced mode ◦ Eye-Tracking module ◦ Combination of brain signals and eye gaze ◦ User skill level as weighting parameter ◦ Composite Behaviour DESIGN AND IMPLEMENTATION OF MODULES FOR THE EXTRACTION OF BIOMETRIC PARAMETERS IN AN AUGMENTED BCI FRAMEWORK S. La Bua 15
16. The Proposed Solution Utilisation Modes Bio-Hybrid Mode ◦ Complete mode ◦ Feature Extraction and Analysis functional block ◦ Response Modulation functional block ◦ Intention, Attention and Stress indices computation ◦ Modulated Behaviour DESIGN AND IMPLEMENTATION OF MODULES FOR THE EXTRACTION OF BIOMETRIC PARAMETERS IN AN AUGMENTED BCI FRAMEWORK S. La Bua 16
17. Architecture Eye-Tracking module P300 6x6 spelling matrix 3x3 spelling window areas DESIGN AND IMPLEMENTATION OF MODULES FOR THE EXTRACTION OF BIOMETRIC PARAMETERS IN AN AUGMENTED BCI FRAMEWORK S. La Bua 17
18. Architecture Eye-Tracking module Preliminary tests results DESIGN AND IMPLEMENTATION OF MODULES FOR THE EXTRACTION OF BIOMETRIC PARAMETERS IN AN AUGMENTED BCI FRAMEWORK SUBCATEGORIES FOR SINGLE ELEMENT FOCUS % CENTRAL FOCUS % LATERAL FOCUS % EXTERNAL FOCUS % 3-BY-3, 700X700PX 100 99.9000 0.1000 0 3-BY-3, 300X300PX 98.4562 93.2697 6.7303 1.5438 6-BY-6, 700X700PX 100 84.7408 2.7592 0 6-BY-6, 300X300PX 99.5997 75.9943 24.0057 0.4003 SUBCATEGORIES FOR ROW SPAN SELECTION FOCUS % CENTRAL FOCUS % LATERAL FOCUS % EXTERNAL FOCUS % 3-BY-3, 700X700PX 74.2632 93.9192 6.0808 25.7368 3-BY-3, 300X300PX 77.1340 89.9075 10.0925 22.8660 6-BY-6, 700X700PX 69.5037 96.3287 3.6713 30.4963 6-BY-6, 300X300PX 75.0674 71.7202 28.2798 24.9326 AVERAGE BY PARAMETER FOCUS % CENTRAL FOCUS % 700X700PX 85.9417 93.7222 300X300PX 87.5643 82.7229 GAIN WITH LARGER WINDOW -1.8530% +13.2966% AVERAGE BY PARAMETER FOCUS % CENTRAL FOCUS % 3-BY-3 87.4634 94.2491 6-BY-6 86.0427 82.1960 GAIN WITH LESS DENSE MATRIX +1.6512% +14.6639% S. La Bua 18
19. Architecture Data Structures Generic signal data structure fields N fields dedicated to the brain signals acquisition ◦ Ch 1 – Ch 16 3 auxiliary fields to carry peculiar information ◦ A, B, C DESIGN AND IMPLEMENTATION OF MODULES FOR THE EXTRACTION OF BIOMETRIC PARAMETERS IN AN AUGMENTED BCI FRAMEWORK CH 1 CH 2 · · · CH N A B C S. La Bua 19
20. Architecture Data Structures Baseline Calibration signal DESIGN AND IMPLEMENTATION OF MODULES FOR THE EXTRACTION OF BIOMETRIC PARAMETERS IN AN AUGMENTED BCI FRAMEWORK A (RED) B (CYAN) C (MAGENTA) BASELINE CALIBRATION -2 EYES STATUS 0 S. La Bua 20
21. Architecture Data Structures Game Session signal DESIGN AND IMPLEMENTATION OF MODULES FOR THE EXTRACTION OF BIOMETRIC PARAMETERS IN AN AUGMENTED BCI FRAMEWORK A (RED) B (CYAN) C (MAGENTA) GAME SESSION TRIAL STATUS TRIAL SUB-PHASE GAZE TRACKING S. La Bua 21
22. Architecture Data Structures P300 Calibration signal P300 Spelling signal DESIGN AND IMPLEMENTATION OF MODULES FOR THE EXTRACTION OF BIOMETRIC PARAMETERS IN AN AUGMENTED BCI FRAMEWORK A B C P300 Spelling -1 Flashing tag 0 A B C P300 Calibration Calibration target Flashing tag 0 S. La Bua 22
23. The Framework Main Interface 1. Basic settings ◦ P300-related settings ◦ Preset modes 2. Main functionalities ◦ Signal quality check ◦ P300 Calibration and Recognition ◦ Game session control 3. Interface modality ◦ Alphabetic or Symbolic 4. Devices ◦ Eye-Tracker settings 5. Plots and Indicators ◦ Signals and Indices visualisation 6. Output panel ◦ Feedback for the operator DESIGN AND IMPLEMENTATION OF MODULES FOR THE EXTRACTION OF BIOMETRIC PARAMETERS IN AN AUGMENTED BCI FRAMEWORK 1 2 3 4 5 6 S. La Bua 24
24. The Framework Baseline Acquisition Interface Control dialog window User dialog window DESIGN AND IMPLEMENTATION OF MODULES FOR THE EXTRACTION OF BIOMETRIC PARAMETERS IN AN AUGMENTED BCI FRAMEWORK S. La Bua 25
25. The Framework Game Session Interface 1. Game modality ◦ Fair ◦ Cheat-to-Win/Lose 2. Trials number per session ◦ Initial Fair sub-session ◦ Middle Cheating sub-session ◦ Terminal Fair sub-session 3. Devices ◦ BCI signal acquisition ◦ Kinect gesture recognition ◦ Play against a robotic agent 4. Session panel ◦ Moves selection ◦ Trial temporal progress DESIGN AND IMPLEMENTATION OF MODULES FOR THE EXTRACTION OF BIOMETRIC PARAMETERS IN AN AUGMENTED BCI FRAMEWORK 1 2 3 4 S. La Bua 27
26. Experiments Introduction ▪Purpose ◦ Investigate the effects of the interaction with a cheating robotic agent on the mental status of the human player ◦ Rock-Paper-Scissors game session ▪Scenarios ◦ The robot behaves according to the game’s rules ◦ The robot exhibits a cheat-to-win behaviour ◦ The robot exhibits a cheat-to-lose behaviour ▪Game Session DESIGN AND IMPLEMENTATION OF MODULES FOR THE EXTRACTION OF BIOMETRIC PARAMETERS IN AN AUGMENTED BCI FRAMEWORK Initial Fair sub-session Cheating sub-session Terminal Fair sub-session S. La Bua 28
27. Experiments Set-up DESIGN AND IMPLEMENTATION OF MODULES FOR THE EXTRACTION OF BIOMETRIC PARAMETERS IN AN AUGMENTED BCI FRAMEWORK Subjects ◦ 16 Subjects ◦ Aged 18-51 Hardware ◦ g.tec g.USBamp ◦ g.tec g.GAMMAbox ◦ g.tec g.GAMMAcap2 ◦ Secondary standard PC screen ◦ Tobii EyeX eye tracker ◦ Kinect for Xbox One ◦ Telenoid ◦ Camera(s) S. La Bua 29
28. Experiments EEG Electrodes configuration Channels-Electrodes correspondence L R DESIGN AND IMPLEMENTATION OF MODULES FOR THE EXTRACTION OF BIOMETRIC PARAMETERS IN AN AUGMENTED BCI FRAMEWORK Ch 01 F7 Ch 02 F3 Ch 03 FZ Ch 04 T3 Ch 05 C3 Ch 06 T5 Ch 07 P3 Ch 08 O1 Ch 09 F8 Ch 10 F4 Ch 11 T4 Ch 12 C4 Ch 13 T6 Ch 14 P4 Ch 15 PZ Ch 16 O2 S. La Bua 30
29. Experiments Protocol DESIGN AND IMPLEMENTATION OF MODULES FOR THE EXTRACTION OF BIOMETRIC PARAMETERS IN AN AUGMENTED BCI FRAMEWORK S. La Bua 31
30. Experiments Protocol DESIGN AND IMPLEMENTATION OF MODULES FOR THE EXTRACTION OF BIOMETRIC PARAMETERS IN AN AUGMENTED BCI FRAMEWORK S. La Bua 32
31. Experiments Subcategories ▪Sub-Session Analysis ◦ Analysis of the Baseline signal, Fair and Cheating sub-sessions ▪Trials Analysis ◦ Single trial analysis for each subject ▪Intra-Class Comparison ◦ Comparison between Cheat-to-Win and Cheat-to-Lose classes ▪Average Analysis ◦ Average over all subjects, by class and by sub-sessions DESIGN AND IMPLEMENTATION OF MODULES FOR THE EXTRACTION OF BIOMETRIC PARAMETERS IN AN AUGMENTED BCI FRAMEWORK S. La Bua 33
32. Experiments Sub-Session Analysis Entropy Energy DESIGN AND IMPLEMENTATION OF MODULES FOR THE EXTRACTION OF BIOMETRIC PARAMETERS IN AN AUGMENTED BCI FRAMEWORK S. La Bua 34
33. Experiments Sub-Session Analysis Mental Workload Visual Focus % DESIGN AND IMPLEMENTATION OF MODULES FOR THE EXTRACTION OF BIOMETRIC PARAMETERS IN AN AUGMENTED BCI FRAMEWORK S. La Bua 35
34. Experiments Trials Analysis Summary: Entropy Energy Mental Workload Visual Focus % DESIGN AND IMPLEMENTATION OF MODULES FOR THE EXTRACTION OF BIOMETRIC PARAMETERS IN AN AUGMENTED BCI FRAMEWORK S. La Bua 36
35. Experiments Trials Analysis Entropy: Cheat-to-Win Cheat-to-Lose DESIGN AND IMPLEMENTATION OF MODULES FOR THE EXTRACTION OF BIOMETRIC PARAMETERS IN AN AUGMENTED BCI FRAMEWORK S. La Bua 37
36. Experiments Trials Analysis Energy: Cheat-to-Win Cheat-to-Lose DESIGN AND IMPLEMENTATION OF MODULES FOR THE EXTRACTION OF BIOMETRIC PARAMETERS IN AN AUGMENTED BCI FRAMEWORK S. La Bua 38
37. Experiments Trials Analysis Workload: Cheat-to-Win Cheat-to-Lose DESIGN AND IMPLEMENTATION OF MODULES FOR THE EXTRACTION OF BIOMETRIC PARAMETERS IN AN AUGMENTED BCI FRAMEWORK S. La Bua 39
38. Experiments Trials Analysis Focus %: Cheat-to-Win Cheat-to-Lose DESIGN AND IMPLEMENTATION OF MODULES FOR THE EXTRACTION OF BIOMETRIC PARAMETERS IN AN AUGMENTED BCI FRAMEWORK S. La Bua 40
39. Experiments Intra-Class Comparison Entropy: Cheat-to-Win Cheat-to-Lose DESIGN AND IMPLEMENTATION OF MODULES FOR THE EXTRACTION OF BIOMETRIC PARAMETERS IN AN AUGMENTED BCI FRAMEWORK S. La Bua 41
40. Experiments Intra-Class Comparison Energy: Cheat-to-Win Cheat-to-Lose DESIGN AND IMPLEMENTATION OF MODULES FOR THE EXTRACTION OF BIOMETRIC PARAMETERS IN AN AUGMENTED BCI FRAMEWORK S. La Bua 42
41. Experiments Intra-Class Comparison Mental Workload: Cheat-to-Win Cheat-to-Lose DESIGN AND IMPLEMENTATION OF MODULES FOR THE EXTRACTION OF BIOMETRIC PARAMETERS IN AN AUGMENTED BCI FRAMEWORK S. La Bua 43
42. Experiments Intra-Class Comparison Visual Focus percentage: Cheat-to-Win Cheat-to-Lose DESIGN AND IMPLEMENTATION OF MODULES FOR THE EXTRACTION OF BIOMETRIC PARAMETERS IN AN AUGMENTED BCI FRAMEWORK S. La Bua 44
43. Experiments Average Analysis Entropy The entropy values do not show any particular evidence of stress DESIGN AND IMPLEMENTATION OF MODULES FOR THE EXTRACTION OF BIOMETRIC PARAMETERS IN AN AUGMENTED BCI FRAMEWORK ENTROPY FAIR 1 CHEAT FAIR 2 MEAN STD DEV MEAN STD DEV MEAN STD DEV CHEAT WIN 3.8584 0.2191 3.8998 0.2540 3.8742 0.1891 CHEAT LOSE 3.7420 0.0850 3.7632 0.1177 3.7304 0.1074 S. La Bua 45
44. Experiments Average Analysis Energy The energy values show higher concentration level for the Cheat-to-Win class DESIGN AND IMPLEMENTATION OF MODULES FOR THE EXTRACTION OF BIOMETRIC PARAMETERS IN AN AUGMENTED BCI FRAMEWORK ENERGY FAIR 1 CHEAT FAIR 2 MEAN STD DEV MEAN STD DEV MEAN STD DEV CHEAT WIN 0.2572 0.2141 0.3032 0.2267 0.2254 0.1951 CHEAT LOSE 0.1498 0.0596 0.1720 0.0948 0.1143 0.0447 S. La Bua 46
45. Experiments Average Analysis Mental Workload The mental workload values show a slightly lower engagement level for the Cheat-to-Win class DESIGN AND IMPLEMENTATION OF MODULES FOR THE EXTRACTION OF BIOMETRIC PARAMETERS IN AN AUGMENTED BCI FRAMEWORK MENTAL WL FAIR 1 CHEAT FAIR 2 MEAN STD DEV MEAN STD DEV MEAN STD DEV CHEAT WIN 1.3798 1.1625 0.8988 0.4215 0.9437 0.4570 CHEAT LOSE 1.0923 0.2716 1.0382 0.3229 1.0777 0.3936 S. La Bua 47
46. Experiments Average Analysis Visual Focus The visual focus values show higher visual attention level for the Cheat-to-Win class DESIGN AND IMPLEMENTATION OF MODULES FOR THE EXTRACTION OF BIOMETRIC PARAMETERS IN AN AUGMENTED BCI FRAMEWORK FOCUS % FAIR 1 CHEAT FAIR 2 MEAN STD DEV MEAN STD DEV MEAN STD DEV CHEAT WIN 7.89100 8.93670 9.13020 11.3344 12.1404 20.1567 CHEAT LOSE 4.59710 9.91690 3.24540 7.09430 2.20110 4.79480 S. La Bua 48
47. Experiments Demo DESIGN AND IMPLEMENTATION OF MODULES FOR THE EXTRACTION OF BIOMETRIC PARAMETERS IN AN AUGMENTED BCI FRAMEWORK S. La Bua 49
48. Conclusions and Future Works ▪A robotic agent that cheats to win is perceived as more agentic and human-like than a robot that cheats to lose ▪Some of the Questionnaire results ▪Trust related improvement ◦ Biometric features to mitigate or amplify the effects of the robotic agent behaviour on the subject’s emotional response DESIGN AND IMPLEMENTATION OF MODULES FOR THE EXTRACTION OF BIOMETRIC PARAMETERS IN AN AUGMENTED BCI FRAMEWORK Unusual Behaviour Fair Play Intelligence Strongly Disagree Strongly Agree S. La Bua 50
49. Future Works Framework Extension Sensor Aggregation functional block ◦ Galvanic Skin Response (GSR) sensor ◦ Heart Rate (HR) sensor ◦ Other physiological sensors DESIGN AND IMPLEMENTATION OF MODULES FOR THE EXTRACTION OF BIOMETRIC PARAMETERS IN AN AUGMENTED BCI FRAMEWORK S. La Bua 51
50. Future Works Extended Framework DESIGN AND IMPLEMENTATION OF MODULES FOR THE EXTRACTION OF BIOMETRIC PARAMETERS IN AN AUGMENTED BCI FRAMEWORK S. La Bua 52
51. Thank you for your attention Salvatore La Bua slabua@gmail.com WWW.SLBLABS.COM