Reading Activity Recognition using an Off-the-shelf EEG

Detecting Reading Activities and Distinguishing Genres of Documents

Paper Presentation at ICDAR 2013 http://icdar2013.org/

Abstract—The document analysis community spends substan- tial resources towards computer recognition of any type of text (e.g. characters, handwriting, document structure etc.). In this paper, we introduce a new paradigm focusing on recognizing the activities and habits of users while they are reading. We describe the differences to the traditional approaches of document analysis. We present initial work towards recognizing reading activities. We report our initial findings using a commercial, dry electrode Electroencephalography (EEG) system. We show the feasibility to distinguish reading tasks for 3 different document genres with one user and near perfect accuracy. Distinguishing reading tasks for 3 different document types we achieve 97 % with user specific training. We present evidence that reading and non-reading related activities can be separated over 3 users using 6 classes, perfectly separating reading from non-reading. A simple EEG system seems promising for distinguishing the reading of different document genres.

Transcript

1. Reading Activity Recognition using an Off-the-shelf EEG Kai Kunze, Yuki Shiga, Shoya Ishimaru, Koichi Kise Osaka Prefecture University Detecting Reading Activities and Distinguishing Genres of Documents
2. Kai Kunze - Reading Activity Recognition using an Off-the-shelf EEG Overview Motivation Modalities to Explore Reading Activities Approach Experiments Results Conclusion and Future Work 2
3. Kai Kunze - Reading Activity Recognition using an Off-the-shelf EEG Towards “user-centered” Document Analysis ... Traditionally, Document Analysis focuses on the object 3 Documents as a structured source of information
4. Kai Kunze - Reading Activity Recognition using an Off-the-shelf EEG “User-centered” Document Analysis Let’s focus on the readers 1. Analyze the document through the users ... Are they reading? What did they read? How much?How fast? How often? How much do they understand? 2. Analyze the users through the documents/ their reading behavior 4 Similar to Kindle Highlights Imagine annotating documents with: “x readers stopped reading after this sentence.” “most readers felt sad after this paragraph” “the reader never saw this warning label” ...
5. Kai Kunze - Reading Activity Recognition using an Off-the-shelf EEG Interesting Modalities for Tracking Reading Habits Computer Vision: Egocentric Camera Monitoring Brain Activity Electroencephalography (EEG), Near-Infrared Spectroscopy (fNIS) Eye-Tracking (see the “Wordometer” talk this morning) ... we are actively exploring other sensing modalities. Important: Unobtrusive and Robust Tracking. 5
6. Kai Kunze - Reading Activity Recognition using an Off-the-shelf EEG Case Study: Evaluating an Off-the-Shelf EEG Can we distinguish different types of reading activities using a relatively cheap, commercially available EEG device? Is it feasible to segment reading from non-reading activities with such a device? EGG gives us also some additional useful information ... Emotional State, Concentration etc. 6
7. Kai Kunze - Reading Activity Recognition using an Off-the-shelf EEG Approach EEG signal analysis bandpass filter between 1-30Hz. Fast Fourier Transform (FFT) sliding window 500 msec., step size 20 msec. Sum the Frequencies according to the Classification of Brain Bands: Beta (13-30Hz), Alpha (8-12 Hz), Theta (4-8 Hz), and Delta (less than 4 Hz) other features: RMS, FFT Mean, FFT Variance K-nearest Neighbor Classifier 7
8. Kai Kunze - Reading Activity Recognition using an Off-the-shelf EEG Experiments Disclaimer: Very few participants (in the paper all recorded on one day) 3 participants over 5 days recording Emotiv 14 Channel Wet Electrode EEG 10 min. per activity 3 reading: read a science paper, manga, internet news 3 not reading: listening to music, watching a video, playing chess(draw) doing nothing as reference 8
9. Kai Kunze - Reading Activity Recognition using an Off-the-shelf EEG Experimental Results Reading vs. Not Reading Recognition (leave-out-one day): Train on 4 evaluate on 1 day User dependent: 90 % User independent: 69 % Recognition for the Individual Tasks (leave-out-one day): User Dependent: 80 % User Independent: 62.25% 9 a b c d e f g ←)classified)as 86 0 12 1 0 0 0 a=Paper 6 90 0 0 1 2 0 b=Watch 22 1 73 3 0 0 0 c=Manga 9 7 20 63 0 0 0 d=News 0 12 6 1 76 4 0 e=Music 0 14 1 0 26 58 0 f=Nothing 0 0 6 0 0 0 93 g=Chess
10. Kai Kunze - Reading Activity Recognition using an Off-the-shelf EEG Conclusion and Future Work Of course, larger scale experimental setup Segmentation of Reading Activity Emotiv: records for 3 hours charging while recording is not possible needs a laptop (usb plug, proprietary wireless protocol) Evaluate the best electrode positions for tracking reading. Use devices with fewer electrodes long-term recordings 10
11. Kai Kunze - Reading Activity Recognition using an Off-the-shelf EEG Questions? Answers! http://kaikunze.de twitter: @k_garten facebook: kai.kunze app.net: @kkai kai.kunze@gmail.com https://github.com/kkai/ 11