Eye Wear Computing

Augmenting the Human Mind

Slides to my 31C3 talk.
https://www.youtube.com/watch?v=5-chhJ6GLg0

The talk gives an overview about the emerging field of smart glasses and how they can be used to augment our mind (e.g. how to improve our brain with technology). The talk will focus mostly on how to quantify cognitive tasks in real world environments. I also present a first application scenarios on how to use smart eyewear (e.g. google glass or JINS MEME) for short term memory augmentation and cognitive activity recognition.

Considering the last centuries, major scientific breakthroughs aimed at overcoming our pyhsical limitations (faster transportation, higher buildings,
longer, more comfortable lifes).
Yet, I believe the coming big scientific
breakthroughs will focus on
overcoming our cognitive limitations.

Smart glasses can play a vital role in

1. understanding our cognitive actions and limitations
by quantifying them

2. helping us design interventions to improve our mind.

The talk will focus mostly on the first point,
what kind of cognitve tasks can we track already
with the smart glasses that are available in the
market and what will happen in the near future.
I will discuss application examples for
Google Glass and J!NS MEME. J!NS MEME is the first consumer level device measuring eye movements using electrodes also called Electrooculography (EOG). The MEME glasses not a general computing platform. They can only stream sensor data to a computer (e.g. smart phone, laptop, desktop) using Bluetooth LE. Sensor data includes vertical and horizontal EOG channels and accelerometer + gyroscope data. The runtime of the device is 8 hours enabling long term recording and, more important, long term real-time streaming of eye and head movement. They are unobtrusive and look mostly like normal glasses.
For Google Glass I present an open sensor-logging platform (including the infrared sensor to count eye blinks) and a fast interface to do lifelogging.

We will discuss which eye movements correlate with
brain functions and how this fact can be used
to estimate the cognitive task a user is performing,
from fatigue detection, over reading segmentation
to cognitive workload and the advances to track attention and concentration. Challenges discussed in the talk include how to get ground truth and how to evaluate performance in general.

Transcript

1. Augmenting the Human Mind EYE WEAR COMPUTING Kai Kunze kai@kmd.keio.ac.jp Keio Media Design, Japan
2. KAI KUNZE - EYE WEAR COMPUTING OVERVIEW Background Why Eye Wear Computing? Making it Accessible for Everybody Blinks/Eye Movements Electrooculography Measuring Other Cognitive States Cognitive Work Load Summary 2
3. KAI KUNZE - EYE WEAR COMPUTING BACKGROUND Wearable Computing: Recognizing human activities with sensors to support them during everyday life Physical -> Cognitive Activity Recognition 3 25c3 Talk: About Cyborgs and Gargoyles http://www.youtube.com/watch?v=Jex8z57-tQU
4. KAI KUNZE - EYE WEAR COMPUTING EYES - WINDOWS TO OUR MIND? Perception is one of our most important senses A lot of work in psychology and cognitive science shows relations to attention, concentration, intention … 4
5. KAI KUNZE - EYE WEAR COMPUTING TRACKING READING HABITS Using Optical Eyetracking to Count how many words you read What document types (How much you understand) 5 http://kaikunze.de/posts/30c3-toward-a-cognitive-quantified-self/ Ainsley Harriott - PET - I’ve always been a bit of an entertainer and played the funny man. I was a part-time comedian for years, so I learned how to stand in front of audiences. It made me sure of myself. I like being liked and I love making everyone smile. I’ve lived in London all my life and have just moved to a larger house with my wife Clare and our two children, Jimmy and Madeleine. We spend a lot of time just singing and dancing around the house. 5 I grew up with music because my dad is the pianist, Chester Harriott - who’s still playing, by the way. My working day is divided between television and writing cook books, though TV takes most of my time. I spend about five days a fortnight working on the cooking programs I appear in. I eat all sorts of things at home but I only buy quality food. When I’m cooking, I experiment with whatever is in the Ainsley Harriott - PET - I’ve always been a bit of an entertainer and played the funny man. I was a part-time comedian for years, so I learned how to stand in front of audiences. It made me sure of myself. I like being liked and I love making everyone smile. I’ve lived in London all my life and have just moved to a larger house with my wife Clare and our 30C3 Talk ID 5387 Toward a Cognitive "Quantified Self" - Activity Recognition for the Mind
6. KAI KUNZE - EYE WEAR COMPUTING FITBIT FOR YOUR MIND? What are healthy reading habits? Can we track attention, concentration with affordable sensors over the whole day? Can we do this during everyday? 6 http://kaikunze.de/posts/30c3-toward-a-cognitive-quantified-self/ Manga
7. KAI KUNZE - EYE WEAR COMPUTING COGNITIVE ACTIVITY TRACKING FOR EVERYBODY 7
8. DEMO Blink Detection and Activity Recognition on Google Glass https://github.com/shoya140/GlassLogger rooted glass chmod 664 /sys/bus/i2c/devices/4-0035/proxraw Issues with XE22 (working with XE18)
9. KAI KUNZE - EYE WEAR COMPUTING WHAT CAN WE DO WITH BLINKS? Recognizing talking (usually double blink freuqncy) Track social interactions during the day Correlations with focus and concentration Slower blink frequency -> more focus Correlations with content Oberservation: Blink synchron with turning the page on ebook reader Use change blindness during blinks Fatigue detection Ranking system for university lectures? 9
10. KAI KUNZE - EYE WEAR COMPUTING BLINKS AND HEAD MOTION Distinguishing closely related activities 10 reading watching solving sawing talking In the Blink of an Eye - Combining Head Motion and Eye Blink Frequency for Activity Recognition with Google Glass. Shoya Ishimaru, Jens Weppner, Kai Kunze, Andreas Bulling, Koichi Kise, Andreas Dengel and Paul Lukowicz. Proceedings of the 5th Augmented Human International Conference. 2014. 67 % recognition accuracy for eye blink only, 82 % with head motion patterns
11. KAI KUNZE - EYE WEAR COMPUTING ISSUES 11 SMI Eye Tracking Glasses 2 Wireless Tobii Glasses 2
12. KAI KUNZE - EYE WEAR COMPUTING J!NS MEME J!NS: a Japanese glasses company very different idea from Google Glass Not a full fledged computer! no display, NO camera a sensing device connected to phone. Electrooculography (detects eye movements) and motion sensors (accelerometer and gyro) We are working with J!NS on MEME research 12 https://www.jins-jp.com/jinsmeme/en/
13. DEMO J!NS MEME
14. KAI KUNZE - EYE WEAR COMPUTING ELECTROOCULOGRAPHY Eye is a Dipole ( + iris -retina) Eye Potential can be measured over electrodes Advantages: higher sampling rate -> able to detect fast saccades “Disadvantages”: just relative eye movements sometimes noisy no pupil diameter etc. 14 ＋ ＋
15. KAI KUNZE - EYE WEAR COMPUTING WEARABLE EOG PIONEERS 15 *Eye Movement Analysis for Activity Recognition Using Electrooculography
 Andreas Bulling, Jamie A. Ward, Hans Gellersen and Gerhard Tröster (2011), in: IEEE Transactions on Pattern Analysis and Machine Intelligence, 33:4(741-753) Full-time wearable headphone-type gaze detector. Manabe, Hiroyuki, and Masaaki Fukumoto. CHI'06 Extended Abstracts on Human Factors in Computing Systems. ACM, 2006. https://www.nttdocomo.co.jp/english/binary/pdf/corporate/ technology/rd/technical_journal/bn/vol12_3/vol12_3_012en.pdf
16. KAI KUNZE - EYE WEAR COMPUTING J!NS MEME ELECTRODE SETUP Integration into normal eyeglasses 16 https://www.jins-jp.com/jinsmeme/en/
17. KAI KUNZE - EYE WEAR COMPUTING ACTIVITY RECOGNITION ON MEME 17
18. KAI KUNZE - EYE WEAR COMPUTING DIY EOG 18 http://intelsath.com/downloads/pdfeyeboard/eyeboard.pdf Andreas Bulling and Daniel Roggen and Gerhard Tröster “Wearable EOG Goggles: Seamless sensing and context- awareness in everyday environments”. Journal of Ambient Intelligence and Smart Environments, 2009. https://perceptual.mpi-inf.mpg.de/files/ 2013/03/bulling09_jaise.pdf
19. KAI KUNZE - EYE WEAR COMPUTING EOG TO TRACK READING HABITS? 19
20. KAI KUNZE - EYE WEAR COMPUTING TRACKING COGNITIVE LOAD: BRAIN ACTIVATION SENSING 20 NIRS: near-infrared spectroscopy Estimating user’s brain activation by tracking oxyhemoglobin
21. KAI KUNZE - EYE WEAR COMPUTING NRIS 21
22. KAI KUNZE - EYE WEAR COMPUTING 22
23. KAI KUNZE - EYE WEAR COMPUTING BRAIN ACTIVATION SENSING 23 1-back 2-back 4-back brain activation during dual n-back problems work load 3-back Position Paper: Brain Teasers - Toward Wearable Computing that Engages Our Mind. Ishimaru, Shoya and Kunze, Kai and Kise, Koichi and Inami, Masahiko. Proceedings of UbiComp'14 Adjunct. 2014
24. KAI KUNZE - EYE WEAR COMPUTING IN SUMMARY previous centuries: faster higher more comfortable … 24 Last centuries: trying to overcome our physical limitations … Now: overcoming our cognitive limitations.
25. KAI KUNZE - EYE WEAR COMPUTING THANKS TO … Yuji Umea, Yuta Sugiura, Masahiko Inami, Masa Inakage, Kazutaka Inoue, Masato Kimura, Rina Kanno, Oliver Amft, Koichi Kise, Masakazu Iwamura, Motoi Iwata, Yuzuko Utsumi, Andreas Dengel, Andreas Bulling, David Bannach, Takumi Toyama, Tsutomu Terada, Seiichi Ushida, Stephan Sigg, Christoph Schuba, Cody Sumter, Paul Lukowicz, Bernhard Sick, Jingyuan Cheng, Kamil Kloch, Gerald Pirkl, Albrecht Schmidt, Michael Beigl, Niels Henze, Alireza Sahami, Tilman Dingler, Bastian Pfleging, Dawud Gordon, Till Riedel, Ulf Blanke, Yusuke Sugano, Hans Gellersen, Christian Weichel, Anton Dollmaier, Gernot Bahle, Josef Neuburger, Peter Kolloch 25
26. KAI KUNZE - EYE WEAR COMPUTING SPECIAL THANKS TO the students who do the work … (alphabetic order) Ayano Okoso Eria Chita Hiroki Fujiyoshi Katsuma Tanaka Kazuyo Yoshimura Masai Katsutoshi Mizuki Matsubara Shoya Ishimaru 26
27. KAI KUNZE - EYE WEAR COMPUTING http://kaikunze.de/ @k_garten kai@kmd.keio.ac.jp github.com/kkai Interested in Open Source Eye Wear Platform Shamless Advertisments: Augmented Human 2015, 9th-11th, Singapore FOSS Asia 2015, 13th-15th March, Singapore Superhuman Sports Committee http://superhuman-olympic.org QUESTIONS, REMARKS, VIOLENT DECENT? 27