VR research, 50 years later_Adam Streck_Codemotion Berlin 2019

It's been half a century from the IE Sutherland's seminal paper "A head-mounted three dimensional display". Since then we're already in what's called "The 3rd wave of VR", seeing the technology coming and going. While the consumer interest came and wavered, research has been steadily progressing in all the related areas. In this talk I will touch on the human-computer-interaction related topics of the VR research, summarize the results of the past research and take a peek into what are the upcoming challenges.

About: Adam Streck, PostDoc - DZNE

Adam is postdoctoral researcher at the German Center for Neurodegenerative Diseases where he applies next-gen technologies to psychology and neuroscience research. Currently he focuses on simulation of human interaction in virtual environments and building of believable digital AI agents with the applications in the treatment of anxiety disorders.

Transcript

1. VR Research, 50 years later Dr. Adam Streck 12-13 November, 2019
2. Who is me? 2000 2010 2019 GAMES DEV
3. Who is me? 2000 2010 2019 GAMES DEV RESEARCH - MODELING OF BIOLOGICAL SYSTEMS
4. Who is me? 2000 2010 2019 GAMES DEV RESEARCH - MODELING OF BIOLOGICAL SYSTEMS VIRTUAL REALITY DEV
5. Who is me? 2000 2010 2019 GAMES DEV RESEARCH - MODELING OF BIOLOGICAL SYSTEMS VIRTUAL REALITY DEV RESEARCH - VR PSYCHOLOGY
6. What is VR? (Let’s be formal for a slide)
7. What is VR? (Let’s be formal for a slide) Real-time interactive graphics with three-dimensional models, combined with a display technology that gives the user the immersion in the model world and direct manipulation.* *Brooks Jr, Frederick, et al. "Research directions in virtual environments." Computer Graphics 26.3 (1992): 153.
8. What is VR? (Let’s be formal for a slide) Real-time interactive graphics with three-dimensional models, combined with a display technology that gives the user the immersion in the model world and direct manipulation.* 1. Subset of spatial computing *Brooks Jr, Frederick, et al. "Research directions in virtual environments." Computer Graphics 26.3 (1992): 153.
9. What is VR? (Let’s be formal for a slide) Real-time interactive graphics with three-dimensional models, combined with a display technology that gives the user the immersion in the model world and direct manipulation.* 1. Subset of spatial computing 2. Subset of human-computer interaction *Brooks Jr, Frederick, et al. "Research directions in virtual environments." Computer Graphics 26.3 (1992): 153.
10. What is VR? (Let’s be formal for a slide) Real-time interactive graphics with three-dimensional models, combined with a display technology that gives the user the immersion in the model world and direct manipulation.* 1. Subset of spatial computing 2. Subset of human-computer interaction 3. Applied in: a. Psychology b. Neuroscience c. Medicine d. Education *Brooks Jr, Frederick, et al. "Research directions in virtual environments." Computer Graphics 26.3 (1992): 153.
11. Research into VR technology and principles
12. Half a century ago… The Sword of Damocles • 1st paper describing a 3D HMD* * Sutherland, Ivan E. "A head-mounted three dimensional display." Proceedings of the December 9-11, 1968, fall joint computer conference, part I. ACM, 1968.
13. Half a century ago… The Sword of Damocles • 1st paper describing a 3D HMD* • 4000 lines at 30 FPS * Sutherland, Ivan E. "A head-mounted three dimensional display." Proceedings of the December 9-11, 1968, fall joint computer conference, part I. ACM, 1968.
14. Half a century ago… The Sword of Damocles • 1st paper describing a 3D HMD* • 4000 lines at 30 FPS • Mechanical rotation detection * Sutherland, Ivan E. "A head-mounted three dimensional display." Proceedings of the December 9-11, 1968, fall joint computer conference, part I. ACM, 1968.
15. Half a century ago… The Sword of Damocles • 1st paper describing a 3D HMD* • 4000 lines at 30 FPS • Mechanical rotation detection • Additionally 40 kHz ultrasound • 3 emitters • 4 receivers * Sutherland, Ivan E. "A head-mounted three dimensional display." Proceedings of the December 9-11, 1968, fall joint computer conference, part I. ACM, 1968.
16. Half a century ago… The Sword of Damocles • 1st paper describing a 3D HMD* • 4000 lines at 30 FPS • Mechanical rotation detection • Additionally 40 kHz ultrasound • 3 emitters • 4 receivers • The end goal was to build “Ultimate Display” with sound, force-feedback, taste, smell.** * Sutherland, Ivan E. "A head-mounted three dimensional display." Proceedings of the December 9-11, 1968, fall joint computer conference, part I. ACM, 1968. ** I. Sutherland: The Ultimate Display. Proceedings of IFIP Congress 2, pp. 506-509 (1965)
17. So what happened since then? On the surface of things? Not much….
18. So what happened since then? On the surface of things? Not much…. • Light instead of sound* * US Patent US20160131761A1 image courtesy of Valve Corporation
19. So what happened since then? On the surface of things? Not much…. • Light instead of sound* • Accelerometer instead of rotation * US Patent US20160131761A1 image courtesy of Valve Corporation
20. So what happened since then? On the surface of things? Not much…. • Light instead of sound* • Accelerometer instead of rotation • 2 more tracked objects * US Patent US20160131761A1 image courtesy of Valve Corporation
21. So what happened since then? On the surface of things? Not much…. • Light instead of sound* • Accelerometer instead of rotation • 2 more tracked objects • better graphics (thank you computer games) * US Patent US20160131761A1 image courtesy of Valve Corporation
22. So what happened since then? On the surface of things? Not much…. • Light instead of sound* • Accelerometer instead of rotation • 2 more tracked objects • better graphics (thank you computer games) • wireless communication * US Patent US20160131761A1 image courtesy of Valve Corporation
23. Actually... 1968 Sword of Damocles PC Performance 1956 First computer hard drive
24. Actually another 50 year old field happened 1968 Sword of Damocles 2012 Oculus Kickstarter → Industry takes over. A lot of wacky hardware (next slides) PC Performance 1956 First computer hard drive
25. Actually another 50 year old field happened 1963 Computer Vision Appears* 1968 Sword of Damocles *Roberts, Lawrence G. Machine perception of three-dimensional solids. Diss. Massachusetts Institute of Technology, 1963. **Krizhevsky, Alex, Ilya Sutskever, and Geoffrey E. Hinton. "Imagenet classification with deep convolutional neural networks." Advances in neural information processing systems. 2012. 2012 Oculus Kickstarter → Industry takes over. 2012 Deep Computer Vision** A lot of difficult math (not today) A lot of wacky hardware (next slides) PC Performance 1956 term AI is coined 1956 First computer hard drive
26. Simultaneous Localization and Mapping (SLAM) +From the lab to the living room: The story behind Facebook’s Oculus Insight technology and a new era of consumer VR • Localization - where in this place am I? • Mapping - given I am here, what is this place? • L&M - hen or egg?
27. Simultaneous Localization and Mapping (SLAM) • Deep Computer Vision to the rescue! • Not 100%, but maybe that’s ok? • Basically driving the whole new wave of VR by now. +From the lab to the living room: The story behind Facebook’s Oculus Insight technology and a new era of consumer VR • Localization - where in this place am I? • Mapping - given I am here, what is this place? • L&M - hen or egg?
28. Spatial computing - What Computer Vision is Doing for VR
29. Gestures The history of gestures is a history of gloves • 1977 Sayer Glove • based on photoresistors • actually an art project • Pretty much every other physical principle got turned into a glove at one point or another • CV hand tracking is coming +Picture courtesy of Daniel J. Sandin
30. Eye Tracking • Started with a magnetic prescription lense* • Followed up with light reflection. • Now done with built-in infrared cameras. • Extremely important to researchers. • Also - can display only what is being looked on? +Young, Laurence R., and David Sheena. "Survey of eye movement recording methods." Behavior research methods & instrumentation 7.5 (1975): 397-429. *A method of measuring eye movement using a scleral search coil in a magnetic field, IEEE Transactions on Bio-Medical Electronics, October 1963
31. Pose and remote operation • Basically started as a “full-body glove” • Required the development of high-speed wireless transfer. • For tasks that are: • Too dangerous • Too heavy • Requiring a specialist +Mazuryk, Tomasz, and Michael Gervautz. "Virtual reality-history, applications, technology and future." (1996).
32. Facial expression detection • Difficult because of the mask. • Important for interpersonal communication. • Leads to emotion detection (a rabbit hole for another day) • Currently in the works. +Li, Hao, et al. "Facial performance sensing head-mounted display." ACM Transactions on Graphics (ToG) 34.4 (2015): 47.
33. Remote operation 2 +Ishiguro, Yasuhiro, et al. "High speed whole body dynamic motion experiment with real time master-slave humanoid robot system." 2018 IEEE International Conference on Robotics and Automation (ICRA). IEEE, 2018. The use of feet and core re-introduces the need for external trackers
34. Pretty terrible so far: • Unstable sliding shoes • Expensive unstable treadmills • Walking in place • Redirected walking (see right) • Saccadic redirection* Combined with inside out → real-world redirection.** Movement in VR +Steinicke, Frank, et al. "Estimation of detection thresholds for redirected walking techniques." IEEE transactions on visualization and computer graphics 16.1 (2009): 17-27. *Sun, Qi, et al. "Towards virtual reality infinite walking: dynamic saccadic redirection." ACM Transactions on Graphics (TOG) 37.4 (2018): 67. **Yang, Jackie Junrui, et al. "DreamWalker: Substituting Real-World Walking Experiences with a Virtual Reality." Proceedings of the 32nd Annual ACM Symposium on User Interface Software and Technology. ACM, 2019.
35. Research into humans in VR
36. Presence - let’s clear something out* Presence: the sensation of being “present” in the virtual environment *Mütterlein, Joschka, and Thomas Hess. "Immersion, presence, interactivity: towards a joint understanding of factors influencing virtual reality acceptance and use." (2017).
37. Presence - let’s clear something out* Presence: the sensation of being “present” in the virtual environment Immersion: the ability to induce presence *Mütterlein, Joschka, and Thomas Hess. "Immersion, presence, interactivity: towards a joint understanding of factors influencing virtual reality acceptance and use." (2017).
38. Presence - related factors* Unrelated: • Personality (big 5) not related to experience of presence *Wallach, Helene S., et al. "Presence in Virtual Reality: Importance and Methods to Increase it." 2012). Virtual reality. New York: Nova Science Publishers. Hämtad från https://ebookcentral. proquest. com/lib/umeaubebooks/reader. action (2012).
39. Presence - related factors* Unrelated: • Personality (big 5) not related to experience of presence • Extroversion positively correlated (Laarni et al, 2004) *Wallach, Helene S., et al. "Presence in Virtual Reality: Importance and Methods to Increase it." 2012). Virtual reality. New York: Nova Science Publishers. Hämtad från https://ebookcentral. proquest. com/lib/umeaubebooks/reader. action (2012).
40. Presence - related factors* Unrelated: • Personality (big 5) not related to experience of presence • Extroversion positively correlated (Laarni et al, 2004) • Extroversion negatively correlated (Jurnet et al, 2005) *Wallach, Helene S., et al. "Presence in Virtual Reality: Importance and Methods to Increase it." 2012). Virtual reality. New York: Nova Science Publishers. Hämtad från https://ebookcentral. proquest. com/lib/umeaubebooks/reader. action (2012).
41. Presence - related factors* Unrelated: • Personality (big 5) not related to experience of presence • Extroversion positively correlated (Laarni et al, 2004) • Extroversion negatively correlated (Jurnet et al, 2005) Related: • Empathy, Imagination, Playfulness *Wallach, Helene S., et al. "Presence in Virtual Reality: Importance and Methods to Increase it." 2012). Virtual reality. New York: Nova Science Publishers. Hämtad från https://ebookcentral. proquest. com/lib/umeaubebooks/reader. action (2012).
42. Presence - related factors* Unrelated: • Personality (big 5) not related to experience of presence • Extroversion positively correlated (Laarni et al, 2004) • Extroversion negatively correlated (Jurnet et al, 2005) Related: • Empathy, Imagination, Playfulness • Sense of control over one’s life, anxiety *Wallach, Helene S., et al. "Presence in Virtual Reality: Importance and Methods to Increase it." 2012). Virtual reality. New York: Nova Science Publishers. Hämtad från https://ebookcentral. proquest. com/lib/umeaubebooks/reader. action (2012).
43. Presence - related factors* Unrelated: • Personality (big 5) not related to experience of presence • Extroversion positively correlated (Laarni et al, 2004) • Extroversion negatively correlated (Jurnet et al, 2005) Related: • Empathy, Imagination, Playfulness • Sense of control over one’s life, anxiety • Ethnicity and gender** *Wallach, Helene S., et al. "Presence in Virtual Reality: Importance and Methods to Increase it." 2012). Virtual reality. New York: Nova Science Publishers. Hämtad från https://ebookcentral. proquest. com/lib/umeaubebooks/reader. action (2012). **Almog, I., Wallach, H.S., and Safir, M.P. Ethnicity and sense of presence in a Virtual Environment: Arab women – a case in point. Accepted for publication in Virtual Rehabilitation 2009 International conference proceedings, April, 2009
44. Cybersickness • Commonly explained by the unproven “sensory conflict theory” • mismatch between visual and vestibular system • described e.g. in The Psychology of Sea-sickness, 1862
45. Cybersickness • Commonly explained by the unproven “sensory conflict theory” • mismatch between visual and vestibular system • described e.g. in The Psychology of Sea-sickness, 1862 • Strongly related to how much control user has over motion
46. Cybersickness • Commonly explained by the unproven “sensory conflict theory” • mismatch between visual and vestibular system • described e.g. in The Psychology of Sea-sickness, 1862 • Strongly related to how much control user has over motion • Also decreases presence *Tiiro, Arttu. "Effect of Visual Realism on Cybersickness in Virtual Reality." University of Oulu (2018). ** Clemes, Stacy A., and Peter A. Howarth. "The menstrual cycle and susceptibility to virtual simulation sickness." Journal of biological rhythms 20.1 (2005): 71-82.
47. Cybersickness • Commonly explained by the unproven “sensory conflict theory” • mismatch between visual and vestibular system • described e.g. in The Psychology of Sea-sickness, 1862 • Strongly related to how much control user has over motion • Also decreases presence • Increases with visual realism :( * *Tiiro, Arttu. "Effect of Visual Realism on Cybersickness in Virtual Reality." University of Oulu (2018).
48. Cybersickness • Commonly explained by the unproven “sensory conflict theory” • mismatch between visual and vestibular system • described e.g. in The Psychology of Sea-sickness, 1862 • Strongly related to how much control user has over motion • Also decreases presence • Increases with visual realism :( * • Gender imbalanced**: • Women faster to exhibit symptoms, faster to recover • Changes over the duration of menstrual cycle *Tiiro, Arttu. "Effect of Visual Realism on Cybersickness in Virtual Reality." University of Oulu (2018). ** Clemes, Stacy A., and Peter A. Howarth. "The menstrual cycle and susceptibility to virtual simulation sickness." Journal of biological rhythms 20.1 (2005): 71-82.
49. Immersion - too real? *Gonzalez-Franco, Mar, et al. "Participant concerns for the Learner in a Virtual Reality replication of the Milgram obedience study." PloS one 13.12 (2018): e0209704. • VR Milgram experiment saw psychological toll on subjects.* • Even within a VR simulation (...), participants are aware of the Learner’s suffering, reported being stressed by it, and help to try to prevent it
50. Immersion - too real? *Gonzalez-Franco, Mar, et al. "Participant concerns for the Learner in a Virtual Reality replication of the Milgram obedience study." PloS one 13.12 (2018): e0209704. **Lugrin, Jean-Luc, Johanna Latt, and Marc Erich Latoschik. "Avatar anthropomorphism and illusion of body ownership in VR." 2015 IEEE Virtual Reality (VR). IEEE, 2015. • VR Milgram experiment saw psychological toll on subjects.* • Even within a VR simulation (...), participants are aware of the Learner’s suffering, reported being stressed by it, and help to try to prevent it • Body ownership is real, even for obviously fake visuals.**
51. Immersion - too real? *Gonzalez-Franco, Mar, et al. "Participant concerns for the Learner in a Virtual Reality replication of the Milgram obedience study." PloS one 13.12 (2018): e0209704. **Lugrin, Jean-Luc, Johanna Latt, and Marc Erich Latoschik. "Avatar anthropomorphism and illusion of body ownership in VR." 2015 IEEE Virtual Reality (VR). IEEE, 2015. ***VR exposure was found to be as effective as exposure in vivo on anxiety and avoidance • VR Milgram experiment saw psychological toll on subjects.* • Even within a VR simulation (...), participants are aware of the Learner’s suffering, reported being stressed by it, and help to try to prevent it • Body ownership is real, even for obviously fake visuals.** • The source of fear in VR as fearsome as the real thing.***
52. Application of VR to humans
53. Psychotherapy* • Recall that fears can be easily induced in VR. • Treatment by controlled exposure to the source of fear. • flying, heights, spiders, public speaking* • In cases more effective than traditional therapy.** *North, Max M., Sarah M. North, and Joseph R. Coble. "Virtual Reality Therapy: An Effective Treatment for Psychological." Virtual reality in neuro-psycho-physiology: Cognitive, clinical and methodological issues in assessment and rehabilitation 44 (1997): 59. **Virtual reality compared with in vivo exposure in the treatment of social anxiety disorder: a three-arm randomised controlled trial. +Arachnophobia https://store.steampowered.com/app/485270/Arachnophobia/
54. Neuroscience *Enhancing the Ecological Validity of fMRI Memory Research Using Virtual Reality.
55. Neuroscience *Enhancing the Ecological Validity of fMRI Memory Research Using Virtual Reality **Minderer, Matthias, et al. "Neuroscience: virtual reality explored." Nature 533.7603 (2016): 324.
56. Rehabilitation • Useful for neurological injuries, primarily stroke. • Creates meaningfulness to training by providing feedback. • Brings advantages when combined with traditional therapy.* +Burdea, Grigore C., and Noshir A. Langrana. "Integrated virtual reality rehabilitation system." U.S. Patent No. 5,429,140. 4 Jul. 1995. *Advantages of virtual reality in the rehabilitation of balance and gait Desiderio Cano Porras, Petra Siemonsma, Rivka Inzelberg, Gabriel Zeilig, Meir Plotnik Neurology May 2018, 90 (22) 1017-1025;
57. Education&Training • Improves performance for mechanical tasks (e.g. surgery)* • Games more effective than virtual worlds and simulations** *Seymour, Neal E., et al. "Virtual reality training improves operating room performance: results of a randomized, double-blinded study." Annals of surgery 236.4 (2002): 458. **Merchant, Zahira, et al. "Effectiveness of virtual reality-based instruction on students' learning outcomes in K-12 and higher education: A meta-analysis." Computers & Education 70 (2014): 29-40.
58. And Finally, Self Promotion
59. Navigation neuroscience • Deterioration of navigational abilities with age.* • Research into grid cells - spatially oriented brain cells. *Diersch, Nadine, and Thomas Wolbers. "The potential of virtual reality for spatial navigation research across the adult lifespan." Journal of Experimental Biology 222.Suppl 1 (2019): jeb187252.
60. Treatment of social anxiety* • Treatment of social anxiety. • Logistically much easier than the real thing. • Can be combined with additional sensorik. *Streck, Adam, et al. "neomento-Towards Building a Universal Solution for Virtual Reality Exposure Psychotherapy." 2019 IEEE Conference on Games (CoG). IEEE, 2019.
61. Motion - preliminary results (unvalidated) • Reactions to movement in VR in elderly. • Heart rate not increased with discomfort. • Electrodermal activity has increased. • Similar to fear reaction.
62. Thank You* *In particular, thank you, random citizen!
63. Backup • Alternative theory - postural instability theory This hypothesis proposes that motion sickness occurs when combined motion and visual stimuli actually disturb postural stability, defined as the state in which uncontrolled movements of the kinetic system are minimized • Grid cells The arrangement of spatial firing fields, all at equal distances from their neighbors, led to a hypothesis that these cells encode a neural representation of Euclidean space.* Hafting, T.; Fyhn, M.; Molden, S.; Moser, M. -B.; Moser, E. I. (2005). "Microstructure of a spatial map in the entorhinal cortex". Nature.
64. Backup