Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Virtual and Augmented Reality - Lecture 10 - Next Generation User Interfaces (4018166FNR)

1135dc242dcff3b90ae46fc586ff4da8?s=47 Beat Signer
November 30, 2020

Virtual and Augmented Reality - Lecture 10 - Next Generation User Interfaces (4018166FNR)

This lecture forms part of a course on Next Generation User Interfaces given at the Vrije Universiteit Brussel.

1135dc242dcff3b90ae46fc586ff4da8?s=128

Beat Signer

November 30, 2020
Tweet

Transcript

  1. 2 December 2005 Next Generation User Interfaces Virtual and Augmented

    Reality Prof. Beat Signer Department of Computer Science Vrije Universiteit Brussel beatsigner.com
  2. Beat Signer - Department of Computer Science - bsigner@vub.ac.be 2

    November 30, 2020 Mixed Reality ▪ Reality–Virtuality continuum ▪ introduced by Paul Milgram and Fumio Kishino in 1994 ▪ Merging of real and virtual environments ▪ physical and digital objects co-exist and interact in real time ▪ mixed reality is the spectrum between the real environment and the purely virtual environment ▪ augmented reality as well as augmented virtuality Mixed Reality (MR) Real Environment Virtual Environ- ment (VR) Augmented Reality (AR) Augmented Virtuality (AV)
  3. Beat Signer - Department of Computer Science - bsigner@vub.ac.be 3

    November 30, 2020 Virtual Reality ▪ Virtual Reality (VR) is an artificial environment which is experienced through sensory stimuli (e.g. sight or sound) provided by a computer and in which a user’s actions partially determine what happens in the environment ▪ Main issues ▪ create acceptable substitutes for real world objects or environments ▪ sense the virtual environment ▪ navigate through the virtual environment ▪ interact with the virtual environment ▪ Opportunities ▪ experience situations that would be too dangerous or expensive in the real world
  4. Beat Signer - Department of Computer Science - bsigner@vub.ac.be 4

    November 30, 2020 The Sword of Damocles ▪ First virtual reality (VR) system and augmented reality (AR) head-mounted display (HMD) ▪ developed by Ivan Sutherland and his student Bob Sproull in 1968 ▪ binocular display with simple stereoscopic view of wireframe rooms ▪ heavy HMD attached to a mechanical arm suspended from the ceiling - head tracking to change the perspective shown by the software
  5. Beat Signer - Department of Computer Science - bsigner@vub.ac.be 5

    November 30, 2020 Virtual Reality Applications ▪ Architecture ▪ experience and navigate in virtual buildings ▪ Education ▪ visualisation and interaction with complex data ▪ Medicine ▪ training ▪ virtual robotic surgery ▪ Engineering ▪ Military ▪ Entertainment
  6. Beat Signer - Department of Computer Science - bsigner@vub.ac.be 6

    November 30, 2020 Virtual Reality Applications … ▪ Sport ▪ Simulations ▪ Modelling ▪ Information Systems ▪ Fashion ▪ Games ▪ …
  7. Beat Signer - Department of Computer Science - bsigner@vub.ac.be 7

    November 30, 2020 Immersion ▪ Perceptual immersion (physical immersion or sensory immersion) is about the perception of being physically present in a non-physical virtual environment which is created by surrounding images, sound or other stimuli by the VR system ▪ panoramic 3D vision ▪ 3D surround sound ▪ touch and force feedback ▪ taste ▪ smell ▪ direct connection to the human nervous system
  8. Beat Signer - Department of Computer Science - bsigner@vub.ac.be 8

    November 30, 2020 Non-Immersive Virtual Reality ▪ Non-immersive virtual environments show a real-time 3D environment on a desktop screen ▪ typical applications: CAD, simulations, 3D computer games, … ▪ also called "desktop virtual reality" ▪ Continuum from non-immersive to partially immersive and fully immersive systems
  9. Beat Signer - Department of Computer Science - bsigner@vub.ac.be 9

    November 30, 2020 Virtual Reality Technologies ▪ Large screens ▪ Binocular Omni-Orientation Monitor (BOOM) ▪ Cave Automatic Virtual Environment (CAVE) ▪ Head-mounted Display (HMD)
  10. Beat Signer - Department of Computer Science - bsigner@vub.ac.be 10

    November 30, 2020 Large Screens ▪ Large (panoramic) screen displays ▪ flat displays ▪ cylindrical displays (up to 360°) ▪ hemispherical or spherical displays ▪ Displays can be implemented with or without stereoscopy
  11. Beat Signer - Department of Computer Science - bsigner@vub.ac.be 11

    November 30, 2020 BOOM ▪ Binocular Omni-Orientation Monitor ▪ Head-coupled stereo- scopic display device ▪ Screens are housed in a box that is attached to a multi-link arm ▪ user looks into the box and can move the box to different positions ▪ head tracking is realised via sensors in the joints of the arm holding the box [http://www.umich.edu/~vrl/intro/AndreOnBoom.jpg]
  12. Beat Signer - Department of Computer Science - bsigner@vub.ac.be 12

    November 30, 2020 CAVE ▪ A Cave Automatic Virtual Environment (CAVE) provides the illusion of immersion by projecting stereo images on the walls and floor of a room-sized cube ▪ Users wear stereo glasses ▪ Head tracking system continuously adjusts the stereo projection to the current position of the leading viewer ▪ only one view for multiple users
  13. Beat Signer - Department of Computer Science - bsigner@vub.ac.be 13

    November 30, 2020 Head-mounted Display (HMD) ▪ A head-mounted display is a lightweight virtual reality device that the user wears to have video information directly displayed in front of their eyes ▪ one or two small displays (LCD, OLED) embedded in the helmet, glasses or visor ▪ lenses are used to give the perception that the images are coming from farther away - moves the virtual image to a distance that allows the eye to focus comfortably ▪ connected to a computer or stand-alone solutions such as the Oculus Quest Oculus Quest 2 virtual reality headset
  14. Beat Signer - Department of Computer Science - bsigner@vub.ac.be 14

    November 30, 2020 Navigation and Interaction ▪ Virtual reality is about creating computer-generated scenes in which a user can navigate and interact ▪ Navigation is the ability to move around and explore the features of the virtual environment (3D scene) ▪ walk within a virtual building ▪ Interaction involves the selection and moving of objects in a scene ▪ open a virtual door ▪ move an atom ▪ …
  15. Beat Signer - Department of Computer Science - bsigner@vub.ac.be 15

    November 30, 2020 VR Navigation Techniques ▪ Grabbing in the air ▪ user grabs points in the virtual world and drags and rotates them at will ▪ Lean-based velocity ▪ lean forwards or backwards to move in the virtual world ▪ Path drawing ▪ specify a path to be followed ▪ Walking in place ▪ might be supported via a treadmill Virtuix Omni
  16. Beat Signer - Department of Computer Science - bsigner@vub.ac.be 16

    November 30, 2020 VR Interaction Techniques ▪ Two types of interaction techniques ▪ non-immersive interaction - e.g. via mouse or joystick ▪ immersive interaction - e.g. using wearable device or capturing of limb positions ▪ Gesture recognition ▪ datagloves ▪ optical, vision-based gesture recognition Power Glove for Nintendo, Mattel, 1989
  17. Beat Signer - Department of Computer Science - bsigner@vub.ac.be 17

    November 30, 2020 VR Interaction Techniques … ▪ Interaction with the virtual world through ▪ virtual hand - map the position of a user’s hand to the virtual reality environment ▪ ray casting - virtual light ray is leaving a user’s hand and selects the first object that it hits ▪ image plane - an object is selected with a pointing gesture and manipulated via the other hand Ray casting
  18. Beat Signer - Department of Computer Science - bsigner@vub.ac.be 18

    November 30, 2020 Video: Haptic PIVOT
  19. Beat Signer - Department of Computer Science - bsigner@vub.ac.be 19

    November 30, 2020 Augmented Reality Augmented Reality (AR) is a variation of Virtual Environments (VE), or Virtual Reality as it is more commonly called. VE technologies completely immerse a user inside a synthetic environment. While immersed, the user cannot see the real world around him. In contrast, AR allows the user to see the real world, with virtual objects superimposed upon or composited with the real world. Therefore, AR supplements reality, rather than completely replacing it. Ideally, it would appear to the user that the virtual and real objects coexisted in the same space […] Ronald T. Azuma, A Survey of Augmented Reality, Teleoperators and Virtual Environments, 6(4), 1997
  20. Beat Signer - Department of Computer Science - bsigner@vub.ac.be 20

    November 30, 2020 Augmented Reality Applications ▪ Maintenance ▪ Architecture ▪ Education ▪ Medicine ▪ Entertainment ▪ Navigation ▪ Gaming ▪ Advertising ▪ …
  21. Beat Signer - Department of Computer Science - bsigner@vub.ac.be 21

    November 30, 2020 Augmented Reality Techniques ▪ Video compositing ▪ Head-up displays ▪ Direct projection ▪ Magic lens metaphor ▪ Magic mirror metaphor ▪ Magic eyeglass metaphor
  22. Beat Signer - Department of Computer Science - bsigner@vub.ac.be 22

    November 30, 2020 Video Compositing ▪ Virtual information is overlaid in a video stream of a real scene ▪ can happen in real time or in post processing LiberoVision
  23. Beat Signer - Department of Computer Science - bsigner@vub.ac.be 23

    November 30, 2020 Video: Augmented Life Music Performance
  24. Beat Signer - Department of Computer Science - bsigner@vub.ac.be 24

    November 30, 2020 Head-up Displays (HUDs) ▪ Head-up displays are used in civil and military aircrafts as well as in some cars ▪ The overlaid information is generally not directly connected to the objects seen through the window ▪ weak blend between virtuality and reality
  25. Beat Signer - Department of Computer Science - bsigner@vub.ac.be 25

    November 30, 2020 SixthSense (Direct Projection) ▪ Wearable augmented reality interface ▪ small camera and projector ▪ developed at MIT Media Lab - Pranav Mistry and Pattie Maes ▪ Visionary wearable augmented reality system ▪ what happened to the SixthSense?
  26. Beat Signer - Department of Computer Science - bsigner@vub.ac.be 26

    November 30, 2020 Video: SixthSense
  27. Beat Signer - Department of Computer Science - bsigner@vub.ac.be 27

    November 30, 2020 Wikitude World Browser (Magic Lens) ▪ The WIKITUDE World browser presents information about nearby physical landmarks as well as content added by other users ▪ Real-time augmentation of mobile phone camera view ▪ location-based augmented reality based on GPS, compass and accelerometer ▪ WIKITUDE.me authoring tool to add points of interest ▪ WIKITUDE API for augmented reality applications
  28. Beat Signer - Department of Computer Science - bsigner@vub.ac.be 28

    November 30, 2020 Google Sky Map ▪ Mobile phone application that can be used as a magic lens to get information about stars in the sky ▪ Real-time augmentation of the sky based on the mobile phone’s position (e.g. via GPS) and orientation
  29. Beat Signer - Department of Computer Science - bsigner@vub.ac.be 29

    November 30, 2020 ARToolKit ▪ Tracking library to overlay virtual imagery ▪ Calculates real-time camera position and orientation relative to square physical markers ▪ Fast enough for real-time AR applications ▪ Free and open source ▪ https://github.com/artoolkitx ▪ Multiple spinoffs ▪ ARTag, MRToolkit, osgART, ARToolKitPlus
  30. Beat Signer - Department of Computer Science - bsigner@vub.ac.be 30

    November 30, 2020 Video: IKEA Augmented Reality
  31. Beat Signer - Department of Computer Science - bsigner@vub.ac.be 31

    November 30, 2020 Magic Mirror Metaphor ▪ Technically the concept of a magic mirror is very similar to the magic lens, except for the orientation of the camera ▪ Typically used to overlay information on the user ▪ e.g. in fashion
  32. Beat Signer - Department of Computer Science - bsigner@vub.ac.be 32

    November 30, 2020 Magic Eyeglass Techniques ▪ See-through head mounted displays ▪ Virtual images mixed with a real view of the world ▪ Three kinds of see-through HMDs ▪ optical see-through HMDs ▪ video see-through HMDs ▪ virtual retinal displays
  33. Beat Signer - Department of Computer Science - bsigner@vub.ac.be 33

    November 30, 2020 Optical See-through HMDs ▪ With optical see-through HMDs, the virtual images are produced on semi-transparent surfaces (LCD panels) or reflected on semi-transparent mirrors ▪ example: Microsoft HoloLens
  34. Beat Signer - Department of Computer Science - bsigner@vub.ac.be 34

    November 30, 2020 Video See-through HMDs ▪ With video see-through HMDs, real video images are captured by one or two video cameras installed in the unit and overlaid with computer graphics (virtual) images
  35. Beat Signer - Department of Computer Science - bsigner@vub.ac.be 35

    November 30, 2020 Virtual Retinal Displays (VRD) ▪ Virtual retinal display (VRD) or retinal scan display (RSD) ▪ Projects three modulated light beams directly onto the retina of the eye producing a rasterised image ▪ Illusion of seeing the source image like a conventional display floating in space in front of the eye ▪ example: Google Glass
  36. Beat Signer - Department of Computer Science - bsigner@vub.ac.be 36

    November 30, 2020 Google Glass ▪ Android-based headset ▪ Bluetooth connection to mobile phone ▪ display with "picture in picture" experience ▪ camera and microphone ▪ voice and touch-based interaction ▪ What about using voice navigation in public space? ▪ do people for example use Siri in public space? ▪ What about privacy and safety? ▪ recording pictures and movies or recognising people’s faces ▪ What about the social acceptance of Google Glass?
  37. Beat Signer - Department of Computer Science - bsigner@vub.ac.be 37

    November 30, 2020 Video: Looking Through Google Glass
  38. Beat Signer - Department of Computer Science - bsigner@vub.ac.be 38

    November 30, 2020 Microsoft HoloLens 2 ▪ Lenses to combine superimposed images ▪ optical waveguides ▪ Integrated processor ▪ all the processing happens on the HoloLens ▪ Depth camera and other sensors to "understand" the environment
  39. Beat Signer - Department of Computer Science - bsigner@vub.ac.be 39

    November 30, 2020 Video: HoloLens 2 on Stage
  40. Beat Signer - Department of Computer Science - bsigner@vub.ac.be 40

    November 30, 2020 Video: HoloLens 2 at Philips
  41. Beat Signer - Department of Computer Science - bsigner@vub.ac.be 41

    November 30, 2020 Video: Dynamics 365 Remote Assist
  42. Beat Signer - Department of Computer Science - bsigner@vub.ac.be 42

    November 30, 2020 WebXR ▪ New Web-based specification (W3C) for the development of VR as well as AR applications ▪ implement once and run on different platforms ▪ Device API ▪ see-through AR on mobile phones, VR headsets, AR headsets, … ▪ Room Tracking API ▪ detection of surfaces and walls ▪ inside-out or outside-in tracking ▪ API for native features ▪ controllers, hand tracking, spatial audio, haptic feedback, … ▪ You will get more details in the WebXR exercise session
  43. Beat Signer - Department of Computer Science - bsigner@vub.ac.be 43

    November 30, 2020 The Future of Augmented Reality? ▪ New forms of less distracting augmentations ▪ e.g. augmented contact lenses ▪ Improvements in tracking technologies ▪ e.g. see Microsoft HoloLens ▪ What about the social acceptance? B.A. Parviz, University of Washington
  44. Beat Signer - Department of Computer Science - bsigner@vub.ac.be 44

    November 30, 2020 References ▪ P. Milgram and F. Kishino, Taxonomy of Mixed Reality Visual Displays, IEICE Transactions on Information Systems, December 1994 ▪ https://www.researchgate.net/publication/231514051_A_Taxonomy_of_Mi xed_Reality_Visual_Displays ▪ SixthSense ▪ https://www.youtube.com/watch?v=nZ-VjUKAsao ▪ Looking Through Google Glass: Real Life Example ▪ https://www.youtube.com/watch?v=d-y3bEjEVV8 ▪ Place IKEA Furniture in Your Home ▪ https://www.youtube.com/watch?v=vDNzTasuYEw
  45. Beat Signer - Department of Computer Science - bsigner@vub.ac.be 45

    November 30, 2020 References … ▪ T. Azuma, A Survey of Augmented Reality, Teleoperators and Virtual Environments, 6(4), 1997 ▪ https://www.cs.unc.edu/~azuma/ARpresence.pdf ▪ Augmented Reality During Live Music Performances ▪ https://www.youtube.com/watch?v=nyVs_5TfN4c ▪ L. Hoste and B. Signer, Expressive Control of Indirect Augmented Reality During Live Music Performances, Proceedings of NIME 2013, 13th International Con- ference on New Interfaces for Musical Expression, Daejeon, Korea Republic, May 2013 ▪ https://beatsigner.com/publications/hoste_NIME2013.pdf
  46. Beat Signer - Department of Computer Science - bsigner@vub.ac.be 46

    November 30, 2020 References … ▪ Haptic PIVOT by Microsoft Research ▪ https://www.youtube.com/watch?v=kj3RdeJUJos ▪ Microsoft HoloLens 2 on Stage ▪ https://www.youtube.com/watch?v=uIHPPtPBgHk ▪ Microsoft HoloLens 2 at Philips ▪ https://www.youtube.com/watch?v=loGxO3L7rFE ▪ Dynamics 365 Remote Assist and HoloLens 2 ▪ https://www.youtube.com/watch?v=d3YT8j0yYl0 ▪ Interview with Fabien Bénétou on WebXR ▪ https://www.inthepocket.com/blog/our-conversation-with-fabien- benetou
  47. 2 December 2005 Next Lecture Implicit and Cross-Device Interaction