Mixed Realities for Web

Mixed Realities for Web

My slides for my talk at Code.Talks 2018 about WebXR and Mixed Realities.

B1fa8b2208e7d7cc68da7fbb519dbc96?s=128

Carsten Sandtner

October 18, 2018
Tweet

Transcript

  1. MIXED REALITIES FOR WEB The immersive web and the WebXR

    device API Carsten Sandtner \\ @casarock Photo:: Bradley Hook https://www.pexels.com/photo/sky-woman-clouds-girl-123335/
  2. ABOUT://ME My name is Carsten and I’m Technical Director at

    mediaman GmbH in Mayence (Mainz). I’m a moz://a Techspeaker and I love the open web and everything about open standards for the web! I’m tweeting as @casarock 
 
 Enough about me, let’s get started!
  3. DID YOU KNOW?

  4. VIEWMASTER

  5. OCULUS RIFT • Kickstarter 2013 • Disruption in VR •

    Many other followed • HTC Vive • PS VR • etc. Image by Oculus VR, LLC
  6. OK, AND DID YOU KNOW..

  7. PROJECT HEADSIGHT

  8. MICROSOFT HOLOLENS

  9. MAGIC LEAP ONE

  10. 2017 - THE AR-REVOLUTION

  11. LET’S START WITH VR!

  12. https://developer.mozilla.org/en-US/docs/Web/API/WebVR_API/Concepts

  13. https://developer.mozilla.org/en-US/docs/Web/API/WebVR_API/Concepts

  14. None
  15. SENSORS https://developer.mozilla.org/en-US/docs/Web/API/WebVR_API/Concepts

  16. STEREOSCOPIC IMAGES https://developer.mozilla.org/en-US/docs/Web/API/WebVR_API/Concepts

  17. „ “ -Carsten Sandtner Mixed realities for WEB!

  18. THERE IS WEBVR

  19. „ “ Browser APIs that enable WebGL rendering to headsets

    and access to VR sensors
  20. AVAILABLE APIS VRDisplay VRDisplayCapabilities VRDisplayEvent VRFrameData VRPose VREyeParameters VRFieldOfView VRLayerInit

    VRStageParameters
  21. GET VR DISPLAYS var frameData = new VRFrameData(); var vrDisplay;

    navigator.getVRDisplays().then(function (displays) { vrDisplay = displays[0]; console.log('Display found'); // Starting the presentation when the button // is clicked: It can only be called in response to a user gesture btn.addEventListener('click', function () { vrDisplay.requestPresent([{ source: canvas }]) .then(function () { drawVRScene(); }); }); });
  22. DRAW YOUR VR SCENE function drawVRScene() { // WebVR: Request

    the next frame of the animation vrSceneFrame = vrDisplay.requestAnimationFrame(drawVRScene); // Populate frameData with the data of the next frame to display vrDisplay.getFrameData(frameData); // You can get the position, orientation, etc. // of the display from the current frame's pose // curFramePose is a VRPose object var curFramePose = frameData.pose; var curPos = curFramePose.position; var curOrient = curFramePose.orientation; // Here happens some WebGL Stuff! // WebVR: Indicate that we are ready to present // the rendered frame to the VR display vrDisplay.submitFrame(); }
  23. BUT THERE IS MORE… Import WebVR polyfill Set up camera

    Initialize scene Set up lights Declare and pass canvas Install VREffect Listen to window resize Create render loop Instantiate renderer Figure out responsiveness Preload assets Deal with metatags and mobile
  24. FRAMEWORK TO THE RESCUE!

  25. „ “ –https://aframe.io Use markup to create VR experiences that

    work across desktop, iOS, Android, and the Oculus Rift.
  26. LOOK, A CUBE! <!DOCTYPE html> <html> <head> <meta charset="utf-8"> <title>Hello,

    World!</title> <script src=“/aframe.min.js“></script> </head> <body> <a-scene> <a-box id="mybox" color="#6173F4" width=„1" height=„1" depth="1" position="1 1 1“ rotation="0 0 0" scale="1 1 1"> </a-box> <a-sky color="#bbb"></a-sky> </a-scene> </body> </html>
  27. LET’S ANIMATE! <!DOCTYPE html> <html> <head>. . .</head> <body> <a-scene>

    <a-box id="mybox" color="#6173F4" width=„1" height="1" depth="1" position="1 1 1“ rotation="0 0 0" scale="1 1 1"> <a-animation attribute="rotation" repeat="indefinite" to="0 180 0“> </a-animation> </a-box> </a-scene> </body> </html>
  28. None
  29. WEBVR IN THE WILD

  30. HTTP://WWW.360SYRIA.COM/

  31. HTTPS://AFRAME.IO/A-PAINTER/

  32. HTTPS://WWW.FORD.COM/SUVS/EXPEDITION/

  33. HTTPS://PEPSIGOBACK.COM/

  34. OK VR - BUT MIXED REALITIES?

  35. Reality Augmented Reality Mixed Reality Virtual Reality MIXED REALITY?

  36. https://www.gizmodo.com.au/2017/04/lemmings-in-hololens-adds-new-dimension-to-animal-suicide/

  37. https://thenextweb.com/insider/2016/06/16/the-force-is-strong-with-this-leap/

  38. DIFFERENT KINDS OF AR AND MR

  39. MARKER BASED AR https://www.flickr.com/photos/piair/8679229646/

  40. MARKERLESS AR https://www.flickr.com/photos/wikitude/30944213892/

  41. TRACKING

  42. Reality Augmented Reality Mixed Reality Virtual Reality XR - whateveryouwant

    Reality WHAT THE … XR?
  43. OK, AND WHY WEBXR?

  44. THERE IS WAS WEBVR

  45. WEBXR One API for AR, VR and MR Step towards

    the immersive Web Not compatible with WebVR! WebVR V2.0 -> 
 WebXR Device API
  46. LIFECYCLE OF A WEBXR EXPERIENCE

  47. EASY AS 1, 2, 3… 1. Ask for access to

    a device and detect XR modes 2. Request a XR session and allow starting the session 3. Within the session start your render loop with awesome graphics 4. Execute the render loop until the user has enough 5. Stop the XR Session - Clean up your mess. ,4, 5
  48. DETECT DEVICES! function checkForXRDevice() { navigator.xr.requestDevice().then(device => { // At

    least one Device found onXRAvailable(device); }, err => { if (err.name === 'NotFoundError') { // no device console.error('No XR Device: ', err); } else { // other error while accessing a device console.error('Error while requesting XR device: ', err); } }); } 1
  49. DETECT/REQUEST XR MODE let xrDevice = null; async function onXRAvailable(device)

    { xrDevice = device; // Check if the devices is capable for // immersive mode (VR). Offer a button to start your // Session xrDevice.supportsSession({ immersive: true }).then(() => { var enterXrBtn = document.createElement("button"); enterXrBtn.innerHTML = "Start VR-Mode“; enterXrBtn.addEventListener("click", beginXRSession); document.body.appendChild(enterXrBtn); }).catch((reason) => { console.log("Session not supported: " + reason); }); } 1
  50. START REQUESTED SESSION function beginXRSession() { // requestSession must be

    called within a // user gesture event like click or touch // when requesting an immersive session. xrDevice.requestSession({ immersive: true }) .then(onSessionStarted) .catch(err => { // Fallback when session couldn't // bestarted window.requestAnimationFrame(onDrawFrame); }); } 2
  51. START RENDER LOOP let xrSession = null; let xrFrameOfRef =

    null; function onSessionStarted(session) { // store the session xrSession = session; xrSession.requestFrameOfReference("head-model") .then((frameOfRef) => { xrFrameOfRef = frameOfRef; }) .then(setupWebGLLayer) .then(() => { // start your render loop xrSession.requestAnimationFrame(onDrawFrame); }); } 3
  52. SETUP RENDER LAYER let glCanvas = document.createElement("canvas"); let gl =

    glCanvas.getContext("webgl"); function setupWebGLLayer() { // check if canvas context is compatible return gl.setCompatibleXRDevice(xrDevice).then(() => { // baselayer shows your content! xrSession.baseLayer = new XRWebGLLayer(xrSession, gl); }); } 3
  53. DRAW YOUR SCENE function onDrawFrame(timestamp, xrFrame) { // Do we

    have an active session? if (xrSession) { let pose = xrFrame.getDevicePose(xrFrameOfRef); gl.bindFramebuffer(gl.FRAMEBUFFER, xrSession.baseLayer.framebuffer); for (let view of xrFrame.views) { let viewport = xrSession.baseLayer.getViewport(view); gl.viewport(viewport.x, viewport.y, viewport.width, viewport.height); drawScene(view, pose); } // Request the next animation callback xrSession.requestAnimationFrame(onDrawFrame); } else { // No session available, so render a default mono view. gl.viewport(0, 0, glCanvas.width, glCanvas.height); drawScene(); // Request the next window callback window.requestAnimationFrame(onDrawFrame); } } 4
  54. DRAW YOUR SCENE function drawScene(view, pose) { let viewMatrix =

    null; let projectionMatrix = null; if (view) { viewMatrix = pose.getViewMatrix(view); projectionMatrix = view.projectionMatrix; } else { viewMatrix = defaultViewMatrix; projectionMatrix = defaultProjectionMatrix; } // Set uniforms as appropriate for shaders being used // Draw Scene - This is where your awesome GFX-Skills // are needed! } 4
  55. CLEAN UP YOUR MESS… function endXRSession() { if (xrSession) {

    // stop active session xrSession.end().then(onSessionEnd); } } // Clean up function onSessionEnd() { gl.bindFramebuffer(gl.FRAMEBUFFER, null); xrSession = null; // If you still want to have a rendering outside // your XR Session, feel free! window.requestAnimationFrame(onDrawFrame); } 5
  56. THERE IS MORE Photo by Alexander Hafemann on Unsplash

  57. MIRRORING

  58. MAGIC WINDOWS

  59. INPUT HANDLING

  60. ROOM TRACKING

  61. AND MANY MORE https://github.com/immersive-web/webxr

  62. I WANT TO PLAY WITH IT! XR Viewer WebARonARKit WebARonARCore

  63. FRAMEWORK TO THE RESCUE, PART II!

  64. HTTPS://GITHUB.COM/MOZILLA/AFRAME-XR/

  65. WEBXR WITH A-FRAME <a-scene> <a-entity xr="ar: true; vr: false; magicWindow:

    false“> </a-entity> <!-- ... --> </a-scene>
  66. None
  67. ADVANTAGES OF WEBXR

  68. WEBXR IS PROGRESSIVE

  69. None
  70. NO GATEKEEPERS

  71. INSTANT

  72. CONNECTED

  73. None
  74. GO AND CREATE SOME AWESOME WEBXR STUFF! Thank you! @casarock

    for