$30 off During Our Annual Pro Sale. View Details »

Mixed Realities for Web

Mixed Realities for Web

My slides for my talk at Code.Talks 2018 about WebXR and Mixed Realities.

Carsten Sandtner

October 18, 2018
Tweet

More Decks by Carsten Sandtner

Other Decks in Technology

Transcript

  1. MIXED REALITIES
    FOR WEB
    The immersive web and the WebXR device API
    Carsten Sandtner \\ @casarock
    Photo:: Bradley Hook
    https://www.pexels.com/photo/sky-woman-clouds-girl-123335/

    View Slide

  2. ABOUT://ME
    My name is Carsten and I’m Technical Director at
    mediaman GmbH in Mayence (Mainz).
    I’m a moz://a Techspeaker and I love the open web and
    everything about open standards for the web!
    I’m tweeting as @casarock


    Enough about me, let’s get started!

    View Slide

  3. DID YOU KNOW?

    View Slide

  4. VIEWMASTER

    View Slide

  5. OCULUS RIFT
    • Kickstarter 2013
    • Disruption in VR
    • Many other followed
    • HTC Vive
    • PS VR
    • etc.
    Image by Oculus VR, LLC

    View Slide

  6. OK, AND DID YOU KNOW..

    View Slide

  7. PROJECT HEADSIGHT

    View Slide

  8. MICROSOFT HOLOLENS

    View Slide

  9. MAGIC LEAP ONE

    View Slide

  10. 2017 - THE AR-REVOLUTION

    View Slide

  11. LET’S START WITH VR!

    View Slide

  12. https://developer.mozilla.org/en-US/docs/Web/API/WebVR_API/Concepts

    View Slide

  13. https://developer.mozilla.org/en-US/docs/Web/API/WebVR_API/Concepts

    View Slide

  14. View Slide

  15. SENSORS
    https://developer.mozilla.org/en-US/docs/Web/API/WebVR_API/Concepts

    View Slide

  16. STEREOSCOPIC IMAGES
    https://developer.mozilla.org/en-US/docs/Web/API/WebVR_API/Concepts

    View Slide

  17. „ “
    -Carsten Sandtner
    Mixed realities for WEB!

    View Slide

  18. THERE IS WEBVR

    View Slide

  19. „ “
    Browser APIs that enable WebGL rendering
    to headsets and access to VR sensors

    View Slide

  20. AVAILABLE APIS
    VRDisplay
    VRDisplayCapabilities
    VRDisplayEvent
    VRFrameData
    VRPose
    VREyeParameters
    VRFieldOfView
    VRLayerInit
    VRStageParameters

    View Slide

  21. GET VR DISPLAYS
    var frameData = new VRFrameData();
    var vrDisplay;
    navigator.getVRDisplays().then(function (displays) {
    vrDisplay = displays[0];
    console.log('Display found');
    // Starting the presentation when the button
    // is clicked: It can only be called in response to a user gesture
    btn.addEventListener('click', function () {
    vrDisplay.requestPresent([{ source: canvas }])
    .then(function () {
    drawVRScene();
    });
    });
    });

    View Slide

  22. DRAW YOUR VR SCENE
    function drawVRScene() {
    // WebVR: Request the next frame of the animation
    vrSceneFrame = vrDisplay.requestAnimationFrame(drawVRScene);
    // Populate frameData with the data of the next frame to display
    vrDisplay.getFrameData(frameData);
    // You can get the position, orientation, etc.
    // of the display from the current frame's pose
    // curFramePose is a VRPose object
    var curFramePose = frameData.pose;
    var curPos = curFramePose.position;
    var curOrient = curFramePose.orientation;
    // Here happens some WebGL Stuff!
    // WebVR: Indicate that we are ready to present
    // the rendered frame to the VR display
    vrDisplay.submitFrame();
    }

    View Slide

  23. BUT THERE IS MORE…
    Import WebVR polyfill
    Set up camera
    Initialize scene
    Set up lights
    Declare and pass canvas
    Install VREffect
    Listen to window resize
    Create render loop
    Instantiate renderer
    Figure out responsiveness
    Preload assets
    Deal with metatags and mobile

    View Slide

  24. FRAMEWORK TO THE RESCUE!

    View Slide

  25. „ “
    –https://aframe.io
    Use markup to create VR experiences that
    work across desktop, iOS, Android, and the
    Oculus Rift.

    View Slide

  26. LOOK, A CUBE!




    Hello, World!




    height=„1" depth="1" position="1 1 1“
    rotation="0 0 0" scale="1 1 1">





    View Slide

  27. LET’S ANIMATE!


    . . .


    height="1" depth="1" position="1 1 1“
    rotation="0 0 0" scale="1 1 1">
    repeat="indefinite"
    to="0 180 0“>





    View Slide

  28. View Slide

  29. WEBVR IN THE WILD

    View Slide

  30. HTTP://WWW.360SYRIA.COM/

    View Slide

  31. HTTPS://AFRAME.IO/A-PAINTER/

    View Slide

  32. HTTPS://WWW.FORD.COM/SUVS/EXPEDITION/

    View Slide

  33. HTTPS://PEPSIGOBACK.COM/

    View Slide

  34. OK VR - BUT MIXED
    REALITIES?

    View Slide

  35. Reality
    Augmented
    Reality
    Mixed
    Reality
    Virtual
    Reality
    MIXED REALITY?

    View Slide

  36. https://www.gizmodo.com.au/2017/04/lemmings-in-hololens-adds-new-dimension-to-animal-suicide/

    View Slide

  37. https://thenextweb.com/insider/2016/06/16/the-force-is-strong-with-this-leap/

    View Slide

  38. DIFFERENT KINDS OF
    AR AND MR

    View Slide

  39. MARKER BASED AR
    https://www.flickr.com/photos/piair/8679229646/

    View Slide

  40. MARKERLESS AR
    https://www.flickr.com/photos/wikitude/30944213892/

    View Slide

  41. TRACKING

    View Slide

  42. Reality
    Augmented
    Reality
    Mixed
    Reality
    Virtual
    Reality
    XR -
    whateveryouwant
    Reality
    WHAT THE … XR?

    View Slide

  43. OK, AND WHY WEBXR?

    View Slide

  44. THERE IS WAS WEBVR

    View Slide

  45. WEBXR
    One API for AR, VR and MR
    Step towards the immersive
    Web
    Not compatible with WebVR!
    WebVR V2.0 -> 

    WebXR Device API

    View Slide

  46. LIFECYCLE OF A WEBXR
    EXPERIENCE

    View Slide

  47. EASY AS 1, 2, 3…
    1. Ask for access to a device and detect XR modes
    2. Request a XR session and allow starting the session
    3. Within the session start your render loop with awesome
    graphics
    4. Execute the render loop until the user has enough
    5. Stop the XR Session - Clean up your mess.
    ,4, 5

    View Slide

  48. DETECT DEVICES!
    function checkForXRDevice() {
    navigator.xr.requestDevice().then(device => {
    // At least one Device found
    onXRAvailable(device);
    }, err => {
    if (err.name === 'NotFoundError') {
    // no device
    console.error('No XR Device: ', err);
    } else {
    // other error while accessing a device
    console.error('Error while requesting XR device: ', err);
    }
    });
    }
    1

    View Slide

  49. DETECT/REQUEST XR MODE
    let xrDevice = null;
    async function onXRAvailable(device) {
    xrDevice = device;
    // Check if the devices is capable for
    // immersive mode (VR). Offer a button to start your
    // Session
    xrDevice.supportsSession({ immersive: true }).then(() => {
    var enterXrBtn = document.createElement("button");
    enterXrBtn.innerHTML = "Start VR-Mode“;
    enterXrBtn.addEventListener("click", beginXRSession);
    document.body.appendChild(enterXrBtn);
    }).catch((reason) => {
    console.log("Session not supported: " + reason);
    });
    }
    1

    View Slide

  50. START REQUESTED SESSION
    function beginXRSession() {
    // requestSession must be called within a
    // user gesture event like click or touch
    // when requesting an immersive session.
    xrDevice.requestSession({ immersive: true })
    .then(onSessionStarted)
    .catch(err => {
    // Fallback when session couldn't
    // bestarted
    window.requestAnimationFrame(onDrawFrame);
    });
    }
    2

    View Slide

  51. START RENDER LOOP
    let xrSession = null;
    let xrFrameOfRef = null;
    function onSessionStarted(session) {
    // store the session
    xrSession = session;
    xrSession.requestFrameOfReference("head-model")
    .then((frameOfRef) => {
    xrFrameOfRef = frameOfRef;
    })
    .then(setupWebGLLayer)
    .then(() => {
    // start your render loop
    xrSession.requestAnimationFrame(onDrawFrame);
    });
    }
    3

    View Slide

  52. SETUP RENDER LAYER
    let glCanvas = document.createElement("canvas");
    let gl = glCanvas.getContext("webgl");
    function setupWebGLLayer() {
    // check if canvas context is compatible
    return gl.setCompatibleXRDevice(xrDevice).then(() => {
    // baselayer shows your content!
    xrSession.baseLayer = new XRWebGLLayer(xrSession, gl);
    });
    }
    3

    View Slide

  53. DRAW YOUR SCENE
    function onDrawFrame(timestamp, xrFrame) {
    // Do we have an active session?
    if (xrSession) {
    let pose = xrFrame.getDevicePose(xrFrameOfRef);
    gl.bindFramebuffer(gl.FRAMEBUFFER, xrSession.baseLayer.framebuffer);
    for (let view of xrFrame.views) {
    let viewport = xrSession.baseLayer.getViewport(view);
    gl.viewport(viewport.x, viewport.y, viewport.width, viewport.height);
    drawScene(view, pose);
    }
    // Request the next animation callback
    xrSession.requestAnimationFrame(onDrawFrame);
    } else {
    // No session available, so render a default mono view.
    gl.viewport(0, 0, glCanvas.width, glCanvas.height);
    drawScene();
    // Request the next window callback
    window.requestAnimationFrame(onDrawFrame);
    }
    }
    4

    View Slide

  54. DRAW YOUR SCENE
    function drawScene(view, pose) {
    let viewMatrix = null;
    let projectionMatrix = null;
    if (view) {
    viewMatrix = pose.getViewMatrix(view);
    projectionMatrix = view.projectionMatrix;
    } else {
    viewMatrix = defaultViewMatrix;
    projectionMatrix = defaultProjectionMatrix;
    }
    // Set uniforms as appropriate for shaders being used
    // Draw Scene - This is where your awesome GFX-Skills
    // are needed!
    }
    4

    View Slide

  55. CLEAN UP YOUR MESS…
    function endXRSession() {
    if (xrSession) {
    // stop active session
    xrSession.end().then(onSessionEnd);
    }
    }
    // Clean up
    function onSessionEnd() {
    gl.bindFramebuffer(gl.FRAMEBUFFER, null);
    xrSession = null;
    // If you still want to have a rendering outside
    // your XR Session, feel free!
    window.requestAnimationFrame(onDrawFrame);
    }
    5

    View Slide

  56. THERE IS MORE
    Photo by Alexander Hafemann on Unsplash

    View Slide

  57. MIRRORING

    View Slide

  58. MAGIC WINDOWS

    View Slide

  59. INPUT HANDLING

    View Slide

  60. ROOM TRACKING

    View Slide

  61. AND MANY MORE
    https://github.com/immersive-web/webxr

    View Slide

  62. I WANT TO PLAY WITH IT!
    XR Viewer
    WebARonARKit WebARonARCore

    View Slide

  63. FRAMEWORK TO THE RESCUE, PART II!

    View Slide

  64. HTTPS://GITHUB.COM/MOZILLA/AFRAME-XR/

    View Slide

  65. WEBXR WITH A-FRAME

    View Slide

  66. View Slide

  67. ADVANTAGES OF WEBXR

    View Slide

  68. WEBXR IS PROGRESSIVE

    View Slide

  69. View Slide

  70. NO GATEKEEPERS

    View Slide

  71. INSTANT

    View Slide

  72. CONNECTED

    View Slide

  73. View Slide

  74. GO AND CREATE SOME
    AWESOME WEBXR STUFF!
    Thank you!
    @casarock for

    View Slide