WebXR: Beyond WebGL

Transcript

1. WebXR
   Beyond

   WebGL
   HTML5 Conference 2018 | 2018-11-25
   

   View Slide

2. Frontend Engineer
   at PixelGrid, Inc.
   Akihiro Oyamada
   @yomotsu
   

   View Slide

3. https://www.youtube.com/watch?v=ttDLGV5IJCM
   

   View Slide

4. This will possible on

   every mobile phone
   

   View Slide

5. 1. WebXR Device API
   

   View Slide

6. • The API for both AR and VR

   (WebVR API will be replaced by WebXR Device API)
   • Close relationship

   with Khronos’ OpenXR
   WebXR Device API
   

   View Slide

7. • Currently, only available in

   Chrome Canary with flags
   • Only works on either

   https or localhost
   As of Nov. 2018
   

   View Slide

8. Connect your Android Chrome to localhost

   with DevTools on desktop
   

   View Slide

9. FYI:
   Origin Trials Token
   is required

   except localhost
   https://github.com/GoogleChrome/OriginTrials/blob/gh-pages/developer-guide.md
   

   View Slide

10. View Slide

11. View Slide

12. Consist of 2 sources
    

    View Slide

13. WebGL scene
    (Under your control)
    XR imaginary

    (Camera input)
    WebGL framebuffer
    

    View Slide

14. WebGL framebuffer
    

    View Slide

15. How does it work?
    

    View Slide

16. How does it work?
    const device = await navigator.xr.requestDevice();
    

    View Slide

17. How does it work?
    const device = await navigator.xr.requestDevice();
    

    View Slide

18. XRSession
    device.requestSession( {
    immersive: false,
    outputContext: xrContext,
    environmentIntegration: true,
    } );
    

    View Slide

19. XRSession
    const glCanvas = document.createElement( 'canvas' );
    const gl = glCanvas.getContext( 'webgl', { xrCompatible: true } );
    xrSession.baseLayer = new XRWebGLLayer( xrSession, gl );
    WebGL context to be used

    as a source for XR imagery
    

    View Slide

20. The framebuffer can be filled with XR imagery
    gl.bindFramebuffer( gl.FRAMEBUFFER, xrSession.baseLayer.framebuffer );
    

    View Slide

21. Draw your 3D scene on it
    

    View Slide

22. View Slide

23. frameOfReference
    Real world coordinate
    

    View Slide

24. XRFrame

    (delta)
    frameOfReference
    

    View Slide

25. xrFrame.view[n].projectionMatrix

    

    const pose =
    xrFrame.getDevicePose(frameOfRef)
    pose.getViewMatrix( view )
    frameOfReference
    XRFrame

    (delta)
    Position and

    Orientation
    

    View Slide

26. Demo
    

    View Slide

27. Dive into the code!
    

    View Slide

28. const width = window.innerWidth;
    const height = window.innerHeight;
    navigator.xr.requestDevice().then( ( device ) => {
    const outputCanvas = document.getElementById( 'xrCanvas' );
    outputCanvas.width = width;
    outputCanvas.height = height;
    const xrContext = outputCanvas.getContext( 'xrpresent' );
    // session request must be done in user action such as click
    window.addEventListener( 'click', onEnterAR );
    async function onEnterAR() {
    const xrSession = await device.requestSession( {
    outputContext: xrContext,
    environmentIntegration: true,
    } );
    const renderer = new THREE.WebGLRenderer();
    renderer.autoClear = false;
    renderer.setSize( width, height );
    

    View Slide

29. const width = window.innerWidth;
    const height = window.innerHeight;
    navigator.xr.requestDevice().then( ( device ) => {
    const outputCanvas = document.getElementById( 'xrCanvas' );
    outputCanvas.width = width;
    outputCanvas.height = height;
    const xrContext = outputCanvas.getContext( 'xrpresent' );
    // session request must be done in user action such as click
    window.addEventListener( 'click', onEnterAR );
    async function onEnterAR() {
    const xrSession = await device.requestSession( {
    outputContext: xrContext,
    environmentIntegration: true,
    } );
    const renderer = new THREE.WebGLRenderer();
    renderer.autoClear = false;
    renderer.setSize( width, height );
    

    View Slide

30. const width = window.innerWidth;
    const height = window.innerHeight;
    navigator.xr.requestDevice().then( ( device ) => {
    const outputCanvas = document.getElementById( 'xrCanvas' );
    outputCanvas.width = width;
    outputCanvas.height = height;
    const xrContext = outputCanvas.getContext( 'xrpresent' );
    // session request must be done in user action such as click
    window.addEventListener( 'click', onEnterAR );
    async function onEnterAR() {
    const xrSession = await device.requestSession( {
    outputContext: xrContext,
    environmentIntegration: true,
    } );
    const renderer = new THREE.WebGLRenderer();
    renderer.autoClear = false;
    renderer.setSize( width, height );
    

    View Slide

31. window.addEventListener( 'click', onEnterAR );
    async function onEnterAR() {
    const xrSession = await device.requestSession( {
    outputContext: xrContext,
    environmentIntegration: true,
    } );
    const renderer = new THREE.WebGLRenderer();
    renderer.autoClear = false;
    renderer.setSize( width, height );
    // bind gl context to XR session
    const gl = renderer.getContext();
    gl.setCompatibleXRDevice( xrSession.device );
    xrSession.baseLayer = new XRWebGLLayer( xrSession, gl );
    const scene = new THREE.Scene();
    const camera = new THREE.PerspectiveCamera();
    camera.matrixAutoUpdate = false;
    const box = new THREE.Mesh(
    new THREE.BoxBufferGeometry( .2, .2, .2 ),
    new THREE.MeshNormalMaterial()
    

    View Slide

32. window.addEventListener( 'click', onEnterAR );
    async function onEnterAR() {
    const xrSession = await device.requestSession( {
    outputContext: xrContext,
    environmentIntegration: true,
    } );
    const renderer = new THREE.WebGLRenderer();
    renderer.autoClear = false;
    renderer.setSize( width, height );
    // bind gl context to XR session
    const gl = renderer.getContext();
    gl.setCompatibleXRDevice( xrSession.device );
    xrSession.baseLayer = new XRWebGLLayer( xrSession, gl );
    const scene = new THREE.Scene();
    const camera = new THREE.PerspectiveCamera();
    camera.matrixAutoUpdate = false;
    const box = new THREE.Mesh(
    new THREE.BoxBufferGeometry( .2, .2, .2 ),
    new THREE.MeshNormalMaterial()
    

    View Slide

33. window.addEventListener( 'click', onEnterAR );
    async function onEnterAR() {
    const xrSession = await device.requestSession( {
    outputContext: xrContext,
    environmentIntegration: true,
    } );
    const renderer = new THREE.WebGLRenderer();
    renderer.autoClear = false;
    renderer.setSize( width, height );
    // bind gl context to XR session
    const gl = renderer.getContext();
    gl.setCompatibleXRDevice( xrSession.device );
    xrSession.baseLayer = new XRWebGLLayer( xrSession, gl );
    const scene = new THREE.Scene();
    const camera = new THREE.PerspectiveCamera();
    camera.matrixAutoUpdate = false;
    const box = new THREE.Mesh(
    new THREE.BoxBufferGeometry( .2, .2, .2 ),
    new THREE.MeshNormalMaterial()
    

    View Slide

34. xrSession.baseLayer = new XRWebGLLayer( xrSession, gl );
    const scene = new THREE.Scene();
    const camera = new THREE.PerspectiveCamera();
    camera.matrixAutoUpdate = false;
    const box = new THREE.Mesh(
    new THREE.BoxBufferGeometry( .2, .2, .2 ),
    new THREE.MeshNormalMaterial()
    );
    scene.add( box );
    const frameOfRef = await xrSession.requestFrameOfReference( 'eye-level' );
    xrSession.requestAnimationFrame( onDrawFrame );
    function onDrawFrame( timestamp, xrFrame ) {
    const session = xrFrame.session; // xrSession === xrFrame.session
    const pose = xrFrame.getDevicePose( frameOfRef );
    session.requestAnimationFrame( onDrawFrame );
    gl.bindFramebuffer( gl.FRAMEBUFFER, session.baseLayer.framebuffer );
    

    View Slide

35. scene.add( box );
    const frameOfRef = await xrSession.requestFrameOfReference( 'eye-level' );
    xrSession.requestAnimationFrame( onDrawFrame );
    function onDrawFrame( timestamp, xrFrame ) {
    const session = xrFrame.session; // xrSession === xrFrame.session
    const pose = xrFrame.getDevicePose( frameOfRef );
    session.requestAnimationFrame( onDrawFrame );
    gl.bindFramebuffer( gl.FRAMEBUFFER, session.baseLayer.framebuffer );
    if ( ! pose ) return;
    // if the session is for both right and left eyes, length of views would be 2.
    // if not, length is 1,
    xrFrame.views.forEach( ( view ) => {
    const viewport = session.baseLayer.getViewport( view );
    renderer.setSize( viewport.width, viewport.height );
    

    View Slide

36. scene.add( box );
    const frameOfRef = await xrSession.requestFrameOfReference( 'eye-level' );
    xrSession.requestAnimationFrame( onDrawFrame );
    function onDrawFrame( timestamp, xrFrame ) {
    const session = xrFrame.session; // xrSession === xrFrame.session
    const pose = xrFrame.getDevicePose( frameOfRef );
    session.requestAnimationFrame( onDrawFrame );
    gl.bindFramebuffer( gl.FRAMEBUFFER, session.baseLayer.framebuffer );
    if ( ! pose ) return;
    // if the session is for both right and left eyes, length of views would be 2.
    // if not, length is 1,
    xrFrame.views.forEach( ( view ) => {
    const viewport = session.baseLayer.getViewport( view );
    renderer.setSize( viewport.width, viewport.height );
    

    View Slide

37. function onDrawFrame( timestamp, xrFrame ) {
    const session = xrFrame.session; // xrSession === xrFrame.session
    const pose = xrFrame.getDevicePose( frameOfRef );
    session.requestAnimationFrame( onDrawFrame );
    gl.bindFramebuffer( gl.FRAMEBUFFER, session.baseLayer.framebuffer );
    if ( ! pose ) return;
    // if the session is for both right and left eyes, length of views would be 2.
    // if not, length is 1,
    xrFrame.views.forEach( ( view ) => {
    const viewport = session.baseLayer.getViewport( view );
    renderer.setSize( viewport.width, viewport.height );
    camera.projectionMatrix.fromArray( view.projectionMatrix );
    const viewMatrix = new THREE.Matrix4().fromArray( pose.getViewMatrix( view ) );
    camera.matrix.getInverse( viewMatrix );
    camera.updateMatrixWorld( true );
    renderer.clearDepth();
    renderer.render( scene, camera );
    

    View Slide

38. function onDrawFrame( timestamp, xrFrame ) {
    const session = xrFrame.session; // xrSession === xrFrame.session
    const pose = xrFrame.getDevicePose( frameOfRef );
    session.requestAnimationFrame( onDrawFrame );
    gl.bindFramebuffer( gl.FRAMEBUFFER, session.baseLayer.framebuffer );
    if ( ! pose ) return;
    // if the session is for both right and left eyes, length of views would be 2.
    // if not, length is 1,
    xrFrame.views.forEach( ( view ) => {
    const viewport = session.baseLayer.getViewport( view );
    renderer.setSize( viewport.width, viewport.height );
    camera.projectionMatrix.fromArray( view.projectionMatrix );
    const viewMatrix = new THREE.Matrix4().fromArray( pose.getViewMatrix( view ) );
    camera.matrix.getInverse( viewMatrix );
    camera.updateMatrixWorld( true );
    renderer.clearDepth();
    renderer.render( scene, camera );
    

    View Slide

39. if ( ! pose ) return;
    // if the session is for both right and left eyes, length of views would be 2.
    // if not, length is 1,
    xrFrame.views.forEach( ( view ) => {
    const viewport = session.baseLayer.getViewport( view );
    renderer.setSize( viewport.width, viewport.height );
    camera.projectionMatrix.fromArray( view.projectionMatrix );
    const viewMatrix = new THREE.Matrix4().fromArray( pose.getViewMatrix( view ) );
    camera.matrix.getInverse( viewMatrix );
    camera.updateMatrixWorld( true );
    renderer.clearDepth();
    renderer.render( scene, camera );
    } );
    }
    }
    } );
    

    View Slide

40. Hit test
    

    View Slide

41. source: https://codelabs.developers.google.com/codelabs/ar-with-webxr/index.html#4
    Find intersection point

    that ray with the real-world surface
    

    View Slide

42. Demo
    

    View Slide

43. The code
    

    View Slide

44. const width = window.innerWidth;
    const height = window.innerHeight;
    const startButton = document.getElementById( 'startButton' );
    navigator.xr.requestDevice().then( ( device ) => {
    const outputCanvas = document.getElementById( 'xrCanvas' );
    outputCanvas.width = width;
    outputCanvas.height = height;
    const xrContext = outputCanvas.getContext( 'xrpresent' );
    // session request must be done in user action such as click
    startButton.addEventListener( 'click', onEnterAR );
    async function onEnterAR() {
    startButton.style.display = 'none';
    const xrSession = await device.requestSession( {
    The same as previous one…
    

    View Slide

45. renderer.render( scene, camera );
    } );
    }
    window.addEventListener( 'click', onClick );
    // use Raycaster to make ray origin and direction
    const raycaster = new THREE.Raycaster();
    // onClick must be async, since hitTest will be done with await
    async function onClick() {
    const x = 0;
    const y = 0;
    raycaster.setFromCamera( { x, y }, camera );
    const origin = new Float32Array( raycaster.ray.origin.toArray() );
    const direction = new Float32Array( raycaster.ray.direction.toArray() );
    const hits = await xrSession.requestHitTest( origin, direction, frameOfRef );
    if ( hits.length ) {
    Add a click action
    

    View Slide

46. renderer.render( scene, camera );
    } );
    }
    window.addEventListener( 'click', onClick );
    // use Raycaster to make ray origin and direction
    const raycaster = new THREE.Raycaster();
    // onClick must be async, since hitTest will be done with await
    async function onClick() {
    const x = 0;
    const y = 0;
    raycaster.setFromCamera( { x, y }, camera );
    const origin = new Float32Array( raycaster.ray.origin.toArray() );
    const direction = new Float32Array( raycaster.ray.direction.toArray() );
    const hits = await xrSession.requestHitTest( origin, direction, frameOfRef );
    if ( hits.length ) {
    

    View Slide

47. window.addEventListener( 'click', onClick );
    // use Raycaster to make ray origin and direction
    const raycaster = new THREE.Raycaster();
    // onClick must be async, since hitTest will be done with await
    async function onClick() {
    const x = 0;
    const y = 0;
    raycaster.setFromCamera( { x, y }, camera );
    const origin = new Float32Array( raycaster.ray.origin.toArray() );
    const direction = new Float32Array( raycaster.ray.direction.toArray() );
    const hits = await xrSession.requestHitTest( origin, direction, frameOfRef );
    if ( hits.length ) {
    const hit = hits[ 0 ];
    const hitMatrix = new THREE.Matrix4().fromArray( hit.hitMatrix );
    const box = new THREE.Mesh(
    

    View Slide

48. const raycaster = new THREE.Raycaster();
    // onClick must be async, since hitTest will be done with await
    async function onClick() {
    const x = 0;
    const y = 0;
    raycaster.setFromCamera( { x, y }, camera );
    const origin = new Float32Array( raycaster.ray.origin.toArray() );
    const direction = new Float32Array( raycaster.ray.direction.toArray() );
    const hits = await xrSession.requestHitTest( origin, direction, frameOfRef );
    if ( hits.length ) {
    const hit = hits[ 0 ];
    const hitMatrix = new THREE.Matrix4().fromArray( hit.hitMatrix );
    const box = new THREE.Mesh(
    new THREE.BoxBufferGeometry( .2, .2, .2 ),
    new THREE.MeshNormalMaterial()
    );
    

    View Slide

49. const raycaster = new THREE.Raycaster();
    // onClick must be async, since hitTest will be done with await
    async function onClick() {
    const x = 0;
    const y = 0;
    raycaster.setFromCamera( { x, y }, camera );
    const origin = new Float32Array( raycaster.ray.origin.toArray() );
    const direction = new Float32Array( raycaster.ray.direction.toArray() );
    const hits = await xrSession.requestHitTest( origin, direction, frameOfRef );
    if ( hits.length ) {
    const hit = hits[ 0 ];
    const hitMatrix = new THREE.Matrix4().fromArray( hit.hitMatrix );
    const box = new THREE.Mesh(
    new THREE.BoxBufferGeometry( .2, .2, .2 ),
    new THREE.MeshNormalMaterial()
    );
    

    View Slide

50. raycaster.setFromCamera( { x, y }, camera );
    const origin = new Float32Array( raycaster.ray.origin.toArray() );
    const direction = new Float32Array( raycaster.ray.direction.toArray() );
    const hits = await xrSession.requestHitTest( origin, direction, frameOfRef );
    if ( hits.length ) {
    const hit = hits[ 0 ];
    const hitMatrix = new THREE.Matrix4().fromArray( hit.hitMatrix );
    const box = new THREE.Mesh(
    new THREE.BoxBufferGeometry( .2, .2, .2 ),
    new THREE.MeshNormalMaterial()
    );
    box.position.setFromMatrixPosition( hitMatrix );
    scene.add( box );
    }
    }
    }
    

    View Slide

51. raycaster.setFromCamera( { x, y }, camera );
    const origin = new Float32Array( raycaster.ray.origin.toArray() );
    const direction = new Float32Array( raycaster.ray.direction.toArray() );
    const hits = await xrSession.requestHitTest( origin, direction, frameOfRef );
    if ( hits.length ) {
    const hit = hits[ 0 ];
    const hitMatrix = new THREE.Matrix4().fromArray( hit.hitMatrix );
    const box = new THREE.Mesh(
    new THREE.BoxBufferGeometry( .2, .2, .2 ),
    new THREE.MeshNormalMaterial()
    );
    box.position.setFromMatrixPosition( hitMatrix );
    scene.add( box );
    }
    }
    }
    

    View Slide

52. Demo
    52
    

    View Slide

53. How about iOS?
    

    View Slide

54. 2. AR Quick Look
    

    View Slide

55. • Upon the release of iOS 12
    • Only works exclusively on Safari

    (Doesn’t work even in iOS Chrome)
    • Special HTML syntax
    • Apple’s propriety
    AR Quick Look
    

    View Slide

56. 
    
    
    

    View Slide

57. 
    
    
    Link to usdz with rel="ar"
    Must contain one or
    

    View Slide

58. 58
    Demo
    

    View Slide

59. USDZ
    

    View Slide

60. • Stands for “Universal Scene Description”
    archived with Zip
    • 3D model format for AR Quick Look
    • Created by Apple and Pixar
    USDZ
    

    View Slide

61. • With “usdz_converter”

    which is a Xcode command line

    Only for macOS
    • Vectary (Web service)

    Cannot configure details such as size
    How to prepare USDZ
    

    View Slide

62. • Up to 10M Polygons
    • Up to 10 seconds for the animation
    • Up to 2048×2048 texture size
    Limitations
    

    View Slide

63. $ xcrun usdz_converter ./my-model.obj my-model.usdz
    -color_map albedo.jpg -metallic_map metallic.jpg

    -roughness_map roughness.jpg -normal_map normal.jpg
    -ao_map ao.jpg -emissive_map emissive.jpg
    USDZ Converter
    Terminal.app
    

    View Slide

64. $ xcrun usdz_converter ./my-model.obj my-model.usdz
    -color_map albedo.jpg -metallic_map metallic.jpg

    -roughness_map roughness.jpg -normal_map normal.jpg
    -ao_map ao.jpg -emissive_map emissive.jpg
    Terminal.app
    

    View Slide

65. $ xcrun usdz_converter ./my-model.obj my-model.usdz
    -color_map albedo.jpg -metallic_map metallic.jpg

    -roughness_map roughness.jpg -normal_map normal.jpg
    -ao_map ao.jpg -emissive_map emissive.jpg
    Terminal.app
    File input Output name
    Input option name Option value
    

    View Slide

66. https://www.vectary.com/
    Vectary
    

    View Slide

67. https://www.vectary.com/
    

    View Slide

68. https://www.vectary.com/
    USDZ export
    

    View Slide

69. • Apple's propriety

    (Hopefully it’s a temporary spec until WebXR Device API)
    • Some limitations of USDZ
    • Just pop and show in AR

    (Cannot be utilized as game and others)
    

    View Slide

70. 3D Model format

    for Web Apps?
    

    View Slide

71. 3. glTF
    

    View Slide

72. • Stands for GL Transmission Format
    • Open standard 3D model format
    • JPEG of 3D
    • Maintained by Khronos
    What is glTF
    

    View Slide

73. • JSON format as the container with binary payloads

    or packed single binary file called glb
    • Animation supported
    • Extensible just like WebGL Spec

    (Like Adobe Fireworks PNG)
    What is glTF
    

    View Slide

74. View Slide

75. • Supported by many 3D modeling tools
    • Several WebGL libraries support glTF loading

    (Such as three.js, BabylonJS, Cesium)
    • Microsoft Paint3D, Office and others use glb as 3D model format
    • Adobe Animate has glTF exporter
    • Facebook's 3D posts use glb
    • VRM: glTF extended format for humanoid avatar

    (For Virtual-YouTuber, VRChat and others)
    glTF of the present
    

    View Slide

76. https://www.youtube.com/watch?v=H2XoeQmkchw
    

    View Slide

77. https://www.youtube.com/watch?v=zzDM42PdqZk
    

    View Slide

78. Load glTF in three.js
    

    View Slide

79. const width = window.innerWidth;
    const height = window.innerHeight;
    const scene = new THREE.Scene();
    const camera = new THREE.PerspectiveCamera( 45, width / height,
    0.001, 100 );
    camera.position.set( 0, 0, 0.5 );
    const renderer = new THREE.WebGLRenderer();
    renderer.setSize( width, height );
    renderer.gammaInput = true;
    renderer.gammaOutput = true;
    document.body.appendChild( renderer.domElement );
    scene.add( new THREE.HemisphereLight( 0xffffff , 0x332222 ) );
    

    View Slide

80. document.body.appendChild( renderer.domElement );
    scene.add( new THREE.HemisphereLight( 0xffffff , 0x332222 ) );
    const loader = new THREE.GLTFLoader();
    loader.load(
    './models/barger/barger.gltf',
    function ( gltf ) {
    scene.add( gltf.scene );
    }
    );
    ( function anim () {
    requestAnimationFrame( anim );
    

    View Slide

81. scene.add( gltf.scene );
    }
    );
    ( function anim () {
    requestAnimationFrame( anim );
    renderer.render( scene, camera );
    } )();
    

    View Slide

82. View Slide

83. Demo
    83
    

    View Slide

84. glTF Animation
    

    View Slide

85. Demo
    

    View Slide

86. glTF is very popular

    as the standard
    

    View Slide

87. Security issues
    

    View Slide

88. • Gaze tracking

    Detect virtual keyboard type with gaze direction in a VR environment
    • Trusted environment

    Motion sickness by low FPS and pose track error
    • Fingerprinting

    Identify users room shape or even face shape with depth
    

    View Slide

89. Conclusion
    

    View Slide

90. XR is coming to the Web
    

    View Slide

91. • A Web API (in development)
    • For both VR and AR
    • Just works on WebBrowsers

    no add-ons nor installation required
    WebXR Device API
    

    View Slide

92. • AR for iOS
    • Using USDZ
    • Basic AR feature
    AR Quick Look
    

    View Slide

93. • The standard
    • 3D model format in JSON or Binary
    • Can be seen many places
    • Loaders are available in JavaScript
    glTF
    

    View Slide

94. Web will be connected

    to the real world
    94
    Source: https://www.netflix.com/jp/title/80182418 © ࢜࿠ਖ਼फɾProduction I.G/ߨஊࣾɾ߈֪ػಈୂ੡࡞ҕһձ
    

    View Slide

95. gl.finish();
    @yomotsu
    

    View Slide