Getting Started with WebAR for HoloLens2 and Meta Quest

Transcript

1. Getting Started with WebAR
   for HoloLens2 and Oculus Quest
   

   View Slide

2. Download Sample
   http://arfukuoka.lolipop.jp/
   WebXR2021/Sample.zip
   

   View Slide

3. Goal
   Let's try hand tracking using WebXR on HoloLens2 and Oculus Quest.
   https://youtu.be/y4WD2wgzKYQ
   

   View Slide

4. Letʼs Try
   You can try demo if setting to use
   WebXR is finished on your device.
   https://webxr-hand-drawing.glitch.me
   

   View Slide

5. Duplicate the Template
   https://glitch.com/~webxr-hand-template
   GET STARTED
   

   View Slide

6. Duplicate the Template
   Scroll
   

   View Slide

7. Duplicate the Template
   Remix Your Own
   

   View Slide

8. Check the Template
   Click on index.html to make sure the code is displayed.
   index.html
   

   View Slide

9. Check the Template
   Editor Preview
   Click on index.html to make sure the code is displayed.
   

   View Slide

10. Change the Content Name (URL)
    Click on the content name on the upper left corner of the screen.
    Click here
    

    View Slide

11. Change the Content Name (URL)
    Change each string to a name that is easier to understand.
    Change to a simple name
    

    View Slide

12. Check the URL
    Confirm the URL by clicking [Change URL] on the top
    of the preview screen.
    Change URL
    URL
    

    View Slide

13. Check on the Device
    Enter the URL
    AR Button
    Enter the URL
    VR Button
    HoloLens2 Quest
    

    View Slide

14. How it works (HoloLens2)
    A Cube will appear.
    

    View Slide

15. Tutorial Procedures
    1.Explain the overview of the sample.
    2.Add hand tracking feature.
    3.Prepare work with hand tracking by javascript.
    4.Describe the action of the right hand.
    a) Spawn an object when pinching starts.
    b) Spawn objects to the trajectory of the hand.
    c) Set the color of the next object to be added.
    when the pinch ends.
    5.Describe the action of the left hand.
    a) Delete all objects at the start of pinch
    6.Extra
    a) Always track the position of the index finger
    

    View Slide

16. Tutorial Procedures
    1.Explain the overview of the sample.
    2.Add hand tracking feature.
    3.Prepare work with hand tracking by javascript.
    4.Describe the action of the right hand.
    a) Spawn an object when pinching starts.
    b) Spawn objects to the trajectory of the hand.
    c) Set the color of the next object to be added.
    when the pinch ends.
    5.Describe the action of the left hand.
    a) Delete all objects at the start of pinch
    6.Extra
    a) Always track the position of the index finger
    

    View Slide

17. Overview of the Sample
    Lesson01
    

    View Slide

18. Overview of the Sample
    Loading A-Frame
    Describes content to be
    shown in 3D space.
    

    View Slide

19. Overview of the Sample
    
    color="#4CC3D9" shadow>
    
    a-box
    position
    0 1.5 -2
    

    View Slide

20. Overview of the Sample
    Describe the
    behavior of
    the hand later.
    

    View Slide

21. Overview of the Sample
    <br/>window.onload = function() {<br/>initRightHand(); //右⼿の初期化<br/>initLeftHand(); //左⼿の初期化<br/>};<br/>//右⼿の初期化を⾏う関数<br/>function initRightHand(){<br/>};<br/>//左⼿の初期化を⾏う関数<br/>function initLeftHand(){<br/>};<br/>
    Automatically called
    once the page has
    loaded.
    Function prepared to
    describe the behavior
    of the right hand.
    Function prepared to
    describe the behavior
    of the left hand.
    

    View Slide

22. Tutorial Procedures
    1.Explain the overview of the sample.
    2.Add hand tracking feature.
    3.Prepare work with hand tracking by javascript.
    4.Describe the action of the right hand.
    a) Spawn an object when pinching starts.
    b) Spawn objects to the trajectory of the hand.
    c) Set the color of the next object to be added.
    when the pinch ends.
    5.Describe the action of the left hand.
    a) Delete all objects at the start of pinch
    6.Extra
    a) Always track the position of the index finger
    

    View Slide

23. Show Hand Objects
    Edit here
    

    View Slide

24. Show Hand Objects
    
    color="#4CC3D9" shadow>
    
    hand-tracking-controls="hand: left; modelStyle: dots;">
    
    hand-tracking-controls="hand: right; modelStyle: dots;">
    
    【hand-tracking-controls】
    hand: left or right
    modelStyle: mesh pr dots
    (mesh doesnʼt work)
    modelColor: color of hand
    (Default is white) Lesson02
    

    View Slide

25. Check on the Device
    Be sure to click the Reload button.
    Reload
    Reload
    

    View Slide

26. Check on the Device
    

    View Slide

27. Tutorial Procedures
    1.Explain the overview of the sample.
    2.Add hand tracking feature.
    3.Prepare work with hand tracking by javascript.
    4.Describe the action of the right hand.
    a) Spawn an object when pinching starts.
    b) Spawn objects to the trajectory of the hand.
    c) Set the color of the next object to be added.
    when the pinch ends.
    5.Describe the action of the left hand.
    a) Delete all objects at the start of pinch
    6.Extra
    a) Always track the position of the index finger
    

    View Slide

28. Prepare Work with Hand Tracking
    Edit here
    

    View Slide

29. Prepare Work with Hand Tracking
    <br/>window.onload = function() {<br/>initRightHand(); //右⼿の初期化<br/>initLeftHand(); //左⼿の初期化<br/>};<br/>//右⼿の初期化を⾏う関数<br/>function initRightHand(){<br/>};<br/>//左⼿の初期化を⾏う関数<br/>function initLeftHand(){<br/>};<br/>
    Add variables here
    

    View Slide

30. Prepare Work with Hand Tracking
    var scene; //3D space to add 3D objects
    var rightHand; //Variable to access the right hand.
    var leftHand; //Variable to access the left hand.
    window.onload = function() {
    initRightHand(); //右⼿の初期化
    initLeftHand(); //左⼿の初期化
    };
    //右⼿の初期化を⾏う関数
    function initRightHand(){
    };
    //左⼿の初期化を⾏う関数
    function initLeftHand(){
    }; Lesson03
    

    View Slide

31. Prepare Work with Hand Tracking
    var scene; //3D space to add 3D objects
    var rightHand; //Variable to access the right hand
    var leftHand; //Variable to access the left hand
    window.onload = function() {
    // Keeps information in 3D space as variables.
    scene = document.querySelector('a-scene');
    initRightHand(); //右⼿の初期化
    initLeftHand(); //左⼿の初期化
    };
    //右⼿の初期化を⾏う関数
    function initRightHand(){
    };
    //左⼿の初期化を⾏う関数
    function initLeftHand(){ };
    Lesson04
    

    View Slide

32. Tutorial Procedures
    1.Explain the overview of the sample.
    2.Add hand tracking feature.
    3.Prepare work with hand tracking by javascript.
    4.Describe the action of the right hand.
    a) Spawn an object when pinching starts.
    b) Spawn objects to the trajectory of the hand.
    c) Set the color of the next object to be added.
    when the pinch ends.
    5.Describe the action of the left hand.
    a) Delete all objects at the start of pinch
    6.Extra
    a) Always track the position of the index finger
    

    View Slide

33. Create Object by Right Hand Operation
    var scene; //描画する3D空間
    var rightHand; //右⼿
    var leftHand; //左⼿
    window.onload = function() {
    //3D空間を変数として保持
    scene = document.querySelector('a-scene');
    initRightHand(); //右⼿の初期化
    initLeftHand(); //左⼿の初期化
    };
    //右⼿の初期化を⾏う関数
    function initRightHand(){
    };
    //左⼿の初期化を⾏う関数
    function initLeftHand(){ };
    Edit here
    

    View Slide

34. Create Object by Right Hand Operation
    function initRightHand(){
    //Get the right hand object (id=rightHand)
    rightHand = document.getElementById("rightHand");
    //Begin pinch
    rightHand.addEventListener('pinchstarted', function (e) {
    });
    //End pinch
    rightHand.addEventListener('pinchended', function (e) {
    });
    //Move hand while pinching
    rightHand.addEventListener('pinchmoved', function (e) {
    });
    }; Lesson05
    

    View Slide

35. Getting the Pinch Start
    function initRightHand(){
    //Get the right hand object (id=rightHand)
    rightHand = document.getElementById("rightHand");
    //Begin pinch
    rightHand.addEventListener('pinchstarted', function (e) {
    });
    //End pinch
    rightHand.addEventListener('pinchended', function (e) {
    });
    // Move hand while pinching
    rightHand.addEventListener('pinchmoved', function (e) {
    });
    };
    Edit here
    

    View Slide

36. Getting the Pinch Start
    function initRightHand(){
    // Get the right hand object (id=rightHand)
    rightHand = document.getElementById("rightHand");
    // Begin pinch
    rightHand.addEventListener('pinchstarted', function (e) {
    var position = e.detail.position;
    //Calling the addBOX function (explained in the next page)
    addBox(position);
    });
    // End pinch
    rightHand.addEventListener('pinchended', function (e) {
    });
    // Move hand while pinching
    rightHand.addEventListener('pinchmoved', function (e) {
    });
    }; Lesson06
    

    View Slide

37. Generating Box Object
    Add the function here
    

    View Slide

38. Generating Box Object
    //左⼿の初期化を⾏う関数
    function initLeftHand(){
    };
    function addBox(position){
    //Create an element named a-box
    var box = document.createElement('a-box');
    // Set the position, size, and color.
    box.setAttribute('position', position);
    box.setAttribute('scale', '0.02 0.02 0.02');
    box.setAttribute('color', '#0062C6');
    // Set class names for easy deletion later.
    box.setAttribute('class', 'box');
    // Add a box in 3D space.
    scene.appendChild(box);
    }
    Lesson07
    

    View Slide

39. Check on the Device
    

    View Slide

40. Tutorial Procedures
    1.Explain the overview of the sample.
    2.Add hand tracking feature.
    3.Prepare work with hand tracking by javascript.
    4.Describe the action of the right hand.
    a) Spawn an object when pinching starts.
    b) Spawn objects to the trajectory of the hand.
    c) Set the color of the next object to be added.
    when the pinch ends.
    5.Describe the action of the left hand.
    a) Delete all objects at the start of pinch
    6.Extra
    a) Always track the position of the index finger
    

    View Slide

41. Follows the Movement of the Hand
    function initRightHand(){
    // Get the right hand object (id=rightHand)
    rightHand = document.getElementById("rightHand");
    // Begin pinch
    rightHand.addEventListener('pinchstarted', function (e) {
    /*omitted*/
    });
    // End pinch
    rightHand.addEventListener('pinchended', function (e) {
    });
    // Move hand while pinching
    rightHand.addEventListener('pinchmoved', function (e) {
    });
    };
    Edit here
    

    View Slide

42. Follows the Movement of the Hand
    // Get the right hand object (id=rightHand)
    rightHand = document.getElementById("rightHand");
    // Begin pinch
    rightHand.addEventListener(ʻpinchstartedʼ, function (e) {
    /*omitted*/
    });
    // End pinch
    rightHand.addEventListener('pinchended', function (e) {
    });
    // Move hand while pinching
    rightHand.addEventListener('pinchmoved', function (e) {
    var position = e.detail.position;
    //generate a box
    addBox(position);
    }); Lesson08
    

    View Slide

43. Check on the Device
    

    View Slide

44. Problem
    If you generate the box too densely, it will quickly consume memory
    

    View Slide

45. Reduce the Density of Boxes
    <br/>var scene;<br/>var rightHand;<br/>var leftHand;<br/>var prev;//To store the position where the last box was placed.<br/>window.onload = function() {<br/>//3D空間を変数として保持<br/>scene = document.querySelector('a-scene');<br/>initRightHand();<br/>initLeftHand();<br/>};<br/>//右⼿の初期化を⾏う関数<br/>function initRightHand(){<br/>/*以下省略*/<br/>Lesson09<br/>

    View Slide

46. Reduce the Density of Boxes
    var prev;//最後にBox置いた位置を記憶
    window.onload = function() {
    /*omitted*/
    };
    //右⼿の初期化を⾏う関数
    function initRightHand(){
    /*omitted*/
    };
    //左⼿の初期化を⾏う関数
    function initLeftHand(){
    };
    //Function to add box
    function addBox(position){
    }
    Edit here
    

    View Slide

47. Reduce the Density of Boxes
    function addBox(position){
    // Create an element named a-box
    var box = document.createElement('a-box');
    // Set the position, size, and color
    box.setAttribute('position', position);
    box.setAttribute('scale', '0.02 0.02 0.02');
    box.setAttribute('color', '#0062C6');
    // Set class names for easy deletion later
    box.setAttribute('class', 'box');
    //Add a box in 3D space.
    scene.appendChild(box);
    // Memorize the position where the box was placed.
    prev=box.object3D.position;
    }
    Lesson09
    

    View Slide

48. Reduce the Density of Boxes
    var prev;//最後にBox置いた位置を記憶
    window.onload = function() {
    /*omitted*/
    };
    //右⼿の初期化を⾏う関数
    function initRightHand(){
    /*omitted*/
    };
    //左⼿の初期化を⾏う関数
    function initLeftHand(){
    };
    //Boxの追加を⾏う関数
    function addBox(position){
    }
    Edit here
    

    View Slide

49. Reduce the Density of Boxes
    function initRightHand(){
    //Get the right hand object (id=rightHand)
    rightHand = document.getElementById("rightHand");
    // Begin pinch
    rightHand.addEventListener('pinchstarted', function (e) {
    /*omitted*/
    });
    // End pinch
    rightHand.addEventListener('pinchended', function (e) {
    });
    // Move hand while pinching
    rightHand.addEventListener('pinchmoved', function (e) {
    });
    };
    Edit here
    

    View Slide

50. Reduce the Density of Boxes
    rightHand.addEventListener('pinchmoved', function (e) {
    var position = e.detail.position;
    // Distance between the last position of the box and
    // the current position of the hand
    var length= prev.distanceTo(position);
    if(length>=0.02){
    //Generate a box
    addBox(position);
    }
    });
    Lesson10
    prev
    position
    一定距離以上
    

    View Slide

51. Check on the Device
    

    View Slide

52. Tutorial Procedures
    1.Explain the overview of the sample.
    2.Add hand tracking feature.
    3.Prepare work with hand tracking by javascript.
    4.Describe the action of the right hand.
    a) Spawn an object when pinching starts.
    b) Spawn objects to the trajectory of the hand.
    c) Set the color of the next object to be added.
    when the pinch ends.
    5.Describe the action of the left hand.
    a) Delete all objects at the start of pinch
    6.Extra
    a) Always track the position of the index finger
    

    View Slide

53. Switch Color of Box
    var scene;
    var rightHand;
    var leftHand;
    var prev;
    // Array of colors to assign to a cube
    var colors = ['#0062C6', '#00AB00', '#FFCF00', '#7F00FF',
    '#FFFFFF', '#FF2000', '#F52394', '#593110'];
    // Index of color array to use
    var index=0;
    window.onload = function() {
    //3D空間を変数として保持
    scene = document.querySelector('a-scene');
    initRightHand(); //右⼿の初期化
    initLeftHand(); //左⼿の初期化
    }; Lesson11
    

    View Slide

54. Switch Color of Box
    function initRightHand(){
    // Get the right hand object (id=rightHand)
    rightHand = document.getElementById("rightHand");
    // Begin pinch
    rightHand.addEventListener('pinchstarted', function (e) {
    /*omitted*/
    });
    // End pinch
    rightHand.addEventListener('pinchended', function (e) {
    });
    // Move hand while pinching
    rightHand.addEventListener('pinchmoved', function (e) {
    /*omitted*/
    });
    };
    Edit here
    

    View Slide

55. Switch Color of Box
    function initRightHand(){
    // Get the right hand object (id=rightHand)
    rightHand = document.getElementById("rightHand");
    // Begin pinch
    rightHand.addEventListener('pinchstarted', function (e) {
    var position = e.detail.position;
    //Generate a box
    addBox(position);
    });
    // End pinch
    rightHand.addEventListener('pinchended', function (e) {
    // Add up the number of colors.
    index++;
    index%=colors.length;
    });
    Lesson12
    

    View Slide

56. Switch Color of Box
    window.onload = function() {
    /*省略*/
    };
    //右⼿の初期化を⾏う関数
    function initRightHand(){
    /*省略*/
    };
    //左⼿の初期化を⾏う関数
    function initLeftHand(){
    };
    //Function to generate a box
    function addBox(position){
    }
    Edit here
    

    View Slide

57. Switch Color of Box
    function addBox(position){
    // Create an element named a-box
    var box = document.createElement('a-box');
    // Set the position, size, and color
    box.setAttribute('position', position);
    box.setAttribute('scale', '0.02 0.02 0.02ʼ);
    box.setAttribute('color', '#0062C6');
    box.setAttribute('color', colors[index]);
    // Set class names for easy deletion later
    box.setAttribute('class', 'box');
    // Add a box in 3D space
    scene.appendChild(box);
    // Memorize the position where the box was placed
    prev=box.object3D.position;
    } Lesson13
    box.setAttribute('color', '#0062C6');
    

    View Slide

58. Chek on the Device
    

    View Slide

59. Tutorial Procedures
    1.Explain the overview of the sample.
    2.Add hand tracking feature.
    3.Prepare work with hand tracking by javascript.
    4.Describe the action of the right hand.
    a) Spawn an object when pinching starts.
    b) Spawn objects to the trajectory of the hand.
    c) Set the color of the next object to be added.
    when the pinch ends.
    5.Describe the action of the left hand.
    a) Delete all objects at the start of pinch
    6.Extra
    a) Always track the position of the index finger
    

    View Slide

60. Delete All Boxes
    var prev;//最後にBox置いた位置を記憶
    window.onload = function() {
    /*omitted*/
    };
    //右⼿の初期化を⾏う関数
    function initRightHand(){
    /* omitted */
    };
    //左⼿の初期化を⾏う関数
    function initLeftHand(){
    };
    //Function to generate a box
    function addBox(position){
    /*omitted */
    }
    Edit here
    

    View Slide

61. Delete All Boxes
    function initLeftHand(){
    // Get the left hand object.
    leftHand = document.getElementById("leftHand");
    // Behavior at the start of a pinch
    leftHand.addEventListener('pinchstarted', function (e) {
    // Get all objects to which the box class is assigned.
    var els = document.querySelectorAll('.boxʼ);
    // Delete the boxes one by one.
    for (var i = 0; i < els.length; i++) {
    els[i].parentNode.removeChild(els[i]);
    }
    });
    };
    Lesson14
    

    View Slide

62. Completed!
    

    View Slide

63. Tutorial Procedures
    1.Explain the overview of the sample.
    2.Add hand tracking feature.
    3.Prepare work with hand tracking by javascript.
    4.Describe the action of the right hand.
    a) Spawn an object when pinching starts.
    b) Spawn objects to the trajectory of the hand.
    c) Set the color of the next object to be added.
    when the pinch ends.
    5.Describe the action of the left hand.
    a) Delete all objects at the start of pinch
    6.Extra
    a) Always track the position of the index finger
    

    View Slide

64. Getting the Position of Finger Tip
    Edit inside a-scene
    

    View Slide

65. Getting the Position of Finger Tip
    
    rotation="0 45 0" color="#4CC3D9" shadow>
    
    
    
    color="#0062C6">
    
    Lesson15
    

    View Slide

66. Write Scripts
    See
    javascript
    

    View Slide

67. Getting the Position of Finger Tip
    // Array of colors to assign to a cube
    var colors = ['#0062C6', '#00AB00', '#FFCF00', '#7F00FF',
    '#FFFFFF', '#FF2000', '#F52394', '#593110'];
    // Index of color array to use
    var index=0;
    //Sphere representing the tip of the finger
    var tip;
    window.onload = function() {
    //3D空間を変数として保持
    scene = document.querySelector('a-scene');
    //Get the sphere representing the fingertip position
    tip = document.getElementById("tip");
    initRightHand(); //右⼿の初期化
    initLeftHand(); //左⼿の初期化
    };
    Lesson16
    

    View Slide

68. Getting the Position of Finger Tip
    var prev;//最後にBox置いた位置を記憶
    window.onload = function() {
    /*省略*/
    };
    //右⼿の初期化を⾏う関数
    function initRightHand(){
    /*omitted*/
    };
    //左⼿の初期化を⾏う関数
    function initLeftHand(){
    /*省略*/
    };
    //Boxの追加を⾏う関数
    function addBox(position){
    /*省略*/
    }
    Edit here
    

    View Slide

69. Getting the Position of Finger Tip
    function initRightHand(){
    // Get the right hand object (id=rightHand)
    rightHand = document.getElementById("rightHand");
    // Track finger tips in real time (to be implemented later)
    tipTracking(rightHand.components['hand-tracking-controls']);
    // Begin pinch
    rightHand.addEventListener('pinchstarted', function (e) {
    /*omitted*/
    });
    // End pinch
    rightHand.addEventListener('pinchended', function (e) {
    /*omitted*/
    });
    Lesson17
    

    View Slide

70. Getting the Position of Finger Tip
    window.onload = function() {
    /*omitted*/
    };
    function initRightHand(){
    /* omitted(右⼿の初期化を⾏う関数)*/
    };
    function initLeftHand(){
    /* omitted(左⼿の初期化を⾏う関数)*/
    };
    function addBox(position){
    /* omitted(Boxの追加を⾏う関数)*/
    }
    Add a function for tracking
    finger position here
    

    View Slide

71. Getting the Position of Finger Tip
    function initLeftHand(){
    /*omitted(左⼿の初期化を⾏う関数)*/
    };
    function addBox(position){
    /*omitted(Boxの追加を⾏う関数)*/
    }
    function tipTracking(ctrl){
    // Finger tip position
    var position = ctrl.indexTipPosition;
    // Assign finger position to the ball
    tip.setAttribute('position', position);
    // Call tipTracking again after 30ms
    setTimeout(function(){tipTracking(ctrl);}, 30);
    };
    Lesson18
    

    View Slide

72. Assign the Box Color to the Fingertip
    function initRightHand(){
    //右⼿のオブジェクト(id=rightHand)を取得
    rightHand = document.getElementById("rightHand");
    //ピンチ開始
    rightHand.addEventListener('pinchstarted', function (e) {
    /*中略*/
    });
    //End pinch
    rightHand.addEventListener('pinchended', function (e) {
    });
    //ピンチしながら⼿を移動
    rightHand.addEventListener('pinchmoved', function (e) {
    /*中略*/
    });
    };
    Edit here
    

    View Slide

73. 次のBoxの⾊を指先の球に反映
    // Begin pinch
    rightHand.addEventListener('pinchstarted', function (e) {
    var position = e.detail.position;
    //BOX⽣成関数を呼び出す
    addBox(position);
    });
    // End pinch
    rightHand.addEventListener('pinchended', function (e) {
    //⾊の番号を加算してboxの⾊に反映
    index++;
    index%=colors.length;
    // Assing the color to the ball at the tip of the finger
    tip.setAttribute('color', colors[index]);
    });
    Lesson19
    

    View Slide

74. Extra part was
    completed!
    

    View Slide