A Proposal of Interactive Projection Mapping Using Kinect

ICT-ROBOT 2018

Transcript

1. A Proposal of Interactive Projection Mapping Using Kinect University of Miyazaki : M.Sakamoto,
   T.Shinoda, M.Hori, T.Ishizu, H.Watanabe, A.Takei Hiroshima University : Takao Ito
2. Contents 1. Introduction 2. Methods 3. Result 4. Discussion 5. Conclusion
3. Contents 1. Introduction 2. Methods 3. Result 4. Discussion 5. Conclusion
4. 1. Introduction • EC (Entertainment Computing) • EC represented by games etc. will greatly entertain us. • Among them, we focused on projection mapping.
5. 1. Introduction • In recent years, projection mapping is increasingly showing excitement. • Mapping to Buildings. • Mapping to clothing, face, notes and other familiar objects. • Artist's concert. etc… Fig 1 : Tokyo Disneyland Once upon a time. Fig 2 : Perfume Cannes Lions International Festival of Creativity
6. 1. Introduction
   Conventional projection mapping • In the conventional projection mapping, since viewers enjoy mostly by watching projected images, feeling of immersion in viewers’ contents is insufficient. • We thought that it would be possible to solve the problem if the viewer became a participant. → Propose projects to participants and propose interactive projection mapping that changes with the participant’s movements.
7. 1. Introduction What is projection mapping ? • Video technique to synchronize real and virtual. • To paste the image material on the surface layer of the projection target. • It is being used in a wide range of fields ranging from entertainment to medical treatment.
8. Contents 1. Introduction 2. Methods 3. Result 4. Discussion 5. Conclusion
9. 2. Methods • Equipment used • Kinect for Windows Kinect for Windows is a peripheral device that enables operation by body movement, gesture and voice without using a controller. • Projector (NEC NP50J) Used for projection Fig 3 : Kinect for Windows. Infrared sensor RGB camera Depth image sensor Microphone
10. 2. Methods • Development environment • OS : Windows10 • Integration : Visual Studio 2017 • Language : C++ • Library • OpenNI2 • NiTE2 • OpenCV • OpenGL Used to retrieve data from Kinect Used for image acquisition, output, drawing
11. 2. Methods • System overview Screen display on PC Fig 4 : “Ball”. Fig 5 : “Color”. Fig 6 : “Depth”. Fig 7 : “User”. Fig 8 : “Combination”, and “Combination_PC”.
12. 2. Methods • Skeleton Numbers are allocated as shown in Fig 10. → Pose detection. Fig 9 : “Skeleton” screen display. Fig 10 : “Skeleton” number. Center of the Head Center of the Body
    Neck Left waist Left shoulder Right waist Right shoulder Left knee Left elbow Right knee Right elbow Left foot Left hand Right foot Right hand
13. 2. Methods • Projection mapping changes according to the movement of the subject by projecting "Combination" to the subject. • Baseball mode (Pitcher’s movement) • Soccer mode (Lifting movement) We implemented baseball and football movements for right-handed people.
14. 2. Methods • At system startup : Recognition of the skeleton becomes unstable. → It may switch to an unintended mode. We pose “Attention”. • When a series of operations of the mode is completed, initialization is performed. → Continuous projection possible. • Before switching to each mode. → The screen remains black.
15. 2. Methods We pose “Attention”. Fig 10 : “Skeleton” number. • . " −
    . " < 100 • |. " − . "| < 150 and
16. 2. Methods • Baseball mode • As you pitcher's movement, baseball balls and baseball stadiums are projected. • As the flight distance increases, the ball gradually decreases. • When the position of the ball becomes larger than the threshold value of the !-coordinate the ball disappears and is initialized.
17. 2. Methods • Baseball mode Fig 11 : Screen display of “Combination” when playing baseball (No background). 1. Hold hands around chest. 2. Roll up so that the right hand is above the head, the ball appears and starts to move.
18. 2. Methods • Baseball mode • It is recognized by the following six conditions. Fig 10 : “Skeleton” number. • . " − . " < 180 • . ( − . ( < 180 • . " − . " < 180 • . ( − . ( < 180 •
    . ( − . ( < 180 •
    . ( − . ( < 180
19. 2. Methods • Soccer mode • As you lifting movement, soccer balls and soccer stadiums are projected. • when the position of the ball is lower than the !-coordinate threshold the ball will disappear and be initialized.
20. 2. Methods • Soccer mode Fig 12 : Screen display of “Combination” when playing soccer (No background). 1. When the right knee is kicked up so as to be as high as the right waist, the ball appears. 2. When the right knee is returned to its original position, the ball falls.
21. 2. Methods • Soccer mode Fig 10 : “Skeleton” number. • . " − 300 <
    . "
22. 2. Methods • Soccer mode • The movement of lifting uses the following physical operation. • The !-coordinate upward is positive and the center of the screen of the projection mapping is set to ! = 0. Fig 13 : Physical operation used to implement the movement of lifting. y ! = $% & − 1 2 *&+ ! = $′% & − 1 2 *&+ take ! = 25 ∗ & − / + *&+ for $% = 25 $% , $′% : Initial velocity & : Time * : Gravitational acceleration (* = 9.8)
23. 2. Methods • Soccer mode Fig 14 : The position of the ball at the time of kicking up the ball. ∴ "# = 2&ℎ ∴ "′# = 4&ℎ In order to perform a continuous lifting operation, the highest point of the ball at
    and the height of the ball kicked up again are made equal. "* − "# * = 2 −& ℎ The speed of the highest point is 0 − "# * = 2 −& ℎ from " = 0 from
    "* − "- # * = 2 −& (2ℎ) The speed of the highest point is 0 − "- # * = 2 −& (2ℎ) from " = 0 from  0 = 35.4 ∗ 5 − 1 2 &5* "′# = 2"# "′# = 25 ∗ 2 ≈ 35.4 from the above y Fig 13 : Physical operation used to implement the movement of lifting.
24. Contents 1. Introduction 2. Methods 3. Result 4. Discussion 5. Conclusion
25. 3. Result • Baseball mode
26. 3. Result • Soccer mode
27. 3. Result • We conducted a questionnaire to 12 college students and evaluated this research.
28. 3. Result • Questionnaire Q1, Is the operation method easy to understand? Q2, The person who answered that it was hard to understand. → What point was hard to understand? Q3, Is there a feeling of immersion? Q4, Do you have any opinions or improvements? Q1 and Q3 were evaluated in 5 grades. Good evaluation is 5, bad evaluation is 1.
29. 3. Result • Answer Q1, Is the operation method easy to understand? → The average score is 4.75 Q3, Is there a feeling of immersion? → The average score is 3.67 0 1 2 3 4 5 operation method Immersion feeling Average score of the questionnaire Fig 15 : Average score of the questionnaire.
30. 3. Result • Answer Q4, Do you have any opinions or improvements?
    Good mark • It was good that it was possible to mapping continuously. • It was interesting that the ball moves according to the pose.
31. 3. Result • Answer Q4, Do you have any opinions or improvements?
    Improvement • Please change the speed of the ball according to the speed of operation. • I want you to add sound. • The movement of the ball is slightly behind. • I wish I could play more than one person in this system. • I want the image to change with not tired action.
32. Contents 1. Introduction 2. Methods 3. Result 4. Discussion 5. Conclusion
33. 4. Discussion • In terms of realizing mapping with a simple operation, high evaluation was obtained. → Many people can enjoy this projection mapping. • Regarding the feeling of immersion, We got the result that it is roughly the case, We also learned that there is still room for improvement.
34. 4. Discussion • "There is a feeling of immersion for those looking at projection mapping, but not for those who do it." → We think that because the subject cannot actually see the projection mapping. • “The movement of the ball is slightly behind.” → It is thought that there is a time lag between when Kinect recognizes the skeleton and reflects it in the projection mapping.
35. 4. Discussion • Future work • Make it available for left-handed people. • Addition of other movements of baseball / football such as batting and shooting. • Make it available for multiple people. • Add other events. • Texture mapping tailored to the event. • Play the sound when the ball is hit, change the speed.
36. Contents 1. Introduction 2. Methods 3. Result 4. Discussion 5. Conclusion
37. 5. Conclusion • In this study, Kinect was used to create an interactive projection mapping whose mapping changes according to human movement.
38. 5. Conclusion • If projection mapping that changes according to the movement of people increases like this in this way .... More realistic stage production. • In the future, We would like to realize the projection mapping which further increased the immersion feeling based on the improvement point. Thank you very much for kind attention.