Study on participatory projection mapping that can be enjoyed by performers

Transcript

1. Study on participatory projection mapping that can be enjoyed by performers Takahiro Shinoda Graduate School of Engineering, University of Miyazaki, Japan
2. Agenda • Background • Precedent case • Purpose • Proposed method • Result • Evaluation experiment • Consideration • Future tasks • Conclusion 2
3. Background EC (Entertainment Computing) - In recent years, it has become more and more exciting - In this study, we focused on projection mapping 3 - EC
4. Background 4 Virtual Real New space ＋ ＝ - projection mapping
5. Precedent case Mapping to buildings - Tokyo Disneyland üMapping to Cinderella Castle 5 Fig.1. Once upon a time .
6. Precedent case Mapping to buildings - Rio Olympics üMainly 20,000 lumen projectors, 333 projectors were used 6 Fig.2. Rio Olympics opening ceremony.
7. Precedent case Events such as live - Cannes Lions "International Creativity Festival" üThere is also an example of projecting to the artist himself 7 Fig.3. Perfume Cannes Lions "International Creativity Festival".
8. Purpose 8 • Performers need to accurately align their motion with the coordinates of the image object in the projection mapping For many, it is difficult
9. Purpose 9 Not only people watching projection mapping but also performers enjoy!
10. Proposed method • We implemented the projection mapping by the following two methods - Method using skeleton coordinates - Method using cascade classifier 10
11. Proposed method 11 - Equipment used Fig.4. Kinect for Windows. Kinect for Windows Projector Fig.5. Projector (NEC NP50J). Infrared sensor RGB camera Depth image sensor Microphone
12. Proposed method • OS - Windows10 • IDE - Visual Studio 2017 • Programming language - C++ • Library - OpenNI2 - NiTE2 - OpenCV - OpenGL 12 - Development environment
13. Proposed method 13 - System overview Fig.6. Ball. Fig.8. Depth. Fig.9. User. Fig.10. Combination, (_PC). Fig.13. Cascade. Fig.12. Gray. Fig.11. Skeleton. Fig.7. Color.
14. Proposed method 14 - Skeleton number Fig.14. Skeleton number. num part name 0 Head 1 Neck 2 Left shoulder 3 Right shoulder 4 Left elbow 5 Right elbow 6 Left hand 7 Right hand 8 Torso 9 Left waist 10 Right waist 11 Left knee 12 Right knee 13 Left foot 14 Right foot
15. Proposed method 15 - baseball mode Fig.16. "Combination" screen when running baseball mode. Fig.15. baseball ground.
16. Proposed method 1. Align your hands near your chest - (-coordinate of left elbow) − (-coordinate of torso) < 200 - (-coordinate of left elbow) − (-coordinate of torso) < 200 - (-coordinate of right elbow) − (-coordinate of torso) < 200 - (-coordinate of right elbow) − (-coordinate of torso) < 200 - (-coordinate of neck) − (-coordinate of left hand) < 200 - (-coordinate of neck) − (-coordinate of right hand) < 200 2. Raise your hand so that your hand is above your head - (-coordinates of the right (left) hand) > (-coordinate of head) 16 - baseball mode
17. Proposed method 17 - soccer mode Fig.18. "Combination" screen when running soccer mode. Fig.17. soccer ground.
18. Proposed method 18 - soccer mode 1. Raise knee to waist high - (-coordinate of right (left) knee) > (-coordinate of right (left) waist − 300) 2. The ball keeps falling when you lower your knees
19. Proposed method 19 - soccer mode • Kinect has a problem of selecting a target person for skeleton tracking randomly from recognized persons. If the user hides in Kinect's field of view and then enters Kinect's field of view again, there is a problem that the re-following of the user's skeleton coordinates may not be executed properly [1]. [1] 濱砂雅⼈, 伊藤暢浩, 幸塚義之, “⼈ごみにおけるKinectセンサの 誤認追跡の改善について”, 30th Fuzzy System Symposium, Kochi, September 1-3, 2014.
20. Proposed method 20 - soccer mode Fig.20. Negative image. Fig.19. Positive image. image The number of samples Positive image (lifting) 1200 Negative image (not lifting) 345
21. Proposed method • Boosting - A learning algorithm that sequentially generates weak classifiers and combines them to create a strong classifier • AdaBoots - One of the boosting methods - A method to create a classifier with high accuracy by adaptively weighting and learning the recognition rate of the classifier during the learning process 21 - soccer mode Fig.18. AdaBoots. ℎ+ : ℎ : ℎ
22. Proposed method • Image feature extraction - Haar-like features - Local Binary Pattern (LBP) features - Histogram of Oriented Gradients (HOG) features 22 - soccer mode
23. Proposed method • Image feature extraction - Haar-like features - Local Binary Pattern (LBP) features - Histogram of Oriented Gradients (HOG) features 23 - soccer mode
24. Proposed method • Local Binary Pattern (LBP) features 24 - soccer mode
25. Proposed method 25 - soccer mode 6 5 2 7 6 1 9 8 7 1 0 0 1 0 1 1 1 1 2 4 128 8 64 32 16 1 0 0 128 0 64 32 16 3x3 brightness value Binarization × Weight of each pixel brightness value after calculation × Attention pixel LBP = 1 + 16 + 32 + 64 + 128 = 241
26. Result 26 Fig.19. Execution result of soccer mode. Fig.20. Execution result of baseball mode.
27. Evaluation experiment 27 - Evaluation method Fig.21. Evaluation experiment environment diagram.
28. Evaluation experiment Q1. Is the operation easy to understand? Q2. If not, what was it hard to understand? Q3. Did you enjoy it? Q4. Are there any future improvements? 28 - Evaluation method In order to obtain the evaluation of the projection mapping proposed in this study, we asked five subjects to experience and conducted a questionnaire.
29. Evaluation experiment 29 - Evaluation method 4 4.2 3 3.25 3.5 3.75 4 4.25 4.5 4.75 5 Q1 Q3 Questionnaire results Average of Q1 and Q3 Rating out of 5
30. Evaluation experiment 30 - Evaluation method Q4. Are there any future improvements? • I want the ball to be realistic • I want to lift other than my knees • I want the cheering of the audience • I want a number display function • I want a tutorial • I want the sound to be realistic • I think you can enjoy more if you meet some kind of charm • It didn't work • I was worried about the ball • I also want to add table tennis version • I want to raise the FPS a little more
31. Consideration • We think that we could get a certain evaluation of whether users can enjoy projection mapping as well as people who see it. • In the method using the cascade classifier, there are occasional misrecognitions and there is room for improvement. 31
32. Future tasks • Implementation of tutorial screen • Texture mapping to ball • Add variation • Enable use by multiple people 32
33. Conclusion • In order to entertain not only those who view projection mapping but also performers, we proposed a participatory projection mapping that changes according to the movement of the performer. • Until now, performers need to precisely match the coordinates of image objects in projection mapping, and tasks that were difficult for many people can now be easily performed. • In the future, by further reducing the deviation of each other's movements, more realistic stage production can be expected 33