You as a Puppet - UIST 2017

This slide was presented at UIST 2017 Physical interfaces session.
https://uist.acm.org/uist2017/

You as a Puppet: Evaluation of Telepresence User Interface for Puppetry
https://doi.org/10.1145/3126594.3126608

【Project page】
http://digitalnature.slis.tsukuba.ac.jp/2016/01/yadori/

【Project movie】
https://www.youtube.com/watch?v=MRy5pgXY8es
(ACE 2016 Creative Showcase Version)

【Presenter】
Mose Sakashita (坂下 申世)
University of Tsukuba, School of Informatics
College of Media Arts, Science and Technology
Digital Nature Group (Yoichi Ochiai)
http://www.u.tsukuba.ac.jp/~s1411453/index.html

【Abstract】
We propose an immersive telepresence system for puppetry that transmits a human performer's body and facial movements into a puppet with audiovisual feedback to the performer. The cameras carried in place of puppet's eyes stream live video to the HMD worn by the performer, so that performers can see the images from the puppet's eyes with their own eyes and have a visual understanding of the puppet's ambience. In conventional methods to manipulate a puppet (a hand-puppet, a string-puppet, and a rod-puppet), there is a need to practice manipulating puppets, and there is difficulty carrying out interactions with the audience. Moreover, puppeteers must be positioned exactly where the puppet is. The proposed system addresses these issues by enabling a human performer to manipulate the puppet remotely using his or her body and facial movements. We conducted several user studies with both beginners and professional puppeteers. The results show that, unlike the conventional method, the proposed system facilitates the manipulation of puppets especially for beginners. Moreover, this system allows performers to enjoy puppetry and fascinate audiences.

1. 1 å You as a Puppet Evaluation of Telepresence User Interface for Puppetry Mose Sakashita Tatsuya Minagawa Amy Koike Ippei Suzuki Keisuke Kawahara Yoichi Ochiai
2. Introduction TV-shows Attractions 3 “Jim Henson's Legacy: A "Rainbow Connection" with UMD” Merrill College of Journalism Press Releases. (CC BY-NC 2.0) Communication Education Library teaches children the fun of reading | Stripes Guam http://guam.stripes.com/base-info/library-teaches-children-fun-reading
3. Introduction — Previous Puppetry Ways 4 Glove-puppet “German children performing a tale with selfmade puppets” Jürgen Heimann. (CC BY 3.0) “Marionette puppet show for kids in Asbury Park, NJ” Jackie, 2006. (CC BY 2.0) String-puppet
4. 5 Under the Desk… ¸¸¸¸¸¸¸¸¸¸¸
5. 6 Image by www.traveljunction.com Character Costume
6. 7 http://i843.photobucket.com/albums/zz352/loaloauk/Disney%20Planets/Mickey1.jpg Character Costume
7. 8 Performer’s location Physical limitations… Puppet’s size and shape “German children performing a tale with selfmade puppets” Jürgen Heimann. (CC BY 3.0) “Marionette puppet show for kids in Asbury Park, NJ” Jackie, 2006. (CC BY 2.0) Image by www.traveljunction.com
8. Related Work — Input System for Puppet 9 [Slyper et al. 2015] [Oshita et al. 2013][Slyper et al. 2011] Webcam Apple iSight Nicolas Esposito from France (CC BY 2.0) Web Camera Tongue Joystick Hand Gesture
9. Related Work — Computational Puppet 10 [Sekiguchi et al. 2001] [Bonanni et al. 2005] [Lee et al. 2008] PlayPals HuggableRobotPHONE
10. Introduction — Concept of Our System 11 Remote Place Objective Place Feedback Transmission
11. Implementation 13 Audiovisual Feedback Sensing & Transmission Performer to Puppet Puppet to Performer
12. Implementation — Sensing & Transmission 14 Head Pose
13. Implementation — Sensing & Transmission 15 Arms’ Movements
14. Implementation — Sensing & Transmission 16 Photoreflectors
15. Implementation — Sensing & Transmission 17 Mouth State
16. Implementation — Audiovisual Feedback 18 “Bones” of the Puppet Camera Microphone
17. Implementation — “Bones” of the Puppet 19
18. 20 How does the telepresence system affect the performer and the audience?
19. User Study 21 Performer Side Audience Side Enjoyable? Cute? Lifelike? Enjoyable? Fatigue? Useful?
20. User Study — Performer Side 22 Purpose To investigate how the participants feel when using the telepresence system and compare the results between the experts and beginners. Participants N = 14 (age 18-24) 5 professional puppeteers 9 beginners
21. User Study — Performer Side 23 1. We prepare a simple stage with two puppets for puppetry. 2. Each participant manipulates a puppet freely for approximately five minutes using our system, and only beginners take time to manipulate a puppet by hand. 3. We ask them to rate each question on a seven-point scale, and write free descriptions. Procedure
22. User Study — Puppet’s POV / Performer Side 24 Q. How useful was it to operate the puppet from the puppet’s point of view? Professional Beginner M SD 5.60 5.44 0.80 0.68 Not at all Very useful
23. User Study — Puppet’s POV / Performer Side 25 “Usually (by hand), I manipulate a puppet by imagining the puppet’s point of view, but it’s difficult to do that, and the puppet’s point of view sometimes goes in a different direction, contrary to my intention. So I feel it was useful that I could see things from the puppet’s point of view.” “I felt like I could became a character. It was similar to the feeling when I wear a costume.” — Professional — Beginner
24. Q. How useful was it to operate by moving your own body? Professional Beginner M SD 4.60 6.44 1.20 0.68 User Study — Moving body / Performer Side 26 Significant difference Not at all Very useful
25. User Study — Moving body / Performer Side 27 “I could experience it as if I had become the puppet itself… I think it has more potential if the performers practice, since it is easier to use than by hand.” “When I was asked to manipulate a puppet in a conventional manner, I attempted to move its hands using my thumb and little finger. However, I did not have a technique for moving its hands up and down.” — Professional — Beginner
26. Q. Overall, how useful was this system? Professional Beginner M SD 4.80 6.33 0.98 0.47 Significant difference User Study — Overall Usefullness/Performer Side 28 Not at all Very useful
27. “With the system, I could express my feelings in the puppet. When I manipulate a puppet directly by hand, it feels like I ‘manipulate’ a puppet, but with the system, I felt like I became a puppet.” — Beginner User Study — Overall Usefullness/Performer Side 29
28. Q. Overall, how enjoyable was it to operate a puppet with this system? Professional Beginner M SD 6.40 6.44 0.49 0.50 User Study — Overall Enjoyment / Performer Side 30 Not at all Very enjoyable
29. “I enjoyed that the mouth of the puppet moved in accordance with my voice.” “I enjoyed seeing the audience, because people come closer to the robot puppet, and they don’t normally do that. This made me laugh when our puppets communicated with each other.” — Professional — Beginner User Study — Overall Enjoyment / Performer Side 31
30. User Study — Audience Side 32 Purpose To investigate how the participants feel about the puppetry played by the telepresence system and compare the results between the hand-based and telepresence-based puppetry. Participants N = 22 (age 18-24)
31. 1. We record videos of the hand-based and telepresence-based puppetry acted by professional puppeteers. 2. Participants watch the two videos. 3. We ask them to rate each question about these puppet shows on a seven-point scale, and write free descriptions. Procedure User Study — Audience Side 33
32. 34 User Study — Audience Side Q. How enjoyable did you feel the puppetry was? Q. How cute did you feel the manipulated puppets were? Q. How lifelike did you feel the manipulated puppets were? Didn’t feel at all Felt very much Ours Hand Ours Hand Ours Hand 4.27 4.55 4.59 5.32 4.00 4.77 1.62 1.67 1.70 1.39 1.60 1.51 M SD
33. Limitations 35
34. Limitations — PositionofaPuppetonStageCannotbeChanged 36 “This robot cannot move its position, because it doesn’t have legs. I will get troubled when I use some props for detailed performance.” — Professional Possible Solution Gravity detection system like a NintendoWii Fit Other orientations detected by Kinect
35. Limitations — PerformerCannotConfirmHowaPuppetMoves 37 “…I couldn’t understand how much I could move a puppet when I moved my body. …” — Professional Possible Solution Display of the puppet’s state in the HMD Display of images from a fixed-point camera
36. Limitations — MovementsLookSomewhatMechanicalorArtificial 38 “I felt that the conventional puppets were more lifelike than the system-controlled ones…” “The hand-manipulated puppets’ movements were subtle and seemed more lifelike.” Possible Solution Extra servo motors / Background musics
37. Limitations — Stress & Fatigue on a Performer 39 “When I moved my body, the video that I saw was shaken a lot and I felt sick.” — Beginner Possible Solution Omnidirectional cameras
38. Limitations — Cost & Space 40 Possible Solution Pose estimation from RGB images / Google Cardboard
39. Future Work 41 Interactive Attractions Telecommunication Computational Design
40. Introduction — Summary of Contribution 42 1. We implemented an immersive telepresence system for puppetry. 2. We reported the effectiveness of the system for both the performer and audience side. 3. We discussed the limitations of the system and future work beyond the puppetry.
41. Research Group 43 Mose Sakashita Tatsuya Minagawa Amy Koike Ippei Suzuki Keisuke Kawahara Yoichi Ochiai Supported by
42. 44 å You as a Puppet Evaluation of Telepresence User Interface for Puppetry Mose Sakashita Tatsuya Minagawa Amy Koike Ippei Suzuki Keisuke Kawahara Yoichi Ochiai Demo Reception Please visit our booth in Today, 6pm - 9pm @Villeray, De Tourny

Transcript

1. 1 å You as a Puppet Evaluation of Telepresence User Interface for Puppetry Mose Sakashita Tatsuya Minagawa Amy Koike Ippei Suzuki Keisuke Kawahara Yoichi Ochiai
3. Introduction TV-shows Attractions 3 “Jim Henson's Legacy: A "Rainbow Connection" with UMD” Merrill College of Journalism Press Releases. (CC BY-NC 2.0) Communication Education Library teaches children the fun of reading | Stripes Guam http://guam.stripes.com/base-info/library-teaches-children-fun-reading
4. Introduction — Previous Puppetry Ways 4 Glove-puppet “German children performing a tale with selfmade puppets” Jürgen Heimann. (CC BY 3.0) “Marionette puppet show for kids in Asbury Park, NJ” Jackie, 2006. (CC BY 2.0) String-puppet
5. 5 Under the Desk… ¸¸¸¸¸¸¸¸¸¸¸
6. 6 Image by www.traveljunction.com Character Costume
7. 7 http://i843.photobucket.com/albums/zz352/loaloauk/Disney%20Planets/Mickey1.jpg Character Costume
8. 8 Performer’s location Physical limitations… Puppet’s size and shape “German children performing a tale with selfmade puppets” Jürgen Heimann. (CC BY 3.0) “Marionette puppet show for kids in Asbury Park, NJ” Jackie, 2006. (CC BY 2.0) Image by www.traveljunction.com
9. Related Work — Input System for Puppet 9 [Slyper et al. 2015] [Oshita et al. 2013] [Slyper et al. 2011] Webcam Apple iSight Nicolas Esposito from France (CC BY 2.0) Web Camera Tongue Joystick Hand Gesture
10. Related Work — Computational Puppet 10 [Sekiguchi et al. 2001] [Bonanni et al. 2005] [Lee et al. 2008] PlayPals Huggable RobotPHONE
11. Introduction — Concept of Our System 11 Remote Place Objective Place Feedback Transmission
13. Implementation 13 Audiovisual Feedback Sensing & Transmission Performer to Puppet Puppet to Performer
14. Implementation — Sensing & Transmission 14 Head Pose
15. Implementation — Sensing & Transmission 15 Arms’ Movements
16. Implementation — Sensing & Transmission 16 Photoreflectors
17. Implementation — Sensing & Transmission 17 Mouth State
18. Implementation — Audiovisual Feedback 18 “Bones” of the Puppet Camera Microphone
19. Implementation — “Bones” of the Puppet 19
20. 20 How does the telepresence system affect the performer and the audience?
21. User Study 21 Performer Side Audience Side Enjoyable? Cute? Lifelike? Enjoyable? Fatigue? Useful?
22. User Study — Performer Side 22 Purpose To investigate how the participants feel when using the telepresence system and compare the results between the experts and beginners. Participants N = 14 (age 18-24) 5 professional puppeteers 9 beginners
23. User Study — Performer Side 23 1. We prepare a simple stage with two puppets for puppetry. 2. Each participant manipulates a puppet freely for approximately five minutes using our system, and only beginners take time to manipulate a puppet by hand. 3. We ask them to rate each question on a seven-point scale, and write free descriptions. Procedure
24. User Study — Puppet’s POV / Performer Side 24 Q. How useful was it to operate the puppet from the puppet’s point of view? Professional Beginner M SD 5.60 5.44 0.80 0.68 Not at all Very useful
25. User Study — Puppet’s POV / Performer Side 25 “Usually (by hand), I manipulate a puppet by imagining the puppet’s point of view, but it’s difficult to do that, and the puppet’s point of view sometimes goes in a different direction, contrary to my intention. So I feel it was useful that I could see things from the puppet’s point of view.” “I felt like I could became a character. It was similar to the feeling when I wear a costume.” — Professional — Beginner
26. Q. How useful was it to operate by moving your own body? Professional Beginner M SD 4.60 6.44 1.20 0.68 User Study — Moving body / Performer Side 26 Significant difference Not at all Very useful
27. User Study — Moving body / Performer Side 27 “I could experience it as if I had become the puppet itself… I think it has more potential if the performers practice, since it is easier to use than by hand.” “When I was asked to manipulate a puppet in a conventional manner, I attempted to move its hands using my thumb and little finger. However, I did not have a technique for moving its hands up and down.” — Professional — Beginner
28. Q. Overall, how useful was this system? Professional Beginner M SD 4.80 6.33 0.98 0.47 Significant difference User Study — Overall Usefullness/Performer Side 28 Not at all Very useful
29. “With the system, I could express my feelings in the puppet. When I manipulate a puppet directly by hand, it feels like I ‘manipulate’ a puppet, but with the system, I felt like I became a puppet.” — Beginner User Study — Overall Usefullness/Performer Side 29
30. Q. Overall, how enjoyable was it to operate a puppet with this system? Professional Beginner M SD 6.40 6.44 0.49 0.50 User Study — Overall Enjoyment / Performer Side 30 Not at all Very enjoyable
31. “I enjoyed that the mouth of the puppet moved in accordance with my voice.” “I enjoyed seeing the audience, because people come closer to the robot puppet, and they don’t normally do that. This made me laugh when our puppets communicated with each other.” — Professional — Beginner User Study — Overall Enjoyment / Performer Side 31
32. User Study — Audience Side 32 Purpose To investigate how the participants feel about the puppetry played by the telepresence system and compare the results between the hand-based and telepresence-based puppetry. Participants N = 22 (age 18-24)
33. 1. We record videos of the hand-based and telepresence-based puppetry acted by professional puppeteers. 2. Participants watch the two videos. 3. We ask them to rate each question about these puppet shows on a seven-point scale, and write free descriptions. Procedure User Study — Audience Side 33
34. 34 User Study — Audience Side Q. How enjoyable did you feel the puppetry was? Q. How cute did you feel the manipulated puppets were? Q. How lifelike did you feel the manipulated puppets were? Didn’t feel at all Felt very much Ours Hand Ours Hand Ours Hand 4.27 4.55 4.59 5.32 4.00 4.77 1.62 1.67 1.70 1.39 1.60 1.51 M SD
35. Limitations 35
36. Limitations — Position of a Puppet on Stage Cannot be Changed 36 “This robot cannot move its position, because it doesn’t have legs. I will get troubled when I use some props for detailed performance.” — Professional Possible Solution Gravity detection system like a NintendoWii Fit Other orientations detected by Kinect
37. Limitations — Performer Cannot Confirm How a Puppet Moves 37 “…I couldn’t understand how much I could move a puppet when I moved my body. …” — Professional Possible Solution Display of the puppet’s state in the HMD Display of images from a fixed-point camera
38. Limitations — Movements Look Somewhat Mechanical or Artificial 38 “I felt that the conventional puppets were more lifelike than the system-controlled ones…” “The hand-manipulated puppets’ movements were subtle and seemed more lifelike.” Possible Solution Extra servo motors / Background musics
39. Limitations — Stress & Fatigue on a Performer 39 “When I moved my body, the video that I saw was shaken a lot and I felt sick.” — Beginner Possible Solution Omnidirectional cameras
40. Limitations — Cost & Space 40 Possible Solution Pose estimation from RGB images / Google Cardboard
41. Future Work 41 Interactive Attractions Telecommunication Computational Design
42. Introduction — Summary of Contribution 42 1. We implemented an immersive telepresence system for puppetry. 2. We reported the effectiveness of the system for both the performer and audience side. 3. We discussed the limitations of the system and future work beyond the puppetry.
43. Research Group 43 Mose Sakashita Tatsuya Minagawa Amy Koike Ippei Suzuki Keisuke Kawahara Yoichi Ochiai Supported by
44. 44 å You as a Puppet Evaluation of Telepresence User Interface for Puppetry Mose Sakashita Tatsuya Minagawa Amy Koike Ippei Suzuki Keisuke Kawahara Yoichi Ochiai Demo Reception Please visit our booth in Today, 6pm - 9pm @Villeray, De Tourny