$30 off During Our Annual Pro Sale. View Details »

You as a Puppet - UIST 2017

You as a Puppet - UIST 2017

This slide was presented at UIST 2017 Physical interfaces session.
https://uist.acm.org/uist2017/

You as a Puppet: Evaluation of Telepresence User Interface for Puppetry
https://doi.org/10.1145/3126594.3126608

【Project page】
http://digitalnature.slis.tsukuba.ac.jp/2016/01/yadori/

【Project movie】
https://www.youtube.com/watch?v=MRy5pgXY8es
(ACE 2016 Creative Showcase Version)

【Presenter】
Mose Sakashita (坂下 申世)
University of Tsukuba, School of Informatics
College of Media Arts, Science and Technology
Digital Nature Group (Yoichi Ochiai)
http://www.u.tsukuba.ac.jp/~s1411453/index.html

【Abstract】
We propose an immersive telepresence system for puppetry that transmits a human performer's body and facial movements into a puppet with audiovisual feedback to the performer. The cameras carried in place of puppet's eyes stream live video to the HMD worn by the performer, so that performers can see the images from the puppet's eyes with their own eyes and have a visual understanding of the puppet's ambience. In conventional methods to manipulate a puppet (a hand-puppet, a string-puppet, and a rod-puppet), there is a need to practice manipulating puppets, and there is difficulty carrying out interactions with the audience. Moreover, puppeteers must be positioned exactly where the puppet is. The proposed system addresses these issues by enabling a human performer to manipulate the puppet remotely using his or her body and facial movements. We conducted several user studies with both beginners and professional puppeteers. The results show that, unlike the conventional method, the proposed system facilitates the manipulation of puppets especially for beginners. Moreover, this system allows performers to enjoy puppetry and fascinate audiences.

1. 1 å You as a Puppet Evaluation of Telepresence User Interface for Puppetry Mose Sakashita Tatsuya Minagawa Amy Koike Ippei Suzuki Keisuke Kawahara Yoichi Ochiai
2. Introduction TV-shows Attractions 3 “Jim Henson's Legacy: A "Rainbow Connection" with UMD” Merrill College of Journalism Press Releases. (CC BY-NC 2.0) Communication Education Library teaches children the fun of reading | Stripes Guam http://guam.stripes.com/base-info/library-teaches-children-fun-reading
3. Introduction — Previous Puppetry Ways 4 Glove-puppet “German children performing a tale with selfmade puppets” Jürgen Heimann. (CC BY 3.0) “Marionette puppet show for kids in Asbury Park, NJ” Jackie, 2006. (CC BY 2.0) String-puppet
4. 5 Under the Desk… ¸¸¸¸¸¸¸¸¸¸¸
5. 6 Image by www.traveljunction.com Character Costume
6. 7 http://i843.photobucket.com/albums/zz352/loaloauk/Disney%20Planets/Mickey1.jpg Character Costume
7. 8 Performer’s location Physical limitations… Puppet’s size and shape “German children performing a tale with selfmade puppets” Jürgen Heimann. (CC BY 3.0) “Marionette puppet show for kids in Asbury Park, NJ” Jackie, 2006. (CC BY 2.0) Image by www.traveljunction.com
8. Related Work — Input System for Puppet 9 [Slyper et al. 2015] [Oshita et al. 2013][Slyper et al. 2011] Webcam Apple iSight Nicolas Esposito from France (CC BY 2.0) Web Camera Tongue Joystick Hand Gesture
9. Related Work — Computational Puppet 10 [Sekiguchi et al. 2001] [Bonanni et al. 2005] [Lee et al. 2008] PlayPals HuggableRobotPHONE
10. Introduction — Concept of Our System 11 Remote Place Objective Place Feedback Transmission
11. Implementation 13 Audiovisual Feedback Sensing & Transmission Performer to Puppet Puppet to Performer
12. Implementation — Sensing & Transmission 14 Head Pose
13. Implementation — Sensing & Transmission 15 Arms’ Movements
14. Implementation — Sensing & Transmission 16 Photoreflectors
15. Implementation — Sensing & Transmission 17 Mouth State
16. Implementation — Audiovisual Feedback 18 “Bones” of the Puppet Camera Microphone
17. Implementation — “Bones” of the Puppet 19
18. 20 How does the telepresence system affect the performer and the audience?
19. User Study 21 Performer Side Audience Side Enjoyable? Cute? Lifelike? Enjoyable? Fatigue? Useful?
20. User Study — Performer Side 22 Purpose To investigate how the participants feel when using the telepresence system and compare the results between the experts and beginners. Participants N = 14 (age 18-24) 5 professional puppeteers 9 beginners
21. User Study — Performer Side 23 1. We prepare a simple stage with two puppets for puppetry. 2. Each participant manipulates a puppet freely for approximately five minutes using our system, and only beginners take time to manipulate a puppet by hand. 3. We ask them to rate each question on a seven-point scale, and write free descriptions. Procedure
22. User Study — Puppet’s POV / Performer Side 24 Q. How useful was it to operate the puppet from the puppet’s point of view? Professional Beginner M SD 5.60 5.44 0.80 0.68 Not at all Very useful
23. User Study — Puppet’s POV / Performer Side 25 “Usually (by hand), I manipulate a puppet by imagining the puppet’s point of view, but it’s difficult to do that, and the puppet’s point of view sometimes goes in a different direction, contrary to my intention. So I feel it was useful that I could see things from the puppet’s point of view.” “I felt like I could became a character. It was similar to the feeling when I wear a costume.” — Professional — Beginner
24. Q. How useful was it to operate by moving your own body? Professional Beginner M SD 4.60 6.44 1.20 0.68 User Study — Moving body / Performer Side 26 Significant difference Not at all Very useful
25. User Study — Moving body / Performer Side 27 “I could experience it as if I had become the puppet itself… I think it has more potential if the performers practice, since it is easier to use than by hand.” “When I was asked to manipulate a puppet in a conventional manner, I attempted to move its hands using my thumb and little finger. However, I did not have a technique for moving its hands up and down.” — Professional — Beginner
26. Q. Overall, how useful was this system? Professional Beginner M SD 4.80 6.33 0.98 0.47 Significant difference User Study — Overall Usefullness/Performer Side 28 Not at all Very useful
27. “With the system, I could express my feelings in the puppet. When I manipulate a puppet directly by hand, it feels like I ‘manipulate’ a puppet, but with the system, I felt like I became a puppet.” — Beginner User Study — Overall Usefullness/Performer Side 29
28. Q. Overall, how enjoyable was it to operate a puppet with this system? Professional Beginner M SD 6.40 6.44 0.49 0.50 User Study — Overall Enjoyment / Performer Side 30 Not at all Very enjoyable
29. “I enjoyed that the mouth of the puppet moved in accordance with my voice.” “I enjoyed seeing the audience, because people come closer to the robot puppet, and they don’t normally do that. This made me laugh when our puppets communicated with each other.” — Professional — Beginner User Study — Overall Enjoyment / Performer Side 31
30. User Study — Audience Side 32 Purpose To investigate how the participants feel about the puppetry played by the telepresence system and compare the results between the hand-based and telepresence-based puppetry. Participants N = 22 (age 18-24)
31. 1. We record videos of the hand-based and telepresence-based puppetry acted by professional puppeteers. 2. Participants watch the two videos. 3. We ask them to rate each question about these puppet shows on a seven-point scale, and write free descriptions. Procedure User Study — Audience Side 33
32. 34 User Study — Audience Side Q. How enjoyable did you feel the puppetry was? Q. How cute did you feel the manipulated puppets were? Q. How lifelike did you feel the manipulated puppets were? Didn’t feel at all Felt very much Ours Hand Ours Hand Ours Hand 4.27 4.55 4.59 5.32 4.00 4.77 1.62 1.67 1.70 1.39 1.60 1.51 M SD
33. Limitations 35
34. Limitations — PositionofaPuppetonStageCannotbeChanged 36 “This robot cannot move its position, because it doesn’t have legs. I will get troubled when I use some props for detailed performance.” — Professional Possible Solution Gravity detection system like a NintendoWii Fit Other orientations detected by Kinect
35. Limitations — PerformerCannotConfirmHowaPuppetMoves 37 “…I couldn’t understand how much I could move a puppet when I moved my body. …” — Professional Possible Solution Display of the puppet’s state in the HMD Display of images from a fixed-point camera
36. Limitations — MovementsLookSomewhatMechanicalorArtificial 38 “I felt that the conventional puppets were more lifelike than the system-controlled ones…” “The hand-manipulated puppets’ movements were subtle and seemed more lifelike.” Possible Solution Extra servo motors / Background musics
37. Limitations — Stress & Fatigue on a Performer 39 “When I moved my body, the video that I saw was shaken a lot and I felt sick.” — Beginner Possible Solution Omnidirectional cameras
38. Limitations — Cost & Space 40 Possible Solution Pose estimation from RGB images / Google Cardboard
39. Future Work 41 Interactive Attractions Telecommunication Computational Design
40. Introduction — Summary of Contribution 42 1. We implemented an immersive telepresence system for puppetry. 2. We reported the effectiveness of the system for both the performer and audience side. 3. We discussed the limitations of the system and future work beyond the puppetry.
41. Research Group 43 Mose Sakashita Tatsuya Minagawa Amy Koike Ippei Suzuki Keisuke Kawahara Yoichi Ochiai Supported by
42. 44 å You as a Puppet Evaluation of Telepresence User Interface for Puppetry Mose Sakashita Tatsuya Minagawa Amy Koike Ippei Suzuki Keisuke Kawahara Yoichi Ochiai Demo Reception Please visit our booth in Today, 6pm - 9pm @Villeray, De Tourny

Digital Nature Group

October 23, 2017
Tweet

More Decks by Digital Nature Group

Other Decks in Research

Transcript

  1. 1
    å
    You as a Puppet
    Evaluation of Telepresence User Interface for Puppetry
    Mose Sakashita Tatsuya Minagawa Amy Koike Ippei Suzuki Keisuke Kawahara Yoichi Ochiai

    View Slide

  2. View Slide

  3. Introduction
    TV-shows Attractions
    3
    “Jim Henson's Legacy: A "Rainbow Connection" with UMD”
    Merrill College of Journalism Press Releases. (CC BY-NC 2.0)
    Communication Education
    Library teaches children the fun of reading | Stripes Guam
    http://guam.stripes.com/base-info/library-teaches-children-fun-reading

    View Slide

  4. Introduction — Previous Puppetry Ways
    4
    Glove-puppet
    “German children performing a tale with selfmade puppets” Jürgen Heimann.
    (CC BY 3.0)
    “Marionette puppet show for kids in Asbury Park, NJ” Jackie, 2006.
    (CC BY 2.0)
    String-puppet

    View Slide

  5. 5
    Under the Desk…
    ¸¸¸¸¸¸¸¸¸¸¸

    View Slide

  6. 6
    Image by www.traveljunction.com
    Character Costume

    View Slide

  7. 7
    http://i843.photobucket.com/albums/zz352/loaloauk/Disney%20Planets/Mickey1.jpg
    Character Costume

    View Slide

  8. 8
    Performer’s location
    Physical limitations…
    Puppet’s size and shape
    “German children performing a tale with selfmade puppets” Jürgen Heimann.
    (CC BY 3.0)
    “Marionette puppet show for kids in Asbury Park, NJ” Jackie, 2006.
    (CC BY 2.0)
    Image by www.traveljunction.com

    View Slide

  9. Related Work — Input System for Puppet
    9
    [Slyper et al. 2015] [Oshita et al. 2013]
    [Slyper et al. 2011]
    Webcam Apple iSight
    Nicolas Esposito from France (CC BY 2.0)
    Web Camera Tongue Joystick Hand Gesture

    View Slide

  10. Related Work — Computational Puppet
    10
    [Sekiguchi et al. 2001] [Bonanni et al. 2005] [Lee et al. 2008]
    PlayPals Huggable
    RobotPHONE

    View Slide

  11. Introduction — Concept of Our System
    11
    Remote Place
    Objective Place
    Feedback Transmission

    View Slide

  12. View Slide

  13. Implementation
    13
    Audiovisual Feedback
    Sensing & Transmission
    Performer to Puppet
    Puppet to Performer

    View Slide

  14. Implementation — Sensing & Transmission
    14
    Head Pose

    View Slide

  15. Implementation — Sensing & Transmission
    15
    Arms’ Movements

    View Slide

  16. Implementation — Sensing & Transmission
    16
    Photoreflectors

    View Slide

  17. Implementation — Sensing & Transmission
    17
    Mouth State

    View Slide

  18. Implementation — Audiovisual Feedback
    18
    “Bones” of the Puppet
    Camera
    Microphone

    View Slide

  19. Implementation — “Bones” of the Puppet
    19

    View Slide

  20. 20
    How does
    the telepresence system
    affect the performer and
    the audience?

    View Slide

  21. User Study
    21
    Performer Side Audience Side
    Enjoyable?
    Cute?
    Lifelike?
    Enjoyable?
    Fatigue?
    Useful?

    View Slide

  22. User Study — Performer Side
    22
    Purpose
    To investigate how the participants feel when
    using the telepresence system and compare
    the results between the experts and beginners.
    Participants
    N = 14 (age 18-24)
    5 professional puppeteers
    9 beginners

    View Slide

  23. User Study — Performer Side
    23
    1. We prepare a simple stage with two puppets for puppetry.
    2. Each participant manipulates a puppet freely for
    approximately five minutes using our system, and only
    beginners take time to manipulate a puppet by hand.
    3. We ask them to rate each question on a seven-point scale,
    and write free descriptions.
    Procedure

    View Slide

  24. User Study — Puppet’s POV / Performer Side
    24
    Q. How useful was it to operate the puppet
    from the puppet’s point of view?
    Professional
    Beginner
    M SD
    5.60
    5.44
    0.80
    0.68
    Not at all Very useful

    View Slide

  25. User Study — Puppet’s POV / Performer Side
    25
    “Usually (by hand), I manipulate a puppet by imagining
    the puppet’s point of view, but it’s difficult to do that, and
    the puppet’s point of view sometimes goes in a different
    direction, contrary to my intention. So I feel it was useful
    that I could see things from the puppet’s point of view.”
    “I felt like I could became a character. It was similar to
    the feeling when I wear a costume.”
    — Professional
    — Beginner

    View Slide

  26. Q. How useful was it to
    operate by moving your own body?
    Professional
    Beginner
    M SD
    4.60
    6.44
    1.20
    0.68
    User Study — Moving body / Performer Side
    26
    Significant difference
    Not at all Very useful

    View Slide

  27. User Study — Moving body / Performer Side
    27
    “I could experience it as if I had become the puppet
    itself… I think it has more potential if the performers
    practice, since it is easier to use than by hand.”
    “When I was asked to manipulate a puppet in a
    conventional manner, I attempted to move its hands
    using my thumb and little finger. However, I did not have
    a technique for moving its hands up and down.”
    — Professional
    — Beginner

    View Slide

  28. Q. Overall, how useful was this system?
    Professional
    Beginner
    M SD
    4.80
    6.33
    0.98
    0.47
    Significant difference
    User Study — Overall Usefullness/Performer Side
    28
    Not at all Very useful

    View Slide

  29. “With the system, I could express my feelings in the
    puppet. When I manipulate a puppet directly by hand, it
    feels like I ‘manipulate’ a puppet, but with the system,
    I felt like I became a puppet.”
    — Beginner
    User Study — Overall Usefullness/Performer Side
    29

    View Slide

  30. Q. Overall, how enjoyable was it to operate
    a puppet with this system?
    Professional
    Beginner
    M SD
    6.40
    6.44
    0.49
    0.50
    User Study — Overall Enjoyment / Performer Side
    30
    Not at all Very enjoyable

    View Slide

  31. “I enjoyed that the mouth of the puppet moved in
    accordance with my voice.”
    “I enjoyed seeing the audience, because people come
    closer to the robot puppet, and they don’t normally do
    that. This made me laugh when our puppets
    communicated with each other.”
    — Professional
    — Beginner
    User Study — Overall Enjoyment / Performer Side
    31

    View Slide

  32. User Study — Audience Side
    32
    Purpose
    To investigate how the participants feel about
    the puppetry played by the telepresence
    system and compare the results between the
    hand-based and telepresence-based puppetry.
    Participants
    N = 22 (age 18-24)

    View Slide

  33. 1. We record videos of the hand-based
    and telepresence-based puppetry
    acted by professional puppeteers.
    2. Participants watch the two videos.
    3. We ask them to rate each question
    about these puppet shows on a
    seven-point scale, and write free
    descriptions.
    Procedure
    User Study — Audience Side
    33

    View Slide

  34. 34
    User Study — Audience Side
    Q. How enjoyable
    did you feel the puppetry was?
    Q. How cute did you feel
    the manipulated puppets were?
    Q. How lifelike did you feel
    the manipulated puppets were?
    Didn’t feel at all Felt very much
    Ours
    Hand
    Ours
    Hand
    Ours
    Hand
    4.27
    4.55
    4.59
    5.32
    4.00
    4.77
    1.62
    1.67
    1.70
    1.39
    1.60
    1.51
    M SD

    View Slide

  35. Limitations
    35

    View Slide

  36. Limitations — Position of a Puppet on Stage Cannot be Changed
    36
    “This robot cannot move its position, because it
    doesn’t have legs. I will get troubled when I use
    some props for detailed performance.”
    — Professional
    Possible Solution
    Gravity detection system like a NintendoWii Fit
    Other orientations detected by Kinect

    View Slide

  37. Limitations — Performer Cannot Confirm How a Puppet Moves
    37
    “…I couldn’t understand how much I could
    move a puppet when I moved my body. …”
    — Professional
    Possible Solution
    Display of the puppet’s state in the HMD
    Display of images from a fixed-point camera

    View Slide

  38. Limitations — Movements Look Somewhat Mechanical or Artificial
    38
    “I felt that the conventional puppets were
    more lifelike than the system-controlled ones…”
    “The hand-manipulated puppets’ movements
    were subtle and seemed more lifelike.”
    Possible Solution
    Extra servo motors / Background musics

    View Slide

  39. Limitations — Stress & Fatigue on a Performer
    39
    “When I moved my body, the video that I saw
    was shaken a lot and I felt sick.”
    — Beginner
    Possible Solution
    Omnidirectional cameras

    View Slide

  40. Limitations — Cost & Space
    40
    Possible Solution
    Pose estimation from RGB images / Google Cardboard

    View Slide

  41. Future Work
    41
    Interactive Attractions Telecommunication Computational Design

    View Slide

  42. Introduction — Summary of Contribution
    42
    1. We implemented an immersive telepresence
    system for puppetry.
    2. We reported the effectiveness of the system
    for both the performer and audience side.
    3. We discussed the limitations of the system and
    future work beyond the puppetry.

    View Slide

  43. Research Group
    43
    Mose Sakashita Tatsuya Minagawa Amy Koike Ippei Suzuki Keisuke Kawahara Yoichi Ochiai
    Supported by

    View Slide

  44. 44
    å
    You as a Puppet
    Evaluation of Telepresence User Interface for Puppetry
    Mose Sakashita Tatsuya Minagawa Amy Koike Ippei Suzuki Keisuke Kawahara Yoichi Ochiai
    Demo Reception
    Please visit our booth in
    Today, 6pm - 9pm
    @Villeray, De Tourny

    View Slide