Designing Gestures for Digital Musical Instruments: Gesture Elicitation Study with Deaf and Hard of Hearing People #ASSETS2022 (Oral presentation by Ryo Iijima)

by Digital Nature Group

Slide 1

Slide 1 text

©︎ R&D Center for Digital Nature / xDiversity 1 Designing Gestures for Digital Musical Instruments: Gesture Elicitation Study with Deaf and Hard of Hearing People Ryo Iijima, Akihisa Shitara, Yoichi Ochiai / University of Tsukuba castanet (AR=0.33) triangle (AR=0.22) guiro (AR=0.40) maracas (AR=0.65) recorder (AR=0.13) guitar (AR=0.31) cymbal (AR=0.40) pellet drum (AR=0.29) jaw harp (AR=0.05) tam-tam (AR=0.05) castanet (AR=0.33) triangle (AR=0.22) guiro (AR=0.40) maracas (AR=0.65) recorder (AR=0.13) guitar (AR=0.31) cymbal (AR=0.40) pellet drum (AR=0.29) jaw harp (AR=0.05) tam-tam (AR=0.05)

Slide 2

Slide 2 text

©︎ R&D Center for Digital Nature / xDiversity 2 Introduction

Slide 3

Slide 3 text

©︎ R&D Center for Digital Nature / xDiversity Introduction / Motivation 3 Smartphone apps allow us to play musical instruments

Slide 4

Slide 4 text

©︎ R&D Center for Digital Nature / xDiversity Introduction / Motivation 4 Di ff icult for DHH users to enjoy applications

Slide 5

Slide 5 text

©︎ R&D Center for Digital Nature / xDiversity Introduction / Previous Work 5 Muss-Bits Ontenna SOUND HUG https://pixiedusttech.com/technologies/sound-hug/ https://ahlab.org/project/muss-bits/ https://ontenna.jp/

Slide 6

Slide 6 text

©︎ R&D Center for Digital Nature / xDiversity Introduction / Our Previous Work 6

Slide 7

Slide 7 text

©︎ R&D Center for Digital Nature / xDiversity Introduction / Research Question 7 How can we design other musical instruments?

Slide 8

Slide 8 text

©︎ R&D Center for Digital Nature / xDiversity 8 Experiment / Result

Slide 9

Slide 9 text

©︎ R&D Center for Digital Nature / xDiversity N=11 (Deaf or hard-of-hearing people) 25.4 years old on average (SD=5.39, range=19-38) Experiment / Purpose & Participants 9 Participants Identify gestures for playing musical instruments that is pleasing and easier to use for DHH people. Purpose

Slide 10

Slide 10 text

©︎ R&D Center for Digital Nature / xDiversity Experiment / Study App 10 Idiophones concussion vessel clappers Idiophones shaken idiophones Idiophones scraped idiophones Membranophones Chordophones Idiophones Idiophones Idiophones Aerophones percussion vessels plucked idiophones percussion sticks Idiophones concussion vessel clappers with manufactured rim

Slide 11

Slide 11 text

©︎ R&D Center for Digital Nature / xDiversity Experiment / How to Play the Study App 11 tap a button vibrational feedback

Slide 12

Slide 12 text

©︎ R&D Center for Digital Nature / xDiversity Experiment / Procedure 12

Slide 13

Slide 13 text

©︎ R&D Center for Digital Nature / xDiversity Experiment / Five Point Likert scale questions 13 This gesture is a good match for its purpose. Goodness of fit This gesture is a easy to perform. Ease of use

Slide 14

Slide 14 text

©︎ R&D Center for Digital Nature / xDiversity castanet (AR=0.33) triangle (AR=0.22) guiro (AR=0.40) maracas (AR=0.65) recorder (AR=0.13) guitar (AR=0.31) cymbal (AR=0.40) pellet drum (AR=0.29) jaw harp (AR=0.05) tam-tam (AR=0.05) Result / Gesture Set 14

Slide 15

Slide 15 text

©︎ R&D Center for Digital Nature / xDiversity Result / Taxonomy 15

Slide 16

Slide 16 text

©︎ R&D Center for Digital Nature / xDiversity Result / Correlation Analysis 16 Goodness of fit & Agreement Rate Ease of use & Agreement Rate Positive correlation Ease of use & Thinking time Negative correlation

Slide 17

Slide 17 text

©︎ R&D Center for Digital Nature / xDiversity 17 Design Implications

Slide 18

Slide 18 text

©︎ R&D Center for Digital Nature / xDiversity Design Implications / 1~3 18 The hand not holding a smartphone be able to move as well. Diverse hand poses should be assumed. Avoid designing gestures based on vibrations. 1. 2. 3.

Slide 19

Slide 19 text

©︎ R&D Center for Digital Nature / xDiversity Design Implications / 4~5 19 Do not disrupt the comfortable holding of the smartphone in normal situations. Encourage users to install a ring-type accessory or remove a notebook-type case. 5. 4.

Slide 20

Slide 20 text

©︎ R&D Center for Digital Nature / xDiversity 20 Designing Gestures for Digital Musical Instruments: Gesture Elicitation Study with Deaf and Hard of Hearing People Ryo Iijima, Akihisa Shitara, Yoichi Ochiai / University of Tsukuba Q&A

Slide 21

Slide 21 text

©︎ R&D Center for Digital Nature / xDiversity Q: How was the vibration pattern designed? 21 Iteratively designed, tested, and improved in a laboratory environment with multiple researchers.

Slide 22

Slide 22 text

©︎ R&D Center for Digital Nature / xDiversity Q: Were they influenced by the sensing & recognition technologies? 22 They were instructed to treat their smartphone as a “ Magic Brick ” that could recognize any gesture. They were less influenced by the sensing and recognition technologies in creating the gestures.

Slide 23

Slide 23 text

©︎ R&D Center for Digital Nature / xDiversity Q: Participants did not play to the rhythm, right? 23 The goal is to use gestures to play music and not to select which instrument to play. To this end, this study aimed to define the gestures as preliminary to playing to a rhythm.

Slide 24

Slide 24 text

©︎ R&D Center for Digital Nature / xDiversity Q: Any cultural differences? 24 Yes, there may be some di ff erences. One participant mimicked a gesture that meant “sutra” in sign language, associating the vibration of a wooden fish that is used in a Buddhist temple, with that of the cymbal. GESs for playing musical instruments with other use groups may provide deeper insights into inclusion.

Slide 25

Slide 25 text

©︎ R&D Center for Digital Nature / xDiversity Q: Why is the AR of jaw harp so low? 25 No participants were familiar with the jaw harp. With other instruments, participants often created gestures that imitated how they imagined they would play the real one. However, this approach did not work with the jaw harp.

Slide 26

Slide 26 text

©︎ R&D Center for Digital Nature / xDiversity Q: Why is the AR of tam-tam so low? 26 The real tam-tam does not require the performer to hold the sonorous object, and it can be played with one hand. All other idiophones are two-handed instruments, so the roles of the hand holding the device and the other hand are easily fixed.

Slide 27

Slide 27 text

©︎ R&D Center for Digital Nature / xDiversity 27 Designing Gestures for Digital Musical Instruments: Gesture Elicitation Study with Deaf and Hard of Hearing People Ryo Iijima, Akihisa Shitara, Yoichi Ochiai / University of Tsukuba Figures