Designing Gestures for Digital Musical Instruments: Gesture Elicitation Study with Deaf and Hard of Hearing People #ASSETS2022 (Oral presentation by Ryo Iijima)

This slide was presented in "Session 4: Composition in Music, Programming and Design" at the 24th International ACM SIGACCESS Conference on Computers and Accessibility（ASSETS 2022）.
https://assets22.sigaccess.org/index.html

【Publication】
Ryo Iijima, Akihisa Shitara, and Yoichi Ochiai. 2022. Designing Gestures for Digital Musical Instruments: Gesture Elicitation Study with Deaf and Hard of Hearing People. In the 24th International ACM SIGACCESS Conference on Computers and Accessibility (ASSETS ’22). October 23–26, 2022, Athens, Greece, 8 pages. https://doi.org/10.1145/3517428.3544828

【Project page】
https://digitalnature.slis.tsukuba.ac.jp/2022/10/gestures-for-digital-musical-instruments/

【Presenter】
Ryo Iijima （飯嶋稜）
M.S. Student (Doctoral Program in Informatics)
Graduate School of Comprehensive Human Sciences
University of Tsukuba (JP)
Digital Nature Group (Yoichi Ochiai)

【Abstract】
When playing musical instruments, deaf and hard-of-hearing (DHH) people typically sense their music from the vibrations transmitted by the instruments or the movements of their bodies while performing. Sensory substitution devices now exist that convert sounds into light and vibrations to support DHH people’s musical activities. However, these devices require specialized hardware, and the marketing profiles assume that standard musical instruments are available. Hence, a significant gap remains between DHH people and their musical performance enjoyment. To address this issue, this study identifies end users’ preferred gestures when using smartphones to emulate the musical experience based on the instrument selected. This gesture elicitation study applies 10 instrument types. Herein, we present the results and a new taxonomy of musical instrument gestures. The findings will support the design of gesture-based instrument interfaces to enable DHH people to more directly enjoy their musical performances.

Transcript

1. ©︎
   R&D Center for Digital Nature / xDiversity
   1
   Designing Gestures for Digital Musical Instruments:
   
   
   Gesture Elicitation Study with Deaf and Hard of Hearing People
   Ryo Iijima, Akihisa Shitara, Yoichi Ochiai / University of Tsukuba
   castanet (AR=0.33) triangle (AR=0.22) guiro (AR=0.40) maracas (AR=0.65)
   recorder (AR=0.13) guitar (AR=0.31) cymbal (AR=0.40)
   pellet drum (AR=0.29)
   jaw harp (AR=0.05)
   tam-tam (AR=0.05)
   castanet (AR=0.33) triangle (AR=0.22) guiro (AR=0.40) maracas (AR=0.65)
   recorder (AR=0.13) guitar (AR=0.31) cymbal (AR=0.40)
   pellet drum (AR=0.29)
   jaw harp (AR=0.05)
   tam-tam (AR=0.05)
   

   View Slide

2. ©︎
   R&D Center for Digital Nature / xDiversity
   2
   Introduction
   

   View Slide

3. ©︎
   R&D Center for Digital Nature / xDiversity
   Introduction / Motivation
   3
   Smartphone apps allow us to play musical instruments
   

   View Slide

4. ©︎
   R&D Center for Digital Nature / xDiversity
   Introduction / Motivation
   4
   Di
   ff
   icult for DHH users to enjoy applications
   

   View Slide

5. ©︎
   R&D Center for Digital Nature / xDiversity
   Introduction / Previous Work
   5
   Muss-Bits
   Ontenna
   SOUND HUG
   https://pixiedusttech.com/technologies/sound-hug/
   https://ahlab.org/project/muss-bits/
   https://ontenna.jp/
   

   View Slide

6. ©︎
   R&D Center for Digital Nature / xDiversity
   Introduction / Our Previous Work
   6
   

   View Slide

7. ©︎
   R&D Center for Digital Nature / xDiversity
   Introduction / Research Question
   7
   How can we design
   
   
   other musical instruments?
   

   View Slide

8. ©︎
   R&D Center for Digital Nature / xDiversity
   8
   Experiment / Result
   

   View Slide

9. ©︎
   R&D Center for Digital Nature / xDiversity
   N=11 (Deaf or hard-of-hearing people)
   
   
   25.4 years old on average (SD=5.39, range=19-38)
   Experiment / Purpose & Participants
   9
   Participants
   Identify gestures for playing musical instruments
   
   
   that is pleasing and easier to use for DHH people.
   Purpose
   

   View Slide

10. ©︎
    R&D Center for Digital Nature / xDiversity
    Experiment / Study App
    10
    Idiophones
    concussion vessel clappers
    Idiophones
    shaken idiophones
    Idiophones
    scraped idiophones
    Membranophones
    Chordophones
    Idiophones
    Idiophones
    Idiophones
    Aerophones
    percussion vessels
    plucked idiophones
    percussion sticks
    Idiophones
    concussion vessel clappers with manufactured rim
    

    View Slide

11. ©︎
    R&D Center for Digital Nature / xDiversity
    Experiment / How to Play the Study App
    11
    tap a button vibrational feedback
    

    View Slide

12. ©︎
    R&D Center for Digital Nature / xDiversity
    Experiment / Procedure
    12
    

    View Slide

13. ©︎
    R&D Center for Digital Nature / xDiversity
    Experiment / Five Point Likert scale questions
    13
    This gesture is a good match for its purpose.
    Goodness of fit
    This gesture is a easy to perform.
    Ease of use
    

    View Slide

14. ©︎
    R&D Center for Digital Nature / xDiversity
    castanet (AR=0.33) triangle (AR=0.22) guiro (AR=0.40) maracas (AR=0.65)
    recorder (AR=0.13) guitar (AR=0.31) cymbal (AR=0.40)
    pellet drum (AR=0.29)
    jaw harp (AR=0.05)
    tam-tam (AR=0.05)
    Result / Gesture Set
    14
    

    View Slide

15. ©︎
    R&D Center for Digital Nature / xDiversity
    Result / Taxonomy
    15
    

    View Slide

16. ©︎
    R&D Center for Digital Nature / xDiversity
    Result / Correlation Analysis
    16
    Goodness of fit & Agreement Rate
    
    
    Ease of use & Agreement Rate
    Positive correlation
    Ease of use & Thinking time
    Negative correlation
    

    View Slide

17. ©︎
    R&D Center for Digital Nature / xDiversity
    17
    Design Implications
    

    View Slide

18. ©︎
    R&D Center for Digital Nature / xDiversity
    Design Implications / 1~3
    18
    The hand not holding a smartphone
    
    
    be able to move as well.
    Diverse hand poses should be assumed.
    Avoid designing gestures based on vibrations.
    1.
    2.
    3.
    

    View Slide

19. ©︎
    R&D Center for Digital Nature / xDiversity
    Design Implications / 4~5
    19
    Do not disrupt the comfortable holding
    
    
    of the smartphone in normal situations.
    Encourage users to install a ring-type accessory
    
    
    or remove a notebook-type case.
    5.
    4.
    

    View Slide

20. ©︎
    R&D Center for Digital Nature / xDiversity
    20
    Designing Gestures for Digital Musical Instruments:
    
    
    Gesture Elicitation Study with Deaf and Hard of Hearing People
    Ryo Iijima, Akihisa Shitara, Yoichi Ochiai / University of Tsukuba
    Q&A
    

    View Slide

21. ©︎
    R&D Center for Digital Nature / xDiversity
    Q: How was the vibration pattern designed?
    21
    Iteratively designed, tested, and improved
    
    
    in a laboratory environment with multiple researchers.
    

    View Slide

22. ©︎
    R&D Center for Digital Nature / xDiversity
    Q: Were they influenced by the sensing & recognition technologies?
    22
    They were instructed to treat their smartphone as a
    
    
    “ Magic Brick ” that could recognize any gesture.
    
    
    They were less influenced by the sensing and
    
    
    recognition technologies in creating the gestures.
    

    View Slide

23. ©︎
    R&D Center for Digital Nature / xDiversity
    Q: Participants did not play to the rhythm, right?
    23
    The goal is to use gestures to play music and
    
    
    not to select which instrument to play.
    
    
    To this end, this study aimed to define the gestures
    
    
    as preliminary to playing to a rhythm.
    

    View Slide

24. ©︎
    R&D Center for Digital Nature / xDiversity
    Q: Any cultural differences?
    24
    Yes, there may be some di
    ff
    erences.
    
    
    One participant mimicked a gesture that meant “sutra”
    
    
    in sign language, associating the vibration of a wooden fish
    
    
    that is used in a Buddhist temple, with that of the cymbal.
    
    
    GESs for playing musical instruments with other use groups
    
    
    may provide deeper insights into inclusion.
    

    View Slide

25. ©︎
    R&D Center for Digital Nature / xDiversity
    Q: Why is the AR of jaw harp so low?
    25
    No participants were familiar with the jaw harp.
    
    
    With other instruments, participants often created gestures
    
    
    that imitated how they imagined they would play the real one.
    
    
    However, this approach did not work with the jaw harp.
    

    View Slide

26. ©︎
    R&D Center for Digital Nature / xDiversity
    Q: Why is the AR of tam-tam so low?
    26
    The real tam-tam does not require the performer
    
    
    to hold the sonorous object,
    
    
    and it can be played with one hand.
    
    
    All other idiophones are two-handed instruments,
    
    
    so the roles of the hand holding the device and
    
    
    the other hand are easily fixed.
    

    View Slide

27. ©︎
    R&D Center for Digital Nature / xDiversity
    27
    Designing Gestures for Digital Musical Instruments:
    
    
    Gesture Elicitation Study with Deaf and Hard of Hearing People
    Ryo Iijima, Akihisa Shitara, Yoichi Ochiai / University of Tsukuba
    Figures
    

    View Slide

28. ©︎
    R&D Center for Digital Nature / xDiversity
    Correlation Analysis
    28
    

    View Slide

29. ©︎
    R&D Center for Digital Nature / xDiversity
    Randomizing process
    29
    

    View Slide

30. ©︎
    R&D Center for Digital Nature / xDiversity
    Agreement Rate
    30
    

    View Slide

31. ©︎
    R&D Center for Digital Nature / xDiversity
    Gesture Grouping Criteria
    31
    

    View Slide

32. ©︎
    R&D Center for Digital Nature / xDiversity
    Other instruments to be explored
    32
    

    View Slide

33. ©︎
    R&D Center for Digital Nature / xDiversity
    10 instruments
    33
    

    View Slide

34. ©︎
    R&D Center for Digital Nature / xDiversity
    Classification
    34
    

    View Slide