$30 off During Our Annual Pro Sale. View Details »

Designing Gestures for Digital Musical Instruments: Gesture Elicitation Study with Deaf and Hard of Hearing People #ASSETS2022 (Oral presentation by Ryo Iijima)

Designing Gestures for Digital Musical Instruments: Gesture Elicitation Study with Deaf and Hard of Hearing People #ASSETS2022 (Oral presentation by Ryo Iijima)

This slide was presented in "Session 4: Composition in Music, Programming and Design" at the 24th International ACM SIGACCESS Conference on Computers and Accessibility(ASSETS 2022).
https://assets22.sigaccess.org/index.html

【Publication】
Ryo Iijima, Akihisa Shitara, and Yoichi Ochiai. 2022. Designing Gestures for Digital Musical Instruments: Gesture Elicitation Study with Deaf and Hard of Hearing People. In the 24th International ACM SIGACCESS Conference on Computers and Accessibility (ASSETS ’22). October 23–26, 2022, Athens, Greece, 8 pages. https://doi.org/10.1145/3517428.3544828

【Project page】
https://digitalnature.slis.tsukuba.ac.jp/2022/10/gestures-for-digital-musical-instruments/

【Presenter】
Ryo Iijima (飯嶋稜)
M.S. Student (Doctoral Program in Informatics)
Graduate School of Comprehensive Human Sciences
University of Tsukuba (JP)
Digital Nature Group (Yoichi Ochiai)

【Abstract】
When playing musical instruments, deaf and hard-of-hearing (DHH) people typically sense their music from the vibrations transmitted by the instruments or the movements of their bodies while performing. Sensory substitution devices now exist that convert sounds into light and vibrations to support DHH people’s musical activities. However, these devices require specialized hardware, and the marketing profiles assume that standard musical instruments are available. Hence, a significant gap remains between DHH people and their musical performance enjoyment. To address this issue, this study identifies end users’ preferred gestures when using smartphones to emulate the musical experience based on the instrument selected. This gesture elicitation study applies 10 instrument types. Herein, we present the results and a new taxonomy of musical instrument gestures. The findings will support the design of gesture-based instrument interfaces to enable DHH people to more directly enjoy their musical performances.

Digital Nature Group

October 25, 2022
Tweet

More Decks by Digital Nature Group

Other Decks in Research

Transcript

  1. ©︎
    R&D Center for Digital Nature / xDiversity
    1
    Designing Gestures for Digital Musical Instruments:


    Gesture Elicitation Study with Deaf and Hard of Hearing People
    Ryo Iijima, Akihisa Shitara, Yoichi Ochiai / University of Tsukuba
    castanet (AR=0.33) triangle (AR=0.22) guiro (AR=0.40) maracas (AR=0.65)
    recorder (AR=0.13) guitar (AR=0.31) cymbal (AR=0.40)
    pellet drum (AR=0.29)
    jaw harp (AR=0.05)
    tam-tam (AR=0.05)
    castanet (AR=0.33) triangle (AR=0.22) guiro (AR=0.40) maracas (AR=0.65)
    recorder (AR=0.13) guitar (AR=0.31) cymbal (AR=0.40)
    pellet drum (AR=0.29)
    jaw harp (AR=0.05)
    tam-tam (AR=0.05)

    View Slide

  2. ©︎
    R&D Center for Digital Nature / xDiversity
    2
    Introduction

    View Slide

  3. ©︎
    R&D Center for Digital Nature / xDiversity
    Introduction / Motivation
    3
    Smartphone apps allow us to play musical instruments

    View Slide

  4. ©︎
    R&D Center for Digital Nature / xDiversity
    Introduction / Motivation
    4
    Di
    ff
    icult for DHH users to enjoy applications

    View Slide

  5. ©︎
    R&D Center for Digital Nature / xDiversity
    Introduction / Previous Work
    5
    Muss-Bits
    Ontenna
    SOUND HUG
    https://pixiedusttech.com/technologies/sound-hug/
    https://ahlab.org/project/muss-bits/
    https://ontenna.jp/

    View Slide

  6. ©︎
    R&D Center for Digital Nature / xDiversity
    Introduction / Our Previous Work
    6

    View Slide

  7. ©︎
    R&D Center for Digital Nature / xDiversity
    Introduction / Research Question
    7
    How can we design


    other musical instruments?

    View Slide

  8. ©︎
    R&D Center for Digital Nature / xDiversity
    8
    Experiment / Result

    View Slide

  9. ©︎
    R&D Center for Digital Nature / xDiversity
    N=11 (Deaf or hard-of-hearing people)


    25.4 years old on average (SD=5.39, range=19-38)
    Experiment / Purpose & Participants
    9
    Participants
    Identify gestures for playing musical instruments


    that is pleasing and easier to use for DHH people.
    Purpose

    View Slide

  10. ©︎
    R&D Center for Digital Nature / xDiversity
    Experiment / Study App
    10
    Idiophones
    concussion vessel clappers
    Idiophones
    shaken idiophones
    Idiophones
    scraped idiophones
    Membranophones
    Chordophones
    Idiophones
    Idiophones
    Idiophones
    Aerophones
    percussion vessels
    plucked idiophones
    percussion sticks
    Idiophones
    concussion vessel clappers with manufactured rim

    View Slide

  11. ©︎
    R&D Center for Digital Nature / xDiversity
    Experiment / How to Play the Study App
    11
    tap a button vibrational feedback

    View Slide

  12. ©︎
    R&D Center for Digital Nature / xDiversity
    Experiment / Procedure
    12

    View Slide

  13. ©︎
    R&D Center for Digital Nature / xDiversity
    Experiment / Five Point Likert scale questions
    13
    This gesture is a good match for its purpose.
    Goodness of fit
    This gesture is a easy to perform.
    Ease of use

    View Slide

  14. ©︎
    R&D Center for Digital Nature / xDiversity
    castanet (AR=0.33) triangle (AR=0.22) guiro (AR=0.40) maracas (AR=0.65)
    recorder (AR=0.13) guitar (AR=0.31) cymbal (AR=0.40)
    pellet drum (AR=0.29)
    jaw harp (AR=0.05)
    tam-tam (AR=0.05)
    Result / Gesture Set
    14

    View Slide

  15. ©︎
    R&D Center for Digital Nature / xDiversity
    Result / Taxonomy
    15

    View Slide

  16. ©︎
    R&D Center for Digital Nature / xDiversity
    Result / Correlation Analysis
    16
    Goodness of fit & Agreement Rate


    Ease of use & Agreement Rate
    Positive correlation
    Ease of use & Thinking time
    Negative correlation

    View Slide

  17. ©︎
    R&D Center for Digital Nature / xDiversity
    17
    Design Implications

    View Slide

  18. ©︎
    R&D Center for Digital Nature / xDiversity
    Design Implications / 1~3
    18
    The hand not holding a smartphone


    be able to move as well.
    Diverse hand poses should be assumed.
    Avoid designing gestures based on vibrations.
    1.
    2.
    3.

    View Slide

  19. ©︎
    R&D Center for Digital Nature / xDiversity
    Design Implications / 4~5
    19
    Do not disrupt the comfortable holding


    of the smartphone in normal situations.
    Encourage users to install a ring-type accessory


    or remove a notebook-type case.
    5.
    4.

    View Slide

  20. ©︎
    R&D Center for Digital Nature / xDiversity
    20
    Designing Gestures for Digital Musical Instruments:


    Gesture Elicitation Study with Deaf and Hard of Hearing People
    Ryo Iijima, Akihisa Shitara, Yoichi Ochiai / University of Tsukuba
    Q&A

    View Slide

  21. ©︎
    R&D Center for Digital Nature / xDiversity
    Q: How was the vibration pattern designed?
    21
    Iteratively designed, tested, and improved


    in a laboratory environment with multiple researchers.

    View Slide

  22. ©︎
    R&D Center for Digital Nature / xDiversity
    Q: Were they influenced by the sensing & recognition technologies?
    22
    They were instructed to treat their smartphone as a


    “ Magic Brick ” that could recognize any gesture.


    They were less influenced by the sensing and


    recognition technologies in creating the gestures.

    View Slide

  23. ©︎
    R&D Center for Digital Nature / xDiversity
    Q: Participants did not play to the rhythm, right?
    23
    The goal is to use gestures to play music and


    not to select which instrument to play.


    To this end, this study aimed to define the gestures


    as preliminary to playing to a rhythm.

    View Slide

  24. ©︎
    R&D Center for Digital Nature / xDiversity
    Q: Any cultural differences?
    24
    Yes, there may be some di
    ff
    erences.


    One participant mimicked a gesture that meant “sutra”


    in sign language, associating the vibration of a wooden fish


    that is used in a Buddhist temple, with that of the cymbal.


    GESs for playing musical instruments with other use groups


    may provide deeper insights into inclusion.

    View Slide

  25. ©︎
    R&D Center for Digital Nature / xDiversity
    Q: Why is the AR of jaw harp so low?
    25
    No participants were familiar with the jaw harp.


    With other instruments, participants often created gestures


    that imitated how they imagined they would play the real one.


    However, this approach did not work with the jaw harp.

    View Slide

  26. ©︎
    R&D Center for Digital Nature / xDiversity
    Q: Why is the AR of tam-tam so low?
    26
    The real tam-tam does not require the performer


    to hold the sonorous object,


    and it can be played with one hand.


    All other idiophones are two-handed instruments,


    so the roles of the hand holding the device and


    the other hand are easily fixed.

    View Slide

  27. ©︎
    R&D Center for Digital Nature / xDiversity
    27
    Designing Gestures for Digital Musical Instruments:


    Gesture Elicitation Study with Deaf and Hard of Hearing People
    Ryo Iijima, Akihisa Shitara, Yoichi Ochiai / University of Tsukuba
    Figures

    View Slide

  28. ©︎
    R&D Center for Digital Nature / xDiversity
    Correlation Analysis
    28

    View Slide

  29. ©︎
    R&D Center for Digital Nature / xDiversity
    Randomizing process
    29

    View Slide

  30. ©︎
    R&D Center for Digital Nature / xDiversity
    Agreement Rate
    30

    View Slide

  31. ©︎
    R&D Center for Digital Nature / xDiversity
    Gesture Grouping Criteria
    31

    View Slide

  32. ©︎
    R&D Center for Digital Nature / xDiversity
    Other instruments to be explored
    32

    View Slide

  33. ©︎
    R&D Center for Digital Nature / xDiversity
    10 instruments
    33

    View Slide

  34. ©︎
    R&D Center for Digital Nature / xDiversity
    Classification
    34

    View Slide