Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Auditory-centered Vocal Feedback System Using Solmization for Training Absolute Pitch without GUI

Auditory-centered Vocal Feedback System Using Solmization for Training Absolute Pitch without GUI

6d667e747a4a1cda7d201c3358424257?s=128

Digital Nature Group

August 21, 2021
Tweet

More Decks by Digital Nature Group

Other Decks in Technology

Transcript

  1. Proceedings session: Interaction techniques 2021-09-01 @ 19:25 – 19:45 Asia/Tokyo

    (UTC+09:00) Auditory-centered Vocal Feedback System Using Solmization for Training Absolute Pitch without GU I Nozomu Yoshida1, Kosaku Namikawa1, Yusuke Koroyasu1, Yoshiki Nagatani2, Yoichi Ochiai1, 2 1University of Tsukuba, 2Pixie Dust Technologies, Inc. #1232
  2. "VEJUPSZDFOUFSFE7PDBM'FFECBDL4ZTUFN6TJOH 
 4PMNJ[BUJPOGPS5SBJOJOH"CTPMVUF1JUDIXJUIPVU(6*

  3. “Absolute Pitch” 3 */5&3"$5 "VEJUPSZDFOUFSFE7PDBM'FFECBDL4ZTUFN6TJOH 
 4PMNJ[BUJPOGPS5SBJOJOH"CTPMVUF1JUDIXJUIPVU(6* — Ability to

    name a note of a given musical tone without an external reference ton e — They hear music as words rather than envisioning it as notes
  4. 4 */5&3"$5 "VEJUPSZDFOUFSFE7PDBM'FFECBDL4ZTUFN6TJOH 
 4PMNJ[BUJPOGPS5SBJOJOH"CTPMVUF1JUDIXJUIPVU(6*

  5. Most current pitch training system = Visual Feedback 5 */5&3"$5

    "VEJUPSZDFOUFSFE7PDBM'FFECBDL4ZTUFN6TJOH 
 4PMNJ[BUJPOGPS5SBJOJOH"CTPMVUF1JUDIXJUIPVU(6* Visualized melody Pitch guessing quiz
  6. < APs hear music as words 6 */5&3"$5 "VEJUPSZDFOUFSFE7PDBM'FFECBDL4ZTUFN6TJOH 


    4PMNJ[BUJPOGPS5SBJOJOH"CTPMVUF1JUDIXJUIPVU(6*
  7. — A method of singing while calling out each sound

    by its note name — Solmization provides singers with vocal feedback of notes in the form of words 7 */5&3"$5 "VEJUPSZDFOUFSFE7PDBM'FFECBDL4ZTUFN6TJOH 
 4PMNJ[BUJPOGPS5SBJOJOH"CTPMVUF1JUDIXJUIPVU(6* “Solmization” *Vocal feedback refers to the feedback of information via the user’s hearing by speech
  8. 8 Introduction Target in This Study */5&3"$5 "VEJUPSZDFOUFSFE7PDBM'FFECBDL4ZTUFN6TJOH 
 4PMNJ[BUJPOGPS5SBJOJOH"CTPMVUF1JUDIXJUIPVU(6*

    Vocal feedback system doesn’t require a GUI Ear Training Myu-Tre Require a GUI Pitch Quiz Karaoke Tuner “Chromatic digital tuner” © Aurinko Needless of GUI Vocal Feedback Live Dam Stadium DX-G Do Mi Sol
  9. 9 Related Work */5&3"$5 "VEJUPSZDFOUFSFE7PDBM'FFECBDL4ZTUFN6TJOH 
 4PMNJ[BUJPOGPS5SBJOJOH"CTPMVUF1JUDIXJUIPVU(6* Writing assistance with

    speech instruction Memory game with non-visual stimuli HCI Study with Vocal Feedback [1] Kuber, R., Tretter, M., Murphy, E.: Developing and evaluating a non-visual memory game. In: Campos, P., Graham, N., Jorge, J., Nunes, N., Palanque, P., Winckler, M. (eds.) Human-Computer Interaction – INTERACT 2011. pp. 541–553. Springer Berlin Heidelberg, Berlin, Heidelberg (2011) Vocal Feedback Vocal Feedback [2] Guardati, L., Casamassima, F., Farella, E., Benini, L.: Paper, pen and ink: An innovative system and software framework to assist writing rehabilitation. In: 2015 Design, Automation Test in Europe Conference Exhibition (DATE). pp. 1473–1478 (2015). https://doi.org/10.7873/DATE.2015.0453
  10. 10 Related Works Position of this study */5&3"$5 "VEJUPSZDFOUFSFE7PDBM'FFECBDL4ZTUFN6TJOH 


    4PMNJ[BUJPOGPS5SBJOJOH"CTPMVUF1JUDIXJUIPVU(6* Pitch training non-pitch training Visual
 Feedback Vocal
 Feedback Our research Writing assistance with speech Memory game with non-visual stimuli Quiz app Karaoke Tuner non-pitch training using visual feedback Categorize the training methods
  11. 11 Introduction Our contributions */5&3"$5 "VEJUPSZDFOUFSFE7PDBM'FFECBDL4ZTUFN6TJOH 
 4PMNJ[BUJPOGPS5SBJOJOH"CTPMVUF1JUDIXJUIPVU(6* Constructed a

    system
 for improving absolute pitch via solmization Evaluated the effectiveness
 By comparing with visual feedback system Training system construction and evaluation
  12. 12 Implementation System Design */5&3"$5 "VEJUPSZDFOUFSFE7PDBM'FFECBDL4ZTUFN6TJOH 
 4PMNJ[BUJPOGPS5SBJOJOH"CTPMVUF1JUDIXJUIPVU(6* Test Generator

    Oscillator User input Vocal Feedback Display
  13. 13 Imagine the pitch Operate the slide bar Visual Feedback


    * Staff notation & note name Listen the exact correct pitch Move to next Bleep sound Visual Vocal Feedback
 * Note name with correct pitch Audio Audio Guess the exact correct pitch 
 * Enter & Check the answer "VEJUPSZDFOUFSFE7PDBM'FFECBDL4ZTUFN6TJOH 
 4PMNJ[BUJPOGPS5SBJOJOH"CTPMVUF1JUDIXJUIPVU(6* */5&3"$5 Vocal Feedback
 Proposed system Visual Feedback
 Counterpart Pitch Test
 For the system-effectiveness-evaluation Implementation Vocal feedback training
  14. 14 Imagine the pitch Operate the slide bar Visual Feedback


    * Staff notation & note name Listen the exact correct pitch Move to next Bleep sound Visual Vocal Feedback
 * Note name with correct pitch Audio Audio Guess the exact correct pitch 
 * Enter & Check the answer "VEJUPSZDFOUFSFE7PDBM'FFECBDL4ZTUFN6TJOH 
 4PMNJ[BUJPOGPS5SBJOJOH"CTPMVUF1JUDIXJUIPVU(6* */5&3"$5 Vocal Feedback
 Proposed system Visual Feedback
 Counterpart Pitch Test
 For the system-effectiveness-evaluation Implementation Vocal feedback training
  15. 15 Imagine the pitch Operate the slide bar Visual Feedback


    * Staff notation & note name Listen the exact correct pitch Move to next Bleep sound Visual Vocal Feedback
 * Note name with correct pitch Audio Audio Guess the exact correct pitch 
 * Enter & Check the answer "VEJUPSZDFOUFSFE7PDBM'FFECBDL4ZTUFN6TJOH 
 4PMNJ[BUJPOGPS5SBJOJOH"CTPMVUF1JUDIXJUIPVU(6* */5&3"$5 Vocal Feedback
 Proposed system Visual Feedback
 Counterpart Pitch Test
 For the system-effectiveness-evaluation Implementation Vocal feedback training
  16. 16 Imagine the pitch Operate the slide bar Visual Feedback


    * Staff notation & note name Listen the exact correct pitch Move to next Bleep sound Visual Vocal Feedback
 * Note name with correct pitch Audio Audio Guess the exact correct pitch 
 * Enter & Check the answer "VEJUPSZDFOUFSFE7PDBM'FFECBDL4ZTUFN6TJOH 
 4PMNJ[BUJPOGPS5SBJOJOH"CTPMVUF1JUDIXJUIPVU(6* */5&3"$5 Vocal Feedback
 Proposed system Visual Feedback
 Counterpart Pitch Test
 For the system-effectiveness-evaluation Implementation Vocal feedback training
  17. 17 Imagine the pitch Operate the slide bar Visual Feedback


    * Staff notation & note name Listen the exact correct pitch Move to next Bleep sound Visual Vocal Feedback
 * Note name with correct pitch Audio Audio Guess the exact correct pitch 
 * Enter & Check the answer "VEJUPSZDFOUFSFE7PDBM'FFECBDL4ZTUFN6TJOH 
 4PMNJ[BUJPOGPS5SBJOJOH"CTPMVUF1JUDIXJUIPVU(6* */5&3"$5 Vocal Feedback
 Proposed system Visual Feedback
 Counterpart Pitch Test
 For the system-effectiveness-evaluation Implementation Vocal feedback training Oscillator Sound Vocal Feedback Bleep Re— System User *User makes sure that the pitch name and pitch are associated and memorize d Listen
  18. 18 Imagine the pitch Operate the slide bar Visual Feedback


    * Staff notation & note name Listen the exact correct pitch Move to next Bleep sound Visual Vocal Feedback
 * Note name with correct pitch Audio Audio Guess the exact correct pitch 
 * Enter & Check the answer "VEJUPSZDFOUFSFE7PDBM'FFECBDL4ZTUFN6TJOH 
 4PMNJ[BUJPOGPS5SBJOJOH"CTPMVUF1JUDIXJUIPVU(6* */5&3"$5 Vocal Feedback
 Proposed system Visual Feedback
 Counterpart Pitch Test
 For the system-effectiveness-evaluation Implementation Vocal feedback training
  19. 19 Imagine the pitch Operate the slide bar Visual Feedback


    * Staff notation & note name Listen the exact correct pitch Move to next Bleep sound Visual Vocal Feedback
 * Note name with correct pitch Audio Audio Guess the exact correct pitch 
 * Enter & Check the answer "VEJUPSZDFOUFSFE7PDBM'FFECBDL4ZTUFN6TJOH 
 4PMNJ[BUJPOGPS5SBJOJOH"CTPMVUF1JUDIXJUIPVU(6* */5&3"$5 Vocal Feedback
 Proposed system Visual Feedback
 Counterpart Pitch Test
 For the system-effectiveness-evaluation Implementation Vocal feedback training
  20. 20 Imagine the pitch Operate the slide bar Visual Feedback


    * Staff notation & note name Listen the exact correct pitch Move to next Bleep sound Visual Vocal Feedback
 * Note name with correct pitch Audio Audio Guess the exact correct pitch 
 * Enter & Check the answer "VEJUPSZDFOUFSFE7PDBM'FFECBDL4ZTUFN6TJOH 
 4PMNJ[BUJPOGPS5SBJOJOH"CTPMVUF1JUDIXJUIPVU(6* */5&3"$5 Vocal Feedback
 Proposed system Visual Feedback
 Counterpart Pitch Test
 For the system-effectiveness-evaluation Implementation Visual feedback training
  21. 21 Imagine the pitch Operate the slide bar Visual Feedback


    * Staff notation & note name Listen the exact correct pitch Move to next Bleep sound Visual Vocal Feedback
 * Note name with correct pitch Audio Audio Guess the exact correct pitch 
 * Enter & Check the answer "VEJUPSZDFOUFSFE7PDBM'FFECBDL4ZTUFN6TJOH 
 4PMNJ[BUJPOGPS5SBJOJOH"CTPMVUF1JUDIXJUIPVU(6* */5&3"$5 Vocal Feedback
 Proposed system Visual Feedback
 Counterpart Pitch Test
 For the system-effectiveness-evaluation Implementation Visual feedback training
  22. 22 Imagine the pitch Operate the slide bar Visual Feedback


    * Staff notation & note name Listen the exact correct pitch Move to next Bleep sound Visual Vocal Feedback
 * Note name with correct pitch Audio Audio Guess the exact correct pitch 
 * Enter & Check the answer "VEJUPSZDFOUFSFE7PDBM'FFECBDL4ZTUFN6TJOH 
 4PMNJ[BUJPOGPS5SBJOJOH"CTPMVUF1JUDIXJUIPVU(6* */5&3"$5 Vocal Feedback
 Proposed system Visual Feedback
 Counterpart Pitch Test
 For the system-effectiveness-evaluation Oscillator Sound Visual Feedback Bleep Staff notation & note name System User *User makes sure that the pitch name and pitch are associated and memorize d Listen See Implementation Visual feedback training
  23. 23 Imagine the pitch Operate the slide bar Visual Feedback


    * Staff notation & note name Listen the exact correct pitch Move to next Bleep sound Visual Vocal Feedback
 * Note name with correct pitch Audio Audio Guess the exact correct pitch 
 * Enter & Check the answer "VEJUPSZDFOUFSFE7PDBM'FFECBDL4ZTUFN6TJOH 
 4PMNJ[BUJPOGPS5SBJOJOH"CTPMVUF1JUDIXJUIPVU(6* */5&3"$5 Vocal Feedback
 Proposed system Visual Feedback
 Counterpart Pitch Test
 For the system-effectiveness-evaluation Implementation Absolute pitch test
  24. 24 Imagine the pitch Operate the slide bar Visual Feedback


    * Staff notation & note name Listen the exact correct pitch Move to next Bleep sound Visual Vocal Feedback
 * Note name with correct pitch Audio Audio Guess the exact correct pitch 
 * Enter & Check the answer "VEJUPSZDFOUFSFE7PDBM'FFECBDL4ZTUFN6TJOH 
 4PMNJ[BUJPOGPS5SBJOJOH"CTPMVUF1JUDIXJUIPVU(6* */5&3"$5 Vocal Feedback
 Proposed system Visual Feedback
 Counterpart Pitch Test
 For the system-effectiveness-evaluation Implementation Absolute pitch test
  25. 25 Imagine the pitch Operate the slide bar Visual Feedback


    * Staff notation & note name Listen the exact correct pitch Move to next Bleep sound Visual Vocal Feedback
 * Note name with correct pitch Audio Audio Guess the exact correct pitch 
 * Enter & Check the answer "VEJUPSZDFOUFSFE7PDBM'FFECBDL4ZTUFN6TJOH 
 4PMNJ[BUJPOGPS5SBJOJOH"CTPMVUF1JUDIXJUIPVU(6* */5&3"$5 Vocal Feedback
 Proposed system Visual Feedback
 Counterpart Pitch Test
 For the system-effectiveness-evaluation Oscillator Sound Bleep System User *User makes sure that the pitch name and pitch are associated and memorize d Listen Implementation Absolute pitch test
  26. 26 Imagine the pitch Operate the slide bar Visual Feedback


    * Staff notation & note name Listen the exact correct pitch Move to next Bleep sound Visual Vocal Feedback
 * Note name with correct pitch Audio Audio Guess the exact correct pitch 
 * Enter & Check the answer "VEJUPSZDFOUFSFE7PDBM'FFECBDL4ZTUFN6TJOH 
 4PMNJ[BUJPOGPS5SBJOJOH"CTPMVUF1JUDIXJUIPVU(6* */5&3"$5 Vocal Feedback
 Proposed system Visual Feedback
 Counterpart Pitch Test
 For the system-effectiveness-evaluation Implementation Absolute pitch test
  27. 27 Imagine the pitch Operate the slide bar Visual Feedback


    * Staff notation & note name Listen the exact correct pitch Move to next Bleep sound Visual Vocal Feedback
 * Note name with correct pitch Audio Audio Guess the exact correct pitch 
 * Enter & Check the answer "VEJUPSZDFOUFSFE7PDBM'FFECBDL4ZTUFN6TJOH 
 4PMNJ[BUJPOGPS5SBJOJOH"CTPMVUF1JUDIXJUIPVU(6* */5&3"$5 Vocal Feedback
 Proposed system Visual Feedback
 Counterpart Pitch Test
 For the system-effectiveness-evaluation Implementation Absolute pitch test
  28. 28 Imagine the pitch Operate the slide bar Visual Feedback


    * Staff notation & note name Listen the exact correct pitch Move to next Bleep sound Visual Vocal Feedback
 * Note name with correct pitch Audio Audio Guess the exact correct pitch 
 * Enter & Check the answer "VEJUPSZDFOUFSFE7PDBM'FFECBDL4ZTUFN6TJOH 
 4PMNJ[BUJPOGPS5SBJOJOH"CTPMVUF1JUDIXJUIPVU(6* */5&3"$5 Vocal Feedback
 Proposed system Visual Feedback
 Counterpart Pitch Test
 For the system-effectiveness-evaluation Implementation Absolute pitch test
  29. 29 Implementation Hardware set up */5&3"$5 "VEJUPSZDFOUFSFE7PDBM'FFECBDL4ZTUFN6TJOH 
 4PMNJ[BUJPOGPS5SBJOJOH"CTPMVUF1JUDIXJUIPVU(6* System

    User Mouse Headphone
  30. 30 */5&3"$5 "VEJUPSZDFOUFSFE7PDBM'FFECBDL4ZTUFN6TJOH 
 4PMNJ[BUJPOGPS5SBJOJOH"CTPMVUF1JUDIXJUIPVU(6* Hypothesis < Visual Feedback Vocal

    Feedback in pitch training
  31. 31 */5&3"$5 "VEJUPSZDFOUFSFE7PDBM'FFECBDL4ZTUFN6TJOH 
 4PMNJ[BUJPOGPS5SBJOJOH"CTPMVUF1JUDIXJUIPVU(6* Experiment Overview Pattern # of

    participants Pre-test Main Test 1 5 people Absolute pitch qui z (10 question) C:
 Test Tas k (10 question) A:
 Training Tas k Vocal Feedbac k (5 min) C:
 Test Tas k (10 question) Interva l (5 - 10 min) C:
 Test Tas k (10 question) B:
 Training Tas k Visual Feedbac k (5 min) C:
 Test Tas k (10 question) 2 5 people B:
 Training Tas k Visual Feedbac k (5 min) A:
 Training Tas k Vocal Feedbac k (5 min) *We prepared two patterns to reduce possible bias caused by the training order. Patterns are randomly assigned to be half number. Start End *Additionally, questionnaires was conducted before and after the experiment to survey user’s attributes, the experiment process and the usability of the system .
  32. 32 Experiment Preliminary Test */5&3"$5 "VEJUPSZDFOUFSFE7PDBM'FFECBDL4ZTUFN6TJOH 
 4PMNJ[BUJPOGPS5SBJOJOH"CTPMVUF1JUDIXJUIPVU(6* For grouping

    purpose… Perfect score (10/10) Non-perfect score Potential absolute pitch Non-potential absolute pitch Pitch quiz app 
 Ear Training Myu-Tre: https://apps.apple.com/jp/app/id1015269208
  33. 33 */5&3"$5 Improve value: –0.09 Potential AP → Neither systems

    showed signi fi cant improvement Improve value: –0.07 *Error means the difference between the correct pitch and user’s answer. → For example, if the correct pitch is C, but user answered it as C#, the error will be 1.0 . *The average error is the average difference between the correct pitch and input pitch from the 10 questions in test task C. *The improvement value refers to the difference between the average error before and after training. Result Main Test: Potential absolute pitch Error Error Participants ID
  34. 34 */5&3"$5 Improve value: 0.65 Non-potential AP → Only proposed

    system showed signi fi cant improvement Improve value: –0.49 *Error means the difference between the correct pitch and user’s answer. → For example, if the correct pitch is C, but user answered it as C#, the error will be 1.0 . *The average error is the average difference between the correct pitch and input pitch from the 10 questions in test task C. *The improvement value refers to the difference between the average error before and after training. Result Main Test: Non-potential absolute pitch Error Error Participants ID
  35. 35 Result Statistical Hypothesis Testing */5&3"$5 "VEJUPSZDFOUFSFE7PDBM'FFECBDL4ZTUFN6TJOH 
 4PMNJ[BUJPOGPS5SBJOJOH"CTPMVUF1JUDIXJUIPVU(6* Method:

    Wilcoxon signed-rank tes t H0: There would be no difference in the average error before and after training with the proposed/existing syste m H1: There would be a difference in the average error before and after training with the proposed/existing system. Signi fi cance only when non-potential AP with vocal feedback
  36. 36 Discussion Difference between potential and non-potential participants */5&3"$5 "VEJUPSZDFOUFSFE7PDBM'FFECBDL4ZTUFN6TJOH

    
 4PMNJ[BUJPOGPS5SBJOJOH"CTPMVUF1JUDIXJUIPVU(6* The potential absolute pitch score before training was already high, so the score after training was unlikely to change. Small improvement value Small improvement value because
  37. Discussion Self-evaluation of the pitch */5&3"$5 "VEJUPSZDFOUFSFE7PDBM'FFECBDL4ZTUFN6TJOH 
 4PMNJ[BUJPOGPS5SBJOJOH"CTPMVUF1JUDIXJUIPVU(6* Little

    in fl uence on their self-evaluation of their absolute pitch
  38. 38 Discussion User feedback */5&3"$5 "VEJUPSZDFOUFSFE7PDBM'FFECBDL4ZTUFN6TJOH 
 4PMNJ[BUJPOGPS5SBJOJOH"CTPMVUF1JUDIXJUIPVU(6* 80% prefer

    the vocal feedback system for long-term training
  39. 39 Discussion improvements */5&3"$5 "VEJUPSZDFOUFSFE7PDBM'FFECBDL4ZTUFN6TJOH 
 4PMNJ[BUJPOGPS5SBJOJOH"CTPMVUF1JUDIXJUIPVU(6* Do# → ‘Do

    sharp’ Visual feedback Vocal feedback Redundancy of ‘sharp’ in vocal feedback
  40. 40 Discussion improvements */5&3"$5 "VEJUPSZDFOUFSFE7PDBM'FFECBDL4ZTUFN6TJOH 
 4PMNJ[BUJPOGPS5SBJOJOH"CTPMVUF1JUDIXJUIPVU(6* Do# → ‘Cis’

    Visual feedback Vocal feedback Redundancy of ‘sharp’ in vocal feedback German note names
  41. 41 Application Other examples of system input */5&3"$5 "VEJUPSZDFOUFSFE7PDBM'FFECBDL4ZTUFN6TJOH 


    4PMNJ[BUJPOGPS5SBJOJOH"CTPMVUF1JUDIXJUIPVU(6* Non-GUI Application by limiting the system to auditory system only instead of slide bar on GUI Voice Instrument
  42. 42 Application Other Examples of pitch training */5&3"$5 "VEJUPSZDFOUFSFE7PDBM'FFECBDL4ZTUFN6TJOH 


    4PMNJ[BUJPOGPS5SBJOJOH"CTPMVUF1JUDIXJUIPVU(6* Relative Pitch training https://hellomusictheory.com/learn/relative-pitch/ By giving the reference tone sound
  43. Auditory-centered Vocal Feedback System Using Solmization for Training Absolute Pitch

    without GU I Nozomu Yoshida1, Kosaku Namikawa1, Yusuke Koroyasu1, Yoshiki Nagatani2, Yoichi Ochiai1, 2 1University of Tsukuba, 2Pixie Dust Technologies, Inc. Proceedings session: Interaction techniques 2021-09-01 @ 19:25 – 19:45 Asia/Tokyo (UTC+09:00) Contact: nozo@digitalnature.slis.tsukuba.ac.jp Thank you
  44. 2. Why vocal feedback system adopted a GUI in the

    experiment? 3. Could the participants keep the improvement after the experiment? 1. Which tools or frameworks did you used for system implementation? — In this research experiment, a GUI-based slide bar was used as the system input to unify the experimental conditions between vocal and visual feedback. — We don’t track the improvement after the experiment. It may require continuous training for keeping their pitch improvement. — MaxMSP for the application implementation. FAQ */5&3"$5 "VEJUPSZDFOUFSFE7PDBM'FFECBDL4ZTUFN6TJOH 
 4PMNJ[BUJPOGPS5SBJOJOH"CTPMVUF1JUDIXJUIPVU(6* 4. Could the participants memorized the location of the pitch in slide bar? — The slider’s pitch change rate per width is random for each quiz to prevent the user from memorizing the note according to slider position.