Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Auditory-centered Vocal Feedback System Using Solmization for Training Absolute Pitch without GUI

Auditory-centered Vocal Feedback System Using Solmization for Training Absolute Pitch without GUI

Digital Nature Group

August 21, 2021
Tweet

More Decks by Digital Nature Group

Other Decks in Technology

Transcript

  1. Proceedings session: Interaction techniques 2021-09-01 @ 19:25 – 19:45 Asia/Tokyo

    (UTC+09:00) Auditory-centered Vocal Feedback System Using Solmization for Training Absolute Pitch without GU I Nozomu Yoshida1, Kosaku Namikawa1, Yusuke Koroyasu1, Yoshiki Nagatani2, Yoichi Ochiai1, 2 1University of Tsukuba, 2Pixie Dust Technologies, Inc. #1232
  2. "VEJUPSZDFOUFSFE7PDBM'FFECBDL4ZTUFN6TJOH 
 4PMNJ[BUJPOGPS5SBJOJOH"CTPMVUF1JUDIXJUIPVU(6*

  3. “Absolute Pitch” 3 */5&3"$5 "VEJUPSZDFOUFSFE7PDBM'FFECBDL4ZTUFN6TJOH 
 4PMNJ[BUJPOGPS5SBJOJOH"CTPMVUF1JUDIXJUIPVU(6* — Ability to

    name a note of a given musical tone without an external reference ton e — They hear music as words rather than envisioning it as notes
  4. 4 */5&3"$5 "VEJUPSZDFOUFSFE7PDBM'FFECBDL4ZTUFN6TJOH 
 4PMNJ[BUJPOGPS5SBJOJOH"CTPMVUF1JUDIXJUIPVU(6*

  5. Most current pitch training system = Visual Feedback 5 */5&3"$5

    "VEJUPSZDFOUFSFE7PDBM'FFECBDL4ZTUFN6TJOH 
 4PMNJ[BUJPOGPS5SBJOJOH"CTPMVUF1JUDIXJUIPVU(6* Visualized melody Pitch guessing quiz
  6. < APs hear music as words 6 */5&3"$5 "VEJUPSZDFOUFSFE7PDBM'FFECBDL4ZTUFN6TJOH 


    4PMNJ[BUJPOGPS5SBJOJOH"CTPMVUF1JUDIXJUIPVU(6*
  7. — A method of singing while calling out each sound

    by its note name — Solmization provides singers with vocal feedback of notes in the form of words 7 */5&3"$5 "VEJUPSZDFOUFSFE7PDBM'FFECBDL4ZTUFN6TJOH 
 4PMNJ[BUJPOGPS5SBJOJOH"CTPMVUF1JUDIXJUIPVU(6* “Solmization” *Vocal feedback refers to the feedback of information via the user’s hearing by speech
  8. 8 Introduction Target in This Study */5&3"$5 "VEJUPSZDFOUFSFE7PDBM'FFECBDL4ZTUFN6TJOH 
 4PMNJ[BUJPOGPS5SBJOJOH"CTPMVUF1JUDIXJUIPVU(6*

    Vocal feedback system doesn’t require a GUI Ear Training Myu-Tre Require a GUI Pitch Quiz Karaoke Tuner “Chromatic digital tuner” © Aurinko Needless of GUI Vocal Feedback Live Dam Stadium DX-G Do Mi Sol
  9. 9 Related Work */5&3"$5 "VEJUPSZDFOUFSFE7PDBM'FFECBDL4ZTUFN6TJOH 
 4PMNJ[BUJPOGPS5SBJOJOH"CTPMVUF1JUDIXJUIPVU(6* Writing assistance with

    speech instruction Memory game with non-visual stimuli HCI Study with Vocal Feedback [1] Kuber, R., Tretter, M., Murphy, E.: Developing and evaluating a non-visual memory game. In: Campos, P., Graham, N., Jorge, J., Nunes, N., Palanque, P., Winckler, M. (eds.) Human-Computer Interaction – INTERACT 2011. pp. 541–553. Springer Berlin Heidelberg, Berlin, Heidelberg (2011) Vocal Feedback Vocal Feedback [2] Guardati, L., Casamassima, F., Farella, E., Benini, L.: Paper, pen and ink: An innovative system and software framework to assist writing rehabilitation. In: 2015 Design, Automation Test in Europe Conference Exhibition (DATE). pp. 1473–1478 (2015). https://doi.org/10.7873/DATE.2015.0453
  10. 10 Related Works Position of this study */5&3"$5 "VEJUPSZDFOUFSFE7PDBM'FFECBDL4ZTUFN6TJOH 


    4PMNJ[BUJPOGPS5SBJOJOH"CTPMVUF1JUDIXJUIPVU(6* Pitch training non-pitch training Visual
 Feedback Vocal
 Feedback Our research Writing assistance with speech Memory game with non-visual stimuli Quiz app Karaoke Tuner non-pitch training using visual feedback Categorize the training methods
  11. 11 Introduction Our contributions */5&3"$5 "VEJUPSZDFOUFSFE7PDBM'FFECBDL4ZTUFN6TJOH 
 4PMNJ[BUJPOGPS5SBJOJOH"CTPMVUF1JUDIXJUIPVU(6* Constructed a

    system
 for improving absolute pitch via solmization Evaluated the effectiveness
 By comparing with visual feedback system Training system construction and evaluation
  12. 12 Implementation System Design */5&3"$5 "VEJUPSZDFOUFSFE7PDBM'FFECBDL4ZTUFN6TJOH 
 4PMNJ[BUJPOGPS5SBJOJOH"CTPMVUF1JUDIXJUIPVU(6* Test Generator

    Oscillator User input Vocal Feedback Display
  13. 13 Imagine the pitch Operate the slide bar Visual Feedback


    * Staff notation & note name Listen the exact correct pitch Move to next Bleep sound Visual Vocal Feedback
 * Note name with correct pitch Audio Audio Guess the exact correct pitch 
 * Enter & Check the answer "VEJUPSZDFOUFSFE7PDBM'FFECBDL4ZTUFN6TJOH 
 4PMNJ[BUJPOGPS5SBJOJOH"CTPMVUF1JUDIXJUIPVU(6* */5&3"$5 Vocal Feedback
 Proposed system Visual Feedback
 Counterpart Pitch Test
 For the system-effectiveness-evaluation Implementation Vocal feedback training
  14. 14 Imagine the pitch Operate the slide bar Visual Feedback


    * Staff notation & note name Listen the exact correct pitch Move to next Bleep sound Visual Vocal Feedback
 * Note name with correct pitch Audio Audio Guess the exact correct pitch 
 * Enter & Check the answer "VEJUPSZDFOUFSFE7PDBM'FFECBDL4ZTUFN6TJOH 
 4PMNJ[BUJPOGPS5SBJOJOH"CTPMVUF1JUDIXJUIPVU(6* */5&3"$5 Vocal Feedback
 Proposed system Visual Feedback
 Counterpart Pitch Test
 For the system-effectiveness-evaluation Implementation Vocal feedback training
  15. 15 Imagine the pitch Operate the slide bar Visual Feedback


    * Staff notation & note name Listen the exact correct pitch Move to next Bleep sound Visual Vocal Feedback
 * Note name with correct pitch Audio Audio Guess the exact correct pitch 
 * Enter & Check the answer "VEJUPSZDFOUFSFE7PDBM'FFECBDL4ZTUFN6TJOH 
 4PMNJ[BUJPOGPS5SBJOJOH"CTPMVUF1JUDIXJUIPVU(6* */5&3"$5 Vocal Feedback
 Proposed system Visual Feedback
 Counterpart Pitch Test
 For the system-effectiveness-evaluation Implementation Vocal feedback training
  16. 16 Imagine the pitch Operate the slide bar Visual Feedback


    * Staff notation & note name Listen the exact correct pitch Move to next Bleep sound Visual Vocal Feedback
 * Note name with correct pitch Audio Audio Guess the exact correct pitch 
 * Enter & Check the answer "VEJUPSZDFOUFSFE7PDBM'FFECBDL4ZTUFN6TJOH 
 4PMNJ[BUJPOGPS5SBJOJOH"CTPMVUF1JUDIXJUIPVU(6* */5&3"$5 Vocal Feedback
 Proposed system Visual Feedback
 Counterpart Pitch Test
 For the system-effectiveness-evaluation Implementation Vocal feedback training
  17. 17 Imagine the pitch Operate the slide bar Visual Feedback


    * Staff notation & note name Listen the exact correct pitch Move to next Bleep sound Visual Vocal Feedback
 * Note name with correct pitch Audio Audio Guess the exact correct pitch 
 * Enter & Check the answer "VEJUPSZDFOUFSFE7PDBM'FFECBDL4ZTUFN6TJOH 
 4PMNJ[BUJPOGPS5SBJOJOH"CTPMVUF1JUDIXJUIPVU(6* */5&3"$5 Vocal Feedback
 Proposed system Visual Feedback
 Counterpart Pitch Test
 For the system-effectiveness-evaluation Implementation Vocal feedback training Oscillator Sound Vocal Feedback Bleep Re— System User *User makes sure that the pitch name and pitch are associated and memorize d Listen
  18. 18 Imagine the pitch Operate the slide bar Visual Feedback


    * Staff notation & note name Listen the exact correct pitch Move to next Bleep sound Visual Vocal Feedback
 * Note name with correct pitch Audio Audio Guess the exact correct pitch 
 * Enter & Check the answer "VEJUPSZDFOUFSFE7PDBM'FFECBDL4ZTUFN6TJOH 
 4PMNJ[BUJPOGPS5SBJOJOH"CTPMVUF1JUDIXJUIPVU(6* */5&3"$5 Vocal Feedback
 Proposed system Visual Feedback
 Counterpart Pitch Test
 For the system-effectiveness-evaluation Implementation Vocal feedback training
  19. 19 Imagine the pitch Operate the slide bar Visual Feedback


    * Staff notation & note name Listen the exact correct pitch Move to next Bleep sound Visual Vocal Feedback
 * Note name with correct pitch Audio Audio Guess the exact correct pitch 
 * Enter & Check the answer "VEJUPSZDFOUFSFE7PDBM'FFECBDL4ZTUFN6TJOH 
 4PMNJ[BUJPOGPS5SBJOJOH"CTPMVUF1JUDIXJUIPVU(6* */5&3"$5 Vocal Feedback
 Proposed system Visual Feedback
 Counterpart Pitch Test
 For the system-effectiveness-evaluation Implementation Vocal feedback training
  20. 20 Imagine the pitch Operate the slide bar Visual Feedback


    * Staff notation & note name Listen the exact correct pitch Move to next Bleep sound Visual Vocal Feedback
 * Note name with correct pitch Audio Audio Guess the exact correct pitch 
 * Enter & Check the answer "VEJUPSZDFOUFSFE7PDBM'FFECBDL4ZTUFN6TJOH 
 4PMNJ[BUJPOGPS5SBJOJOH"CTPMVUF1JUDIXJUIPVU(6* */5&3"$5 Vocal Feedback
 Proposed system Visual Feedback
 Counterpart Pitch Test
 For the system-effectiveness-evaluation Implementation Visual feedback training
  21. 21 Imagine the pitch Operate the slide bar Visual Feedback


    * Staff notation & note name Listen the exact correct pitch Move to next Bleep sound Visual Vocal Feedback
 * Note name with correct pitch Audio Audio Guess the exact correct pitch 
 * Enter & Check the answer "VEJUPSZDFOUFSFE7PDBM'FFECBDL4ZTUFN6TJOH 
 4PMNJ[BUJPOGPS5SBJOJOH"CTPMVUF1JUDIXJUIPVU(6* */5&3"$5 Vocal Feedback
 Proposed system Visual Feedback
 Counterpart Pitch Test
 For the system-effectiveness-evaluation Implementation Visual feedback training
  22. 22 Imagine the pitch Operate the slide bar Visual Feedback


    * Staff notation & note name Listen the exact correct pitch Move to next Bleep sound Visual Vocal Feedback
 * Note name with correct pitch Audio Audio Guess the exact correct pitch 
 * Enter & Check the answer "VEJUPSZDFOUFSFE7PDBM'FFECBDL4ZTUFN6TJOH 
 4PMNJ[BUJPOGPS5SBJOJOH"CTPMVUF1JUDIXJUIPVU(6* */5&3"$5 Vocal Feedback
 Proposed system Visual Feedback
 Counterpart Pitch Test
 For the system-effectiveness-evaluation Oscillator Sound Visual Feedback Bleep Staff notation & note name System User *User makes sure that the pitch name and pitch are associated and memorize d Listen See Implementation Visual feedback training
  23. 23 Imagine the pitch Operate the slide bar Visual Feedback


    * Staff notation & note name Listen the exact correct pitch Move to next Bleep sound Visual Vocal Feedback
 * Note name with correct pitch Audio Audio Guess the exact correct pitch 
 * Enter & Check the answer "VEJUPSZDFOUFSFE7PDBM'FFECBDL4ZTUFN6TJOH 
 4PMNJ[BUJPOGPS5SBJOJOH"CTPMVUF1JUDIXJUIPVU(6* */5&3"$5 Vocal Feedback
 Proposed system Visual Feedback
 Counterpart Pitch Test
 For the system-effectiveness-evaluation Implementation Absolute pitch test
  24. 24 Imagine the pitch Operate the slide bar Visual Feedback


    * Staff notation & note name Listen the exact correct pitch Move to next Bleep sound Visual Vocal Feedback
 * Note name with correct pitch Audio Audio Guess the exact correct pitch 
 * Enter & Check the answer "VEJUPSZDFOUFSFE7PDBM'FFECBDL4ZTUFN6TJOH 
 4PMNJ[BUJPOGPS5SBJOJOH"CTPMVUF1JUDIXJUIPVU(6* */5&3"$5 Vocal Feedback
 Proposed system Visual Feedback
 Counterpart Pitch Test
 For the system-effectiveness-evaluation Implementation Absolute pitch test
  25. 25 Imagine the pitch Operate the slide bar Visual Feedback


    * Staff notation & note name Listen the exact correct pitch Move to next Bleep sound Visual Vocal Feedback
 * Note name with correct pitch Audio Audio Guess the exact correct pitch 
 * Enter & Check the answer "VEJUPSZDFOUFSFE7PDBM'FFECBDL4ZTUFN6TJOH 
 4PMNJ[BUJPOGPS5SBJOJOH"CTPMVUF1JUDIXJUIPVU(6* */5&3"$5 Vocal Feedback
 Proposed system Visual Feedback
 Counterpart Pitch Test
 For the system-effectiveness-evaluation Oscillator Sound Bleep System User *User makes sure that the pitch name and pitch are associated and memorize d Listen Implementation Absolute pitch test
  26. 26 Imagine the pitch Operate the slide bar Visual Feedback


    * Staff notation & note name Listen the exact correct pitch Move to next Bleep sound Visual Vocal Feedback
 * Note name with correct pitch Audio Audio Guess the exact correct pitch 
 * Enter & Check the answer "VEJUPSZDFOUFSFE7PDBM'FFECBDL4ZTUFN6TJOH 
 4PMNJ[BUJPOGPS5SBJOJOH"CTPMVUF1JUDIXJUIPVU(6* */5&3"$5 Vocal Feedback
 Proposed system Visual Feedback
 Counterpart Pitch Test
 For the system-effectiveness-evaluation Implementation Absolute pitch test
  27. 27 Imagine the pitch Operate the slide bar Visual Feedback


    * Staff notation & note name Listen the exact correct pitch Move to next Bleep sound Visual Vocal Feedback
 * Note name with correct pitch Audio Audio Guess the exact correct pitch 
 * Enter & Check the answer "VEJUPSZDFOUFSFE7PDBM'FFECBDL4ZTUFN6TJOH 
 4PMNJ[BUJPOGPS5SBJOJOH"CTPMVUF1JUDIXJUIPVU(6* */5&3"$5 Vocal Feedback
 Proposed system Visual Feedback
 Counterpart Pitch Test
 For the system-effectiveness-evaluation Implementation Absolute pitch test
  28. 28 Imagine the pitch Operate the slide bar Visual Feedback


    * Staff notation & note name Listen the exact correct pitch Move to next Bleep sound Visual Vocal Feedback
 * Note name with correct pitch Audio Audio Guess the exact correct pitch 
 * Enter & Check the answer "VEJUPSZDFOUFSFE7PDBM'FFECBDL4ZTUFN6TJOH 
 4PMNJ[BUJPOGPS5SBJOJOH"CTPMVUF1JUDIXJUIPVU(6* */5&3"$5 Vocal Feedback
 Proposed system Visual Feedback
 Counterpart Pitch Test
 For the system-effectiveness-evaluation Implementation Absolute pitch test
  29. 29 Implementation Hardware set up */5&3"$5 "VEJUPSZDFOUFSFE7PDBM'FFECBDL4ZTUFN6TJOH 
 4PMNJ[BUJPOGPS5SBJOJOH"CTPMVUF1JUDIXJUIPVU(6* System

    User Mouse Headphone
  30. 30 */5&3"$5 "VEJUPSZDFOUFSFE7PDBM'FFECBDL4ZTUFN6TJOH 
 4PMNJ[BUJPOGPS5SBJOJOH"CTPMVUF1JUDIXJUIPVU(6* Hypothesis < Visual Feedback Vocal

    Feedback in pitch training
  31. 31 */5&3"$5 "VEJUPSZDFOUFSFE7PDBM'FFECBDL4ZTUFN6TJOH 
 4PMNJ[BUJPOGPS5SBJOJOH"CTPMVUF1JUDIXJUIPVU(6* Experiment Overview Pattern # of

    participants Pre-test Main Test 1 5 people Absolute pitch qui z (10 question) C:
 Test Tas k (10 question) A:
 Training Tas k Vocal Feedbac k (5 min) C:
 Test Tas k (10 question) Interva l (5 - 10 min) C:
 Test Tas k (10 question) B:
 Training Tas k Visual Feedbac k (5 min) C:
 Test Tas k (10 question) 2 5 people B:
 Training Tas k Visual Feedbac k (5 min) A:
 Training Tas k Vocal Feedbac k (5 min) *We prepared two patterns to reduce possible bias caused by the training order. Patterns are randomly assigned to be half number. Start End *Additionally, questionnaires was conducted before and after the experiment to survey user’s attributes, the experiment process and the usability of the system .
  32. 32 Experiment Preliminary Test */5&3"$5 "VEJUPSZDFOUFSFE7PDBM'FFECBDL4ZTUFN6TJOH 
 4PMNJ[BUJPOGPS5SBJOJOH"CTPMVUF1JUDIXJUIPVU(6* For grouping

    purpose… Perfect score (10/10) Non-perfect score Potential absolute pitch Non-potential absolute pitch Pitch quiz app 
 Ear Training Myu-Tre: https://apps.apple.com/jp/app/id1015269208
  33. 33 */5&3"$5 Improve value: –0.09 Potential AP → Neither systems

    showed signi fi cant improvement Improve value: –0.07 *Error means the difference between the correct pitch and user’s answer. → For example, if the correct pitch is C, but user answered it as C#, the error will be 1.0 . *The average error is the average difference between the correct pitch and input pitch from the 10 questions in test task C. *The improvement value refers to the difference between the average error before and after training. Result Main Test: Potential absolute pitch Error Error Participants ID
  34. 34 */5&3"$5 Improve value: 0.65 Non-potential AP → Only proposed

    system showed signi fi cant improvement Improve value: –0.49 *Error means the difference between the correct pitch and user’s answer. → For example, if the correct pitch is C, but user answered it as C#, the error will be 1.0 . *The average error is the average difference between the correct pitch and input pitch from the 10 questions in test task C. *The improvement value refers to the difference between the average error before and after training. Result Main Test: Non-potential absolute pitch Error Error Participants ID
  35. 35 Result Statistical Hypothesis Testing */5&3"$5 "VEJUPSZDFOUFSFE7PDBM'FFECBDL4ZTUFN6TJOH 
 4PMNJ[BUJPOGPS5SBJOJOH"CTPMVUF1JUDIXJUIPVU(6* Method:

    Wilcoxon signed-rank tes t H0: There would be no difference in the average error before and after training with the proposed/existing syste m H1: There would be a difference in the average error before and after training with the proposed/existing system. Signi fi cance only when non-potential AP with vocal feedback
  36. 36 Discussion Difference between potential and non-potential participants */5&3"$5 "VEJUPSZDFOUFSFE7PDBM'FFECBDL4ZTUFN6TJOH

    
 4PMNJ[BUJPOGPS5SBJOJOH"CTPMVUF1JUDIXJUIPVU(6* The potential absolute pitch score before training was already high, so the score after training was unlikely to change. Small improvement value Small improvement value because
  37. Discussion Self-evaluation of the pitch */5&3"$5 "VEJUPSZDFOUFSFE7PDBM'FFECBDL4ZTUFN6TJOH 
 4PMNJ[BUJPOGPS5SBJOJOH"CTPMVUF1JUDIXJUIPVU(6* Little

    in fl uence on their self-evaluation of their absolute pitch
  38. 38 Discussion User feedback */5&3"$5 "VEJUPSZDFOUFSFE7PDBM'FFECBDL4ZTUFN6TJOH 
 4PMNJ[BUJPOGPS5SBJOJOH"CTPMVUF1JUDIXJUIPVU(6* 80% prefer

    the vocal feedback system for long-term training
  39. 39 Discussion improvements */5&3"$5 "VEJUPSZDFOUFSFE7PDBM'FFECBDL4ZTUFN6TJOH 
 4PMNJ[BUJPOGPS5SBJOJOH"CTPMVUF1JUDIXJUIPVU(6* Do# → ‘Do

    sharp’ Visual feedback Vocal feedback Redundancy of ‘sharp’ in vocal feedback
  40. 40 Discussion improvements */5&3"$5 "VEJUPSZDFOUFSFE7PDBM'FFECBDL4ZTUFN6TJOH 
 4PMNJ[BUJPOGPS5SBJOJOH"CTPMVUF1JUDIXJUIPVU(6* Do# → ‘Cis’

    Visual feedback Vocal feedback Redundancy of ‘sharp’ in vocal feedback German note names
  41. 41 Application Other examples of system input */5&3"$5 "VEJUPSZDFOUFSFE7PDBM'FFECBDL4ZTUFN6TJOH 


    4PMNJ[BUJPOGPS5SBJOJOH"CTPMVUF1JUDIXJUIPVU(6* Non-GUI Application by limiting the system to auditory system only instead of slide bar on GUI Voice Instrument
  42. 42 Application Other Examples of pitch training */5&3"$5 "VEJUPSZDFOUFSFE7PDBM'FFECBDL4ZTUFN6TJOH 


    4PMNJ[BUJPOGPS5SBJOJOH"CTPMVUF1JUDIXJUIPVU(6* Relative Pitch training https://hellomusictheory.com/learn/relative-pitch/ By giving the reference tone sound
  43. Auditory-centered Vocal Feedback System Using Solmization for Training Absolute Pitch

    without GU I Nozomu Yoshida1, Kosaku Namikawa1, Yusuke Koroyasu1, Yoshiki Nagatani2, Yoichi Ochiai1, 2 1University of Tsukuba, 2Pixie Dust Technologies, Inc. Proceedings session: Interaction techniques 2021-09-01 @ 19:25 – 19:45 Asia/Tokyo (UTC+09:00) Contact: [email protected] Thank you
  44. 2. Why vocal feedback system adopted a GUI in the

    experiment? 3. Could the participants keep the improvement after the experiment? 1. Which tools or frameworks did you used for system implementation? — In this research experiment, a GUI-based slide bar was used as the system input to unify the experimental conditions between vocal and visual feedback. — We don’t track the improvement after the experiment. It may require continuous training for keeping their pitch improvement. — MaxMSP for the application implementation. FAQ */5&3"$5 "VEJUPSZDFOUFSFE7PDBM'FFECBDL4ZTUFN6TJOH 
 4PMNJ[BUJPOGPS5SBJOJOH"CTPMVUF1JUDIXJUIPVU(6* 4. Could the participants memorized the location of the pitch in slide bar? — The slider’s pitch change rate per width is random for each quiz to prevent the user from memorizing the note according to slider position.