(UTC+09:00) Auditory-centered Vocal Feedback System Using Solmization for Training Absolute Pitch without GU I Nozomu Yoshida1, Kosaku Namikawa1, Yusuke Koroyasu1, Yoshiki Nagatani2, Yoichi Ochiai1, 2 1University of Tsukuba, 2Pixie Dust Technologies, Inc. #1232
by its note name — Solmization provides singers with vocal feedback of notes in the form of words 7 */5&3"$5 "VEJUPSZDFOUFSFE7PDBM'FFECBDL4ZTUFN6TJOH 4PMNJ[BUJPOGPS5SBJOJOH"CTPMVUF1JUDIXJUIPVU(6* “Solmization” *Vocal feedback refers to the feedback of information via the user’s hearing by speech
speech instruction Memory game with non-visual stimuli HCI Study with Vocal Feedback [1] Kuber, R., Tretter, M., Murphy, E.: Developing and evaluating a non-visual memory game. In: Campos, P., Graham, N., Jorge, J., Nunes, N., Palanque, P., Winckler, M. (eds.) Human-Computer Interaction – INTERACT 2011. pp. 541–553. Springer Berlin Heidelberg, Berlin, Heidelberg (2011) Vocal Feedback Vocal Feedback [2] Guardati, L., Casamassima, F., Farella, E., Benini, L.: Paper, pen and ink: An innovative system and software framework to assist writing rehabilitation. In: 2015 Design, Automation Test in Europe Conference Exhibition (DATE). pp. 1473–1478 (2015). https://doi.org/10.7873/DATE.2015.0453
4PMNJ[BUJPOGPS5SBJOJOH"CTPMVUF1JUDIXJUIPVU(6* Pitch training non-pitch training Visual Feedback Vocal Feedback Our research Writing assistance with speech Memory game with non-visual stimuli Quiz app Karaoke Tuner non-pitch training using visual feedback Categorize the training methods
system for improving absolute pitch via solmization Evaluated the effectiveness By comparing with visual feedback system Training system construction and evaluation
* Staff notation & note name Listen the exact correct pitch Move to next Bleep sound Visual Vocal Feedback * Note name with correct pitch Audio Audio Guess the exact correct pitch * Enter & Check the answer "VEJUPSZDFOUFSFE7PDBM'FFECBDL4ZTUFN6TJOH 4PMNJ[BUJPOGPS5SBJOJOH"CTPMVUF1JUDIXJUIPVU(6* */5&3"$5 Vocal Feedback Proposed system Visual Feedback Counterpart Pitch Test For the system-effectiveness-evaluation Implementation Vocal feedback training
* Staff notation & note name Listen the exact correct pitch Move to next Bleep sound Visual Vocal Feedback * Note name with correct pitch Audio Audio Guess the exact correct pitch * Enter & Check the answer "VEJUPSZDFOUFSFE7PDBM'FFECBDL4ZTUFN6TJOH 4PMNJ[BUJPOGPS5SBJOJOH"CTPMVUF1JUDIXJUIPVU(6* */5&3"$5 Vocal Feedback Proposed system Visual Feedback Counterpart Pitch Test For the system-effectiveness-evaluation Implementation Vocal feedback training
* Staff notation & note name Listen the exact correct pitch Move to next Bleep sound Visual Vocal Feedback * Note name with correct pitch Audio Audio Guess the exact correct pitch * Enter & Check the answer "VEJUPSZDFOUFSFE7PDBM'FFECBDL4ZTUFN6TJOH 4PMNJ[BUJPOGPS5SBJOJOH"CTPMVUF1JUDIXJUIPVU(6* */5&3"$5 Vocal Feedback Proposed system Visual Feedback Counterpart Pitch Test For the system-effectiveness-evaluation Implementation Vocal feedback training
* Staff notation & note name Listen the exact correct pitch Move to next Bleep sound Visual Vocal Feedback * Note name with correct pitch Audio Audio Guess the exact correct pitch * Enter & Check the answer "VEJUPSZDFOUFSFE7PDBM'FFECBDL4ZTUFN6TJOH 4PMNJ[BUJPOGPS5SBJOJOH"CTPMVUF1JUDIXJUIPVU(6* */5&3"$5 Vocal Feedback Proposed system Visual Feedback Counterpart Pitch Test For the system-effectiveness-evaluation Implementation Vocal feedback training
* Staff notation & note name Listen the exact correct pitch Move to next Bleep sound Visual Vocal Feedback * Note name with correct pitch Audio Audio Guess the exact correct pitch * Enter & Check the answer "VEJUPSZDFOUFSFE7PDBM'FFECBDL4ZTUFN6TJOH 4PMNJ[BUJPOGPS5SBJOJOH"CTPMVUF1JUDIXJUIPVU(6* */5&3"$5 Vocal Feedback Proposed system Visual Feedback Counterpart Pitch Test For the system-effectiveness-evaluation Implementation Vocal feedback training Oscillator Sound Vocal Feedback Bleep Re— System User *User makes sure that the pitch name and pitch are associated and memorize d Listen
* Staff notation & note name Listen the exact correct pitch Move to next Bleep sound Visual Vocal Feedback * Note name with correct pitch Audio Audio Guess the exact correct pitch * Enter & Check the answer "VEJUPSZDFOUFSFE7PDBM'FFECBDL4ZTUFN6TJOH 4PMNJ[BUJPOGPS5SBJOJOH"CTPMVUF1JUDIXJUIPVU(6* */5&3"$5 Vocal Feedback Proposed system Visual Feedback Counterpart Pitch Test For the system-effectiveness-evaluation Implementation Vocal feedback training
* Staff notation & note name Listen the exact correct pitch Move to next Bleep sound Visual Vocal Feedback * Note name with correct pitch Audio Audio Guess the exact correct pitch * Enter & Check the answer "VEJUPSZDFOUFSFE7PDBM'FFECBDL4ZTUFN6TJOH 4PMNJ[BUJPOGPS5SBJOJOH"CTPMVUF1JUDIXJUIPVU(6* */5&3"$5 Vocal Feedback Proposed system Visual Feedback Counterpart Pitch Test For the system-effectiveness-evaluation Implementation Vocal feedback training
* Staff notation & note name Listen the exact correct pitch Move to next Bleep sound Visual Vocal Feedback * Note name with correct pitch Audio Audio Guess the exact correct pitch * Enter & Check the answer "VEJUPSZDFOUFSFE7PDBM'FFECBDL4ZTUFN6TJOH 4PMNJ[BUJPOGPS5SBJOJOH"CTPMVUF1JUDIXJUIPVU(6* */5&3"$5 Vocal Feedback Proposed system Visual Feedback Counterpart Pitch Test For the system-effectiveness-evaluation Implementation Visual feedback training
* Staff notation & note name Listen the exact correct pitch Move to next Bleep sound Visual Vocal Feedback * Note name with correct pitch Audio Audio Guess the exact correct pitch * Enter & Check the answer "VEJUPSZDFOUFSFE7PDBM'FFECBDL4ZTUFN6TJOH 4PMNJ[BUJPOGPS5SBJOJOH"CTPMVUF1JUDIXJUIPVU(6* */5&3"$5 Vocal Feedback Proposed system Visual Feedback Counterpart Pitch Test For the system-effectiveness-evaluation Implementation Visual feedback training
* Staff notation & note name Listen the exact correct pitch Move to next Bleep sound Visual Vocal Feedback * Note name with correct pitch Audio Audio Guess the exact correct pitch * Enter & Check the answer "VEJUPSZDFOUFSFE7PDBM'FFECBDL4ZTUFN6TJOH 4PMNJ[BUJPOGPS5SBJOJOH"CTPMVUF1JUDIXJUIPVU(6* */5&3"$5 Vocal Feedback Proposed system Visual Feedback Counterpart Pitch Test For the system-effectiveness-evaluation Oscillator Sound Visual Feedback Bleep Staff notation & note name System User *User makes sure that the pitch name and pitch are associated and memorize d Listen See Implementation Visual feedback training
* Staff notation & note name Listen the exact correct pitch Move to next Bleep sound Visual Vocal Feedback * Note name with correct pitch Audio Audio Guess the exact correct pitch * Enter & Check the answer "VEJUPSZDFOUFSFE7PDBM'FFECBDL4ZTUFN6TJOH 4PMNJ[BUJPOGPS5SBJOJOH"CTPMVUF1JUDIXJUIPVU(6* */5&3"$5 Vocal Feedback Proposed system Visual Feedback Counterpart Pitch Test For the system-effectiveness-evaluation Implementation Absolute pitch test
* Staff notation & note name Listen the exact correct pitch Move to next Bleep sound Visual Vocal Feedback * Note name with correct pitch Audio Audio Guess the exact correct pitch * Enter & Check the answer "VEJUPSZDFOUFSFE7PDBM'FFECBDL4ZTUFN6TJOH 4PMNJ[BUJPOGPS5SBJOJOH"CTPMVUF1JUDIXJUIPVU(6* */5&3"$5 Vocal Feedback Proposed system Visual Feedback Counterpart Pitch Test For the system-effectiveness-evaluation Implementation Absolute pitch test
* Staff notation & note name Listen the exact correct pitch Move to next Bleep sound Visual Vocal Feedback * Note name with correct pitch Audio Audio Guess the exact correct pitch * Enter & Check the answer "VEJUPSZDFOUFSFE7PDBM'FFECBDL4ZTUFN6TJOH 4PMNJ[BUJPOGPS5SBJOJOH"CTPMVUF1JUDIXJUIPVU(6* */5&3"$5 Vocal Feedback Proposed system Visual Feedback Counterpart Pitch Test For the system-effectiveness-evaluation Oscillator Sound Bleep System User *User makes sure that the pitch name and pitch are associated and memorize d Listen Implementation Absolute pitch test
* Staff notation & note name Listen the exact correct pitch Move to next Bleep sound Visual Vocal Feedback * Note name with correct pitch Audio Audio Guess the exact correct pitch * Enter & Check the answer "VEJUPSZDFOUFSFE7PDBM'FFECBDL4ZTUFN6TJOH 4PMNJ[BUJPOGPS5SBJOJOH"CTPMVUF1JUDIXJUIPVU(6* */5&3"$5 Vocal Feedback Proposed system Visual Feedback Counterpart Pitch Test For the system-effectiveness-evaluation Implementation Absolute pitch test
* Staff notation & note name Listen the exact correct pitch Move to next Bleep sound Visual Vocal Feedback * Note name with correct pitch Audio Audio Guess the exact correct pitch * Enter & Check the answer "VEJUPSZDFOUFSFE7PDBM'FFECBDL4ZTUFN6TJOH 4PMNJ[BUJPOGPS5SBJOJOH"CTPMVUF1JUDIXJUIPVU(6* */5&3"$5 Vocal Feedback Proposed system Visual Feedback Counterpart Pitch Test For the system-effectiveness-evaluation Implementation Absolute pitch test
* Staff notation & note name Listen the exact correct pitch Move to next Bleep sound Visual Vocal Feedback * Note name with correct pitch Audio Audio Guess the exact correct pitch * Enter & Check the answer "VEJUPSZDFOUFSFE7PDBM'FFECBDL4ZTUFN6TJOH 4PMNJ[BUJPOGPS5SBJOJOH"CTPMVUF1JUDIXJUIPVU(6* */5&3"$5 Vocal Feedback Proposed system Visual Feedback Counterpart Pitch Test For the system-effectiveness-evaluation Implementation Absolute pitch test
participants Pre-test Main Test 1 5 people Absolute pitch qui z (10 question) C: Test Tas k (10 question) A: Training Tas k Vocal Feedbac k (5 min) C: Test Tas k (10 question) Interva l (5 - 10 min) C: Test Tas k (10 question) B: Training Tas k Visual Feedbac k (5 min) C: Test Tas k (10 question) 2 5 people B: Training Tas k Visual Feedbac k (5 min) A: Training Tas k Vocal Feedbac k (5 min) *We prepared two patterns to reduce possible bias caused by the training order. Patterns are randomly assigned to be half number. Start End *Additionally, questionnaires was conducted before and after the experiment to survey user’s attributes, the experiment process and the usability of the system .
showed signi fi cant improvement Improve value: –0.07 *Error means the difference between the correct pitch and user’s answer. → For example, if the correct pitch is C, but user answered it as C#, the error will be 1.0 . *The average error is the average difference between the correct pitch and input pitch from the 10 questions in test task C. *The improvement value refers to the difference between the average error before and after training. Result Main Test: Potential absolute pitch Error Error Participants ID
system showed signi fi cant improvement Improve value: –0.49 *Error means the difference between the correct pitch and user’s answer. → For example, if the correct pitch is C, but user answered it as C#, the error will be 1.0 . *The average error is the average difference between the correct pitch and input pitch from the 10 questions in test task C. *The improvement value refers to the difference between the average error before and after training. Result Main Test: Non-potential absolute pitch Error Error Participants ID
Wilcoxon signed-rank tes t H0: There would be no difference in the average error before and after training with the proposed/existing syste m H1: There would be a difference in the average error before and after training with the proposed/existing system. Signi fi cance only when non-potential AP with vocal feedback
4PMNJ[BUJPOGPS5SBJOJOH"CTPMVUF1JUDIXJUIPVU(6* The potential absolute pitch score before training was already high, so the score after training was unlikely to change. Small improvement value Small improvement value because
4PMNJ[BUJPOGPS5SBJOJOH"CTPMVUF1JUDIXJUIPVU(6* Non-GUI Application by limiting the system to auditory system only instead of slide bar on GUI Voice Instrument
4PMNJ[BUJPOGPS5SBJOJOH"CTPMVUF1JUDIXJUIPVU(6* Relative Pitch training https://hellomusictheory.com/learn/relative-pitch/ By giving the reference tone sound
experiment? 3. Could the participants keep the improvement after the experiment? 1. Which tools or frameworks did you used for system implementation? — In this research experiment, a GUI-based slide bar was used as the system input to unify the experimental conditions between vocal and visual feedback. — We don’t track the improvement after the experiment. It may require continuous training for keeping their pitch improvement. — MaxMSP for the application implementation. FAQ */5&3"$5 "VEJUPSZDFOUFSFE7PDBM'FFECBDL4ZTUFN6TJOH 4PMNJ[BUJPOGPS5SBJOJOH"CTPMVUF1JUDIXJUIPVU(6* 4. Could the participants memorized the location of the pitch in slide bar? — The slider’s pitch change rate per width is random for each quiz to prevent the user from memorizing the note according to slider position.