(UTC+09:00) Auditory-centered Vocal Feedback System Using Solmization for Training Absolute Pitch without GU I Nozomu Yoshida1, Kosaku Namikawa1, Yusuke Koroyasu1, Yoshiki Nagatani2, Yoichi Ochiai1, 2 1University of Tsukuba, 2Pixie Dust Technologies, Inc. #1232
by its note name — Solmization provides singers with vocal feedback of notes in the form of words 7 */5&3"$5 "VEJUPSZDFOUFSFE7PDBM'FFECBDL4ZTUFN6TJOH 4PMNJ[BUJPOGPS5SBJOJOH"CTPMVUF1JUDIXJUIPVU(6* “Solmization” *Vocal feedback refers to the feedback of information via the user’s hearing by speech
speech instruction Memory game with non-visual stimuli HCI Study with Vocal Feedback [1] Kuber, R., Tretter, M., Murphy, E.: Developing and evaluating a non-visual memory game. In: Campos, P., Graham, N., Jorge, J., Nunes, N., Palanque, P., Winckler, M. (eds.) Human-Computer Interaction – INTERACT 2011. pp. 541–553. Springer Berlin Heidelberg, Berlin, Heidelberg (2011) Vocal Feedback Vocal Feedback [2] Guardati, L., Casamassima, F., Farella, E., Benini, L.: Paper, pen and ink: An innovative system and software framework to assist writing rehabilitation. In: 2015 Design, Automation Test in Europe Conference Exhibition (DATE). pp. 1473–1478 (2015). https://doi.org/10.7873/DATE.2015.0453
4PMNJ[BUJPOGPS5SBJOJOH"CTPMVUF1JUDIXJUIPVU(6* Pitch training non-pitch training Visual Feedback Vocal Feedback Our research Writing assistance with speech Memory game with non-visual stimuli Quiz app Karaoke Tuner non-pitch training using visual feedback Categorize the training methods
system for improving absolute pitch via solmization Evaluated the effectiveness By comparing with visual feedback system Training system construction and evaluation
* Staff notation & note name Listen the exact correct pitch Move to next Bleep sound Visual Vocal Feedback * Note name with correct pitch Audio Audio Guess the exact correct pitch * Enter & Check the answer "VEJUPSZDFOUFSFE7PDBM'FFECBDL4ZTUFN6TJOH 4PMNJ[BUJPOGPS5SBJOJOH"CTPMVUF1JUDIXJUIPVU(6* */5&3"$5 Vocal Feedback Proposed system Visual Feedback Counterpart Pitch Test For the system-effectiveness-evaluation Implementation Vocal feedback training
* Staff notation & note name Listen the exact correct pitch Move to next Bleep sound Visual Vocal Feedback * Note name with correct pitch Audio Audio Guess the exact correct pitch * Enter & Check the answer "VEJUPSZDFOUFSFE7PDBM'FFECBDL4ZTUFN6TJOH 4PMNJ[BUJPOGPS5SBJOJOH"CTPMVUF1JUDIXJUIPVU(6* */5&3"$5 Vocal Feedback Proposed system Visual Feedback Counterpart Pitch Test For the system-effectiveness-evaluation Implementation Vocal feedback training
* Staff notation & note name Listen the exact correct pitch Move to next Bleep sound Visual Vocal Feedback * Note name with correct pitch Audio Audio Guess the exact correct pitch * Enter & Check the answer "VEJUPSZDFOUFSFE7PDBM'FFECBDL4ZTUFN6TJOH 4PMNJ[BUJPOGPS5SBJOJOH"CTPMVUF1JUDIXJUIPVU(6* */5&3"$5 Vocal Feedback Proposed system Visual Feedback Counterpart Pitch Test For the system-effectiveness-evaluation Implementation Vocal feedback training
* Staff notation & note name Listen the exact correct pitch Move to next Bleep sound Visual Vocal Feedback * Note name with correct pitch Audio Audio Guess the exact correct pitch * Enter & Check the answer "VEJUPSZDFOUFSFE7PDBM'FFECBDL4ZTUFN6TJOH 4PMNJ[BUJPOGPS5SBJOJOH"CTPMVUF1JUDIXJUIPVU(6* */5&3"$5 Vocal Feedback Proposed system Visual Feedback Counterpart Pitch Test For the system-effectiveness-evaluation Implementation Vocal feedback training
* Staff notation & note name Listen the exact correct pitch Move to next Bleep sound Visual Vocal Feedback * Note name with correct pitch Audio Audio Guess the exact correct pitch * Enter & Check the answer "VEJUPSZDFOUFSFE7PDBM'FFECBDL4ZTUFN6TJOH 4PMNJ[BUJPOGPS5SBJOJOH"CTPMVUF1JUDIXJUIPVU(6* */5&3"$5 Vocal Feedback Proposed system Visual Feedback Counterpart Pitch Test For the system-effectiveness-evaluation Implementation Vocal feedback training Oscillator Sound Vocal Feedback Bleep Re— System User *User makes sure that the pitch name and pitch are associated and memorize d Listen
* Staff notation & note name Listen the exact correct pitch Move to next Bleep sound Visual Vocal Feedback * Note name with correct pitch Audio Audio Guess the exact correct pitch * Enter & Check the answer "VEJUPSZDFOUFSFE7PDBM'FFECBDL4ZTUFN6TJOH 4PMNJ[BUJPOGPS5SBJOJOH"CTPMVUF1JUDIXJUIPVU(6* */5&3"$5 Vocal Feedback Proposed system Visual Feedback Counterpart Pitch Test For the system-effectiveness-evaluation Implementation Vocal feedback training
* Staff notation & note name Listen the exact correct pitch Move to next Bleep sound Visual Vocal Feedback * Note name with correct pitch Audio Audio Guess the exact correct pitch * Enter & Check the answer "VEJUPSZDFOUFSFE7PDBM'FFECBDL4ZTUFN6TJOH 4PMNJ[BUJPOGPS5SBJOJOH"CTPMVUF1JUDIXJUIPVU(6* */5&3"$5 Vocal Feedback Proposed system Visual Feedback Counterpart Pitch Test For the system-effectiveness-evaluation Implementation Vocal feedback training
* Staff notation & note name Listen the exact correct pitch Move to next Bleep sound Visual Vocal Feedback * Note name with correct pitch Audio Audio Guess the exact correct pitch * Enter & Check the answer "VEJUPSZDFOUFSFE7PDBM'FFECBDL4ZTUFN6TJOH 4PMNJ[BUJPOGPS5SBJOJOH"CTPMVUF1JUDIXJUIPVU(6* */5&3"$5 Vocal Feedback Proposed system Visual Feedback Counterpart Pitch Test For the system-effectiveness-evaluation Implementation Visual feedback training
* Staff notation & note name Listen the exact correct pitch Move to next Bleep sound Visual Vocal Feedback * Note name with correct pitch Audio Audio Guess the exact correct pitch * Enter & Check the answer "VEJUPSZDFOUFSFE7PDBM'FFECBDL4ZTUFN6TJOH 4PMNJ[BUJPOGPS5SBJOJOH"CTPMVUF1JUDIXJUIPVU(6* */5&3"$5 Vocal Feedback Proposed system Visual Feedback Counterpart Pitch Test For the system-effectiveness-evaluation Implementation Visual feedback training
* Staff notation & note name Listen the exact correct pitch Move to next Bleep sound Visual Vocal Feedback * Note name with correct pitch Audio Audio Guess the exact correct pitch * Enter & Check the answer "VEJUPSZDFOUFSFE7PDBM'FFECBDL4ZTUFN6TJOH 4PMNJ[BUJPOGPS5SBJOJOH"CTPMVUF1JUDIXJUIPVU(6* */5&3"$5 Vocal Feedback Proposed system Visual Feedback Counterpart Pitch Test For the system-effectiveness-evaluation Oscillator Sound Visual Feedback Bleep Staff notation & note name System User *User makes sure that the pitch name and pitch are associated and memorize d Listen See Implementation Visual feedback training
* Staff notation & note name Listen the exact correct pitch Move to next Bleep sound Visual Vocal Feedback * Note name with correct pitch Audio Audio Guess the exact correct pitch * Enter & Check the answer "VEJUPSZDFOUFSFE7PDBM'FFECBDL4ZTUFN6TJOH 4PMNJ[BUJPOGPS5SBJOJOH"CTPMVUF1JUDIXJUIPVU(6* */5&3"$5 Vocal Feedback Proposed system Visual Feedback Counterpart Pitch Test For the system-effectiveness-evaluation Implementation Absolute pitch test
* Staff notation & note name Listen the exact correct pitch Move to next Bleep sound Visual Vocal Feedback * Note name with correct pitch Audio Audio Guess the exact correct pitch * Enter & Check the answer "VEJUPSZDFOUFSFE7PDBM'FFECBDL4ZTUFN6TJOH 4PMNJ[BUJPOGPS5SBJOJOH"CTPMVUF1JUDIXJUIPVU(6* */5&3"$5 Vocal Feedback Proposed system Visual Feedback Counterpart Pitch Test For the system-effectiveness-evaluation Implementation Absolute pitch test
* Staff notation & note name Listen the exact correct pitch Move to next Bleep sound Visual Vocal Feedback * Note name with correct pitch Audio Audio Guess the exact correct pitch * Enter & Check the answer "VEJUPSZDFOUFSFE7PDBM'FFECBDL4ZTUFN6TJOH 4PMNJ[BUJPOGPS5SBJOJOH"CTPMVUF1JUDIXJUIPVU(6* */5&3"$5 Vocal Feedback Proposed system Visual Feedback Counterpart Pitch Test For the system-effectiveness-evaluation Oscillator Sound Bleep System User *User makes sure that the pitch name and pitch are associated and memorize d Listen Implementation Absolute pitch test
* Staff notation & note name Listen the exact correct pitch Move to next Bleep sound Visual Vocal Feedback * Note name with correct pitch Audio Audio Guess the exact correct pitch * Enter & Check the answer "VEJUPSZDFOUFSFE7PDBM'FFECBDL4ZTUFN6TJOH 4PMNJ[BUJPOGPS5SBJOJOH"CTPMVUF1JUDIXJUIPVU(6* */5&3"$5 Vocal Feedback Proposed system Visual Feedback Counterpart Pitch Test For the system-effectiveness-evaluation Implementation Absolute pitch test
* Staff notation & note name Listen the exact correct pitch Move to next Bleep sound Visual Vocal Feedback * Note name with correct pitch Audio Audio Guess the exact correct pitch * Enter & Check the answer "VEJUPSZDFOUFSFE7PDBM'FFECBDL4ZTUFN6TJOH 4PMNJ[BUJPOGPS5SBJOJOH"CTPMVUF1JUDIXJUIPVU(6* */5&3"$5 Vocal Feedback Proposed system Visual Feedback Counterpart Pitch Test For the system-effectiveness-evaluation Implementation Absolute pitch test
* Staff notation & note name Listen the exact correct pitch Move to next Bleep sound Visual Vocal Feedback * Note name with correct pitch Audio Audio Guess the exact correct pitch * Enter & Check the answer "VEJUPSZDFOUFSFE7PDBM'FFECBDL4ZTUFN6TJOH 4PMNJ[BUJPOGPS5SBJOJOH"CTPMVUF1JUDIXJUIPVU(6* */5&3"$5 Vocal Feedback Proposed system Visual Feedback Counterpart Pitch Test For the system-effectiveness-evaluation Implementation Absolute pitch test
participants Pre-test Main Test 1 5 people Absolute pitch qui z (10 question) C: Test Tas k (10 question) A: Training Tas k Vocal Feedbac k (5 min) C: Test Tas k (10 question) Interva l (5 - 10 min) C: Test Tas k (10 question) B: Training Tas k Visual Feedbac k (5 min) C: Test Tas k (10 question) 2 5 people B: Training Tas k Visual Feedbac k (5 min) A: Training Tas k Vocal Feedbac k (5 min) *We prepared two patterns to reduce possible bias caused by the training order. Patterns are randomly assigned to be half number. Start End *Additionally, questionnaires was conducted before and after the experiment to survey user’s attributes, the experiment process and the usability of the system .
showed signi fi cant improvement Improve value: –0.07 *Error means the difference between the correct pitch and user’s answer. → For example, if the correct pitch is C, but user answered it as C#, the error will be 1.0 . *The average error is the average difference between the correct pitch and input pitch from the 10 questions in test task C. *The improvement value refers to the difference between the average error before and after training. Result Main Test: Potential absolute pitch Error Error Participants ID
system showed signi fi cant improvement Improve value: –0.49 *Error means the difference between the correct pitch and user’s answer. → For example, if the correct pitch is C, but user answered it as C#, the error will be 1.0 . *The average error is the average difference between the correct pitch and input pitch from the 10 questions in test task C. *The improvement value refers to the difference between the average error before and after training. Result Main Test: Non-potential absolute pitch Error Error Participants ID
Wilcoxon signed-rank tes t H0: There would be no difference in the average error before and after training with the proposed/existing syste m H1: There would be a difference in the average error before and after training with the proposed/existing system. Signi fi cance only when non-potential AP with vocal feedback
4PMNJ[BUJPOGPS5SBJOJOH"CTPMVUF1JUDIXJUIPVU(6* The potential absolute pitch score before training was already high, so the score after training was unlikely to change. Small improvement value Small improvement value because
4PMNJ[BUJPOGPS5SBJOJOH"CTPMVUF1JUDIXJUIPVU(6* Non-GUI Application by limiting the system to auditory system only instead of slide bar on GUI Voice Instrument
4PMNJ[BUJPOGPS5SBJOJOH"CTPMVUF1JUDIXJUIPVU(6* Relative Pitch training https://hellomusictheory.com/learn/relative-pitch/ By giving the reference tone sound
experiment? 3. Could the participants keep the improvement after the experiment? 1. Which tools or frameworks did you used for system implementation? — In this research experiment, a GUI-based slide bar was used as the system input to unify the experimental conditions between vocal and visual feedback. — We don’t track the improvement after the experiment. It may require continuous training for keeping their pitch improvement. — MaxMSP for the application implementation. FAQ */5&3"$5 "VEJUPSZDFOUFSFE7PDBM'FFECBDL4ZTUFN6TJOH 4PMNJ[BUJPOGPS5SBJOJOH"CTPMVUF1JUDIXJUIPVU(6* 4. Could the participants memorized the location of the pitch in slide bar? — The slider’s pitch change rate per width is random for each quiz to prevent the user from memorizing the note according to slider position.
digitalnaturegroup
hakky
46ta
applepine1125
harukasakihara
nulabinc
dohara
mokocm
ido_kara_deru
hanaseleb
yoshiakiyamasaki
thatjeffsmith
stajima
phodgson
mojombo
holman
wjessup
jensimmons
samlambert
hursman
edds
akmur
eileencodes
davidbonilla
eitanlees
