Auditory-centered Vocal Feedback System Using Solmization for Training Absolute Pitch without GUI

Proceedings session: Interaction techniques 2021-09-01 @ 19:25 – 19:45 Asia/Tokyo
(UTC+09:00) Auditory-centered Vocal Feedback System Using Solmization for Training Absolute Pitch without GU I Nozomu Yoshida1, Kosaku Namikawa1, Yusuke Koroyasu1, Yoshiki Nagatani2, Yoichi Ochiai1, 2 1University of Tsukuba, 2Pixie Dust Technologies, Inc. #1232

"VEJUPSZDFOUFSFE7PDBM'FFECBDL4ZTUFN6TJOH   4PMNJ[BUJPOGPS5SBJOJOH"CTPMVUF1JUDIXJUIPVU(6*

“Absolute Pitch” 3 */5&3"$5 "VEJUPSZDFOUFSFE7PDBM'FFECBDL4ZTUFN6TJOH   4PMNJ[BUJPOGPS5SBJOJOH"CTPMVUF1JUDIXJUIPVU(6* — Ability to
name a note of a given musical tone without an external reference ton e — They hear music as words rather than envisioning it as notes

4 */5&3"$5 "VEJUPSZDFOUFSFE7PDBM'FFECBDL4ZTUFN6TJOH   4PMNJ[BUJPOGPS5SBJOJOH"CTPMVUF1JUDIXJUIPVU(6*

Most current pitch training system = Visual Feedback 5 */5&3"$5
"VEJUPSZDFOUFSFE7PDBM'FFECBDL4ZTUFN6TJOH   4PMNJ[BUJPOGPS5SBJOJOH"CTPMVUF1JUDIXJUIPVU(6* Visualized melody Pitch guessing quiz

< APs hear music as words 6 */5&3"$5 "VEJUPSZDFOUFSFE7PDBM'FFECBDL4ZTUFN6TJOH  
4PMNJ[BUJPOGPS5SBJOJOH"CTPMVUF1JUDIXJUIPVU(6*

— A method of singing while calling out each sound
by its note name — Solmization provides singers with vocal feedback of notes in the form of words 7 */5&3"$5 "VEJUPSZDFOUFSFE7PDBM'FFECBDL4ZTUFN6TJOH   4PMNJ[BUJPOGPS5SBJOJOH"CTPMVUF1JUDIXJUIPVU(6* “Solmization” *Vocal feedback refers to the feedback of information via the user’s hearing by speech

8 Introduction Target in This Study */5&3"$5 "VEJUPSZDFOUFSFE7PDBM'FFECBDL4ZTUFN6TJOH   4PMNJ[BUJPOGPS5SBJOJOH"CTPMVUF1JUDIXJUIPVU(6*
Vocal feedback system doesn’t require a GUI Ear Training Myu-Tre Require a GUI Pitch Quiz Karaoke Tuner “Chromatic digital tuner” © Aurinko Needless of GUI Vocal Feedback Live Dam Stadium DX-G Do Mi Sol

9 Related Work */5&3"$5 "VEJUPSZDFOUFSFE7PDBM'FFECBDL4ZTUFN6TJOH   4PMNJ[BUJPOGPS5SBJOJOH"CTPMVUF1JUDIXJUIPVU(6* Writing assistance with
speech instruction Memory game with non-visual stimuli HCI Study with Vocal Feedback [1] Kuber, R., Tretter, M., Murphy, E.: Developing and evaluating a non-visual memory game. In: Campos, P., Graham, N., Jorge, J., Nunes, N., Palanque, P., Winckler, M. (eds.) Human-Computer Interaction – INTERACT 2011. pp. 541–553. Springer Berlin Heidelberg, Berlin, Heidelberg (2011) Vocal Feedback Vocal Feedback [2] Guardati, L., Casamassima, F., Farella, E., Benini, L.: Paper, pen and ink: An innovative system and software framework to assist writing rehabilitation. In: 2015 Design, Automation Test in Europe Conference Exhibition (DATE). pp. 1473–1478 (2015). https://doi.org/10.7873/DATE.2015.0453

10 Related Works Position of this study */5&3"$5 "VEJUPSZDFOUFSFE7PDBM'FFECBDL4ZTUFN6TJOH  
4PMNJ[BUJPOGPS5SBJOJOH"CTPMVUF1JUDIXJUIPVU(6* Pitch training non-pitch training Visual  Feedback Vocal  Feedback Our research Writing assistance with speech Memory game with non-visual stimuli Quiz app Karaoke Tuner non-pitch training using visual feedback Categorize the training methods

11 Introduction Our contributions */5&3"$5 "VEJUPSZDFOUFSFE7PDBM'FFECBDL4ZTUFN6TJOH   4PMNJ[BUJPOGPS5SBJOJOH"CTPMVUF1JUDIXJUIPVU(6* Constructed a
system  for improving absolute pitch via solmization Evaluated the effectiveness  By comparing with visual feedback system Training system construction and evaluation

12 Implementation System Design */5&3"$5 "VEJUPSZDFOUFSFE7PDBM'FFECBDL4ZTUFN6TJOH   4PMNJ[BUJPOGPS5SBJOJOH"CTPMVUF1JUDIXJUIPVU(6* Test Generator
Oscillator User input Vocal Feedback Display

13 Imagine the pitch Operate the slide bar Visual Feedback 
* Staff notation & note name Listen the exact correct pitch Move to next Bleep sound Visual Vocal Feedback  * Note name with correct pitch Audio Audio Guess the exact correct pitch   * Enter & Check the answer "VEJUPSZDFOUFSFE7PDBM'FFECBDL4ZTUFN6TJOH   4PMNJ[BUJPOGPS5SBJOJOH"CTPMVUF1JUDIXJUIPVU(6* */5&3"$5 Vocal Feedback  Proposed system Visual Feedback  Counterpart Pitch Test  For the system-effectiveness-evaluation Implementation Vocal feedback training

* Staff notation & note name Listen the exact correct pitch Move to next Bleep sound Visual Vocal Feedback  * Note name with correct pitch Audio Audio Guess the exact correct pitch   * Enter & Check the answer "VEJUPSZDFOUFSFE7PDBM'FFECBDL4ZTUFN6TJOH   4PMNJ[BUJPOGPS5SBJOJOH"CTPMVUF1JUDIXJUIPVU(6* */5&3"$5 Vocal Feedback  Proposed system Visual Feedback  Counterpart Pitch Test  For the system-effectiveness-evaluation Implementation Vocal feedback training Oscillator Sound Vocal Feedback Bleep Re— System User *User makes sure that the pitch name and pitch are associated and memorize d Listen

* Staff notation & note name Listen the exact correct pitch Move to next Bleep sound Visual Vocal Feedback  * Note name with correct pitch Audio Audio Guess the exact correct pitch   * Enter & Check the answer "VEJUPSZDFOUFSFE7PDBM'FFECBDL4ZTUFN6TJOH   4PMNJ[BUJPOGPS5SBJOJOH"CTPMVUF1JUDIXJUIPVU(6* */5&3"$5 Vocal Feedback  Proposed system Visual Feedback  Counterpart Pitch Test  For the system-effectiveness-evaluation Implementation Visual feedback training

* Staff notation & note name Listen the exact correct pitch Move to next Bleep sound Visual Vocal Feedback  * Note name with correct pitch Audio Audio Guess the exact correct pitch   * Enter & Check the answer "VEJUPSZDFOUFSFE7PDBM'FFECBDL4ZTUFN6TJOH   4PMNJ[BUJPOGPS5SBJOJOH"CTPMVUF1JUDIXJUIPVU(6* */5&3"$5 Vocal Feedback  Proposed system Visual Feedback  Counterpart Pitch Test  For the system-effectiveness-evaluation Oscillator Sound Visual Feedback Bleep Staff notation & note name System User *User makes sure that the pitch name and pitch are associated and memorize d Listen See Implementation Visual feedback training

* Staff notation & note name Listen the exact correct pitch Move to next Bleep sound Visual Vocal Feedback  * Note name with correct pitch Audio Audio Guess the exact correct pitch   * Enter & Check the answer "VEJUPSZDFOUFSFE7PDBM'FFECBDL4ZTUFN6TJOH   4PMNJ[BUJPOGPS5SBJOJOH"CTPMVUF1JUDIXJUIPVU(6* */5&3"$5 Vocal Feedback  Proposed system Visual Feedback  Counterpart Pitch Test  For the system-effectiveness-evaluation Implementation Absolute pitch test

* Staff notation & note name Listen the exact correct pitch Move to next Bleep sound Visual Vocal Feedback  * Note name with correct pitch Audio Audio Guess the exact correct pitch   * Enter & Check the answer "VEJUPSZDFOUFSFE7PDBM'FFECBDL4ZTUFN6TJOH   4PMNJ[BUJPOGPS5SBJOJOH"CTPMVUF1JUDIXJUIPVU(6* */5&3"$5 Vocal Feedback  Proposed system Visual Feedback  Counterpart Pitch Test  For the system-effectiveness-evaluation Oscillator Sound Bleep System User *User makes sure that the pitch name and pitch are associated and memorize d Listen Implementation Absolute pitch test

29 Implementation Hardware set up */5&3"$5 "VEJUPSZDFOUFSFE7PDBM'FFECBDL4ZTUFN6TJOH   4PMNJ[BUJPOGPS5SBJOJOH"CTPMVUF1JUDIXJUIPVU(6* System
User Mouse Headphone

30 */5&3"$5 "VEJUPSZDFOUFSFE7PDBM'FFECBDL4ZTUFN6TJOH   4PMNJ[BUJPOGPS5SBJOJOH"CTPMVUF1JUDIXJUIPVU(6* Hypothesis < Visual Feedback Vocal
Feedback in pitch training

31 */5&3"$5 "VEJUPSZDFOUFSFE7PDBM'FFECBDL4ZTUFN6TJOH   4PMNJ[BUJPOGPS5SBJOJOH"CTPMVUF1JUDIXJUIPVU(6* Experiment Overview Pattern # of
participants Pre-test Main Test 1 5 people Absolute pitch qui z (10 question) C:  Test Tas k (10 question) A:  Training Tas k Vocal Feedbac k (5 min) C:  Test Tas k (10 question) Interva l (5 - 10 min) C:  Test Tas k (10 question) B:  Training Tas k Visual Feedbac k (5 min) C:  Test Tas k (10 question) 2 5 people B:  Training Tas k Visual Feedbac k (5 min) A:  Training Tas k Vocal Feedbac k (5 min) *We prepared two patterns to reduce possible bias caused by the training order. Patterns are randomly assigned to be half number. Start End *Additionally, questionnaires was conducted before and after the experiment to survey user’s attributes, the experiment process and the usability of the system .

32 Experiment Preliminary Test */5&3"$5 "VEJUPSZDFOUFSFE7PDBM'FFECBDL4ZTUFN6TJOH   4PMNJ[BUJPOGPS5SBJOJOH"CTPMVUF1JUDIXJUIPVU(6* For grouping
purpose… Perfect score (10/10) Non-perfect score Potential absolute pitch Non-potential absolute pitch Pitch quiz app   Ear Training Myu-Tre: https://apps.apple.com/jp/app/id1015269208

33 */5&3"$5 Improve value: –0.09 Potential AP → Neither systems
showed signi fi cant improvement Improve value: –0.07 *Error means the difference between the correct pitch and user’s answer. → For example, if the correct pitch is C, but user answered it as C#, the error will be 1.0 . *The average error is the average difference between the correct pitch and input pitch from the 10 questions in test task C. *The improvement value refers to the difference between the average error before and after training. Result Main Test: Potential absolute pitch Error Error Participants ID

34 */5&3"$5 Improve value: 0.65 Non-potential AP → Only proposed
system showed signi fi cant improvement Improve value: –0.49 *Error means the difference between the correct pitch and user’s answer. → For example, if the correct pitch is C, but user answered it as C#, the error will be 1.0 . *The average error is the average difference between the correct pitch and input pitch from the 10 questions in test task C. *The improvement value refers to the difference between the average error before and after training. Result Main Test: Non-potential absolute pitch Error Error Participants ID

35 Result Statistical Hypothesis Testing */5&3"$5 "VEJUPSZDFOUFSFE7PDBM'FFECBDL4ZTUFN6TJOH   4PMNJ[BUJPOGPS5SBJOJOH"CTPMVUF1JUDIXJUIPVU(6* Method:
Wilcoxon signed-rank tes t H0: There would be no difference in the average error before and after training with the proposed/existing syste m H1: There would be a difference in the average error before and after training with the proposed/existing system. Signi fi cance only when non-potential AP with vocal feedback

36 Discussion Difference between potential and non-potential participants */5&3"$5 "VEJUPSZDFOUFSFE7PDBM'FFECBDL4ZTUFN6TJOH
  4PMNJ[BUJPOGPS5SBJOJOH"CTPMVUF1JUDIXJUIPVU(6* The potential absolute pitch score before training was already high, so the score after training was unlikely to change. Small improvement value Small improvement value because

Discussion Self-evaluation of the pitch */5&3"$5 "VEJUPSZDFOUFSFE7PDBM'FFECBDL4ZTUFN6TJOH   4PMNJ[BUJPOGPS5SBJOJOH"CTPMVUF1JUDIXJUIPVU(6* Little
in fl uence on their self-evaluation of their absolute pitch

38 Discussion User feedback */5&3"$5 "VEJUPSZDFOUFSFE7PDBM'FFECBDL4ZTUFN6TJOH   4PMNJ[BUJPOGPS5SBJOJOH"CTPMVUF1JUDIXJUIPVU(6* 80% prefer
the vocal feedback system for long-term training

39 Discussion improvements */5&3"$5 "VEJUPSZDFOUFSFE7PDBM'FFECBDL4ZTUFN6TJOH   4PMNJ[BUJPOGPS5SBJOJOH"CTPMVUF1JUDIXJUIPVU(6* Do# → ‘Do
sharp’ Visual feedback Vocal feedback Redundancy of ‘sharp’ in vocal feedback

40 Discussion improvements */5&3"$5 "VEJUPSZDFOUFSFE7PDBM'FFECBDL4ZTUFN6TJOH   4PMNJ[BUJPOGPS5SBJOJOH"CTPMVUF1JUDIXJUIPVU(6* Do# → ‘Cis’
Visual feedback Vocal feedback Redundancy of ‘sharp’ in vocal feedback German note names

41 Application Other examples of system input */5&3"$5 "VEJUPSZDFOUFSFE7PDBM'FFECBDL4ZTUFN6TJOH  
4PMNJ[BUJPOGPS5SBJOJOH"CTPMVUF1JUDIXJUIPVU(6* Non-GUI Application by limiting the system to auditory system only instead of slide bar on GUI Voice Instrument

42 Application Other Examples of pitch training */5&3"$5 "VEJUPSZDFOUFSFE7PDBM'FFECBDL4ZTUFN6TJOH  
4PMNJ[BUJPOGPS5SBJOJOH"CTPMVUF1JUDIXJUIPVU(6* Relative Pitch training https://hellomusictheory.com/learn/relative-pitch/ By giving the reference tone sound

Auditory-centered Vocal Feedback System Using Solmization for Training Absolute Pitch
without GU I Nozomu Yoshida1, Kosaku Namikawa1, Yusuke Koroyasu1, Yoshiki Nagatani2, Yoichi Ochiai1, 2 1University of Tsukuba, 2Pixie Dust Technologies, Inc. Proceedings session: Interaction techniques 2021-09-01 @ 19:25 – 19:45 Asia/Tokyo (UTC+09:00) Contact: [email protected] Thank you

2. Why vocal feedback system adopted a GUI in the
experiment? 3. Could the participants keep the improvement after the experiment? 1. Which tools or frameworks did you used for system implementation? — In this research experiment, a GUI-based slide bar was used as the system input to unify the experimental conditions between vocal and visual feedback. — We don’t track the improvement after the experiment. It may require continuous training for keeping their pitch improvement. — MaxMSP for the application implementation. FAQ */5&3"$5 "VEJUPSZDFOUFSFE7PDBM'FFECBDL4ZTUFN6TJOH   4PMNJ[BUJPOGPS5SBJOJOH"CTPMVUF1JUDIXJUIPVU(6* 4. Could the participants memorized the location of the pitch in slide bar? — The slider’s pitch change rate per width is random for each quiz to prevent the user from memorizing the note according to slider position.

Auditory-centered Vocal Feedback System Using S...

Auditory-centered Vocal Feedback System Using Solmization for Training Absolute Pitch without GUI

More Decks by Digital Nature Group

Other Decks in Technology

Featured

Transcript