Auditory-centered Vocal Feedback System Using Solmization for Training Absolute Pitch without GUI

Proceedings session: Interaction techniques
2021-09-01 @ 19:25 – 19:45 Asia/Tokyo (UTC+09:00)
Auditory-centered Vocal Feedback System Using
Solmization for Training Absolute Pitch without GU
I

Nozomu Yoshida1, Kosaku Namikawa1, Yusuke Koroyasu1, Yoshiki Nagatani2, Yoichi Ochiai1, 2
1University of Tsukuba, 2Pixie Dust Technologies, Inc.
#1232

View Slide

"VEJUPSZDFOUFSFE7PDBM'FFECBDL4ZTUFN6TJOH
 
4PMNJ[BUJPOGPS5SBJOJOH"CTPMVUF1JUDIXJUIPVU(6*

View Slide

“Absolute Pitch”
3
*/5&3"$5
 
— Ability to name a note of a given musical tone without an external reference ton
e

— They hear music as words rather than envisioning it as notes

View Slide

4
*/5&3"$5
 

View Slide

Most current pitch training system = Visual Feedback
5
*/5&3"$5
 
Visualized melody Pitch guessing quiz

View Slide

<
APs hear music as words
6
*/5&3"$5
 

View Slide

— A method of singing while calling out each sound by its note name
— Solmization provides singers with vocal feedback of notes in the form of words
7
*/5&3"$5
 
“Solmization”
*Vocal feedback refers to the feedback of information via the user’s hearing by speech

View Slide

8
Introduction Target in This Study */5&3"$5
 
Vocal feedback system doesn’t require a GUI
Ear Training Myu-Tre
Require a GUI
Pitch Quiz Karaoke Tuner
“Chromatic digital tuner” © Aurinko
Needless of GUI
Vocal Feedback
Live Dam Stadium DX-G

Do
Mi
Sol

View Slide

9
Related Work */5&3"$5
 
Writing assistance with speech instruction
Memory game with non-visual stimuli
HCI Study with Vocal Feedback
[1] Kuber, R., Tretter, M., Murphy, E.: Developing and evaluating a non-visual memory game. In: Campos, P., Graham, N., Jorge, J., Nunes, N., Palanque, P., Winckler, M.
(eds.) Human-Computer Interaction – INTERACT 2011. pp. 541–553. Springer Berlin Heidelberg, Berlin, Heidelberg (2011)
Vocal Feedback Vocal Feedback
[2] Guardati, L., Casamassima, F., Farella, E., Benini, L.: Paper, pen and ink: An innovative system and software framework to assist writing rehabilitation. In: 2015 Design,
Automation Test in Europe Conference Exhibition (DATE). pp. 1473–1478 (2015). https://doi.org/10.7873/DATE.2015.0453

View Slide

10
Related Works Position of this study */5&3"$5
 
Pitch training
non-pitch training
Visual 
Feedback
Vocal 
Feedback
Our research
Writing assistance with speech
Memory game with non-visual stimuli
Quiz app Karaoke Tuner
non-pitch training using visual feedback
Categorize the training methods

View Slide

11
Introduction Our contributions */5&3"$5
 
Constructed a system 
for improving absolute pitch via solmization
Evaluated the effectiveness 
By comparing with visual feedback system
Training system construction and evaluation

View Slide

12
Implementation System Design */5&3"$5
 
Test Generator
Oscillator
User input
Vocal Feedback
Display

View Slide

13
Imagine the pitch
Operate the slide bar
Visual
Feedback 
* Staff notation & note name
Listen the exact
correct pitch
Move to next
Bleep
sound
Visual
Vocal
Feedback 
* Note name with correct pitch
Audio Audio
Guess the exact
correct pitch  
* Enter & Check the answer
 
*/5&3"$5
Vocal Feedback 
Proposed system
Visual Feedback 
Counterpart
Pitch Test 
For the system-effectiveness-evaluation
Implementation Vocal feedback training

View Slide

14
Imagine the pitch
Visual
Feedback 
Listen the exact
correct pitch
Move to next
Bleep
sound
Visual
Vocal
Feedback 
Audio Audio
Guess the exact
correct pitch  
 
*/5&3"$5
Vocal Feedback 
Proposed system
Visual Feedback 
Counterpart
Pitch Test 

View Slide

15
Imagine the pitch
Visual
Feedback 
Listen the exact
correct pitch
Move to next
Bleep
sound
Visual
Vocal
Feedback 
Audio Audio
Guess the exact
correct pitch  
 
*/5&3"$5
Vocal Feedback 
Proposed system
Visual Feedback 
Counterpart
Pitch Test 

View Slide

16
Imagine the pitch
Visual
Feedback 
Listen the exact
correct pitch
Move to next
Bleep
sound
Visual
Vocal
Feedback 
Audio Audio
Guess the exact
correct pitch  
 
*/5&3"$5
Vocal Feedback 
Proposed system
Visual Feedback 
Counterpart
Pitch Test 

View Slide

17
Imagine the pitch
Visual
Feedback 
Listen the exact
correct pitch
Move to next
Bleep
sound
Visual
Vocal
Feedback 
Audio Audio
Guess the exact
correct pitch  
 
*/5&3"$5
Vocal Feedback 
Proposed system
Visual Feedback 
Counterpart
Pitch Test 
Oscillator Sound
Vocal Feedback
Bleep
Re—
System User
*User makes sure that the pitch name and pitch are associated and memorize
d

Listen

View Slide

18
Imagine the pitch
Visual
Feedback 
Listen the exact
correct pitch
Move to next
Bleep
sound
Visual
Vocal
Feedback 
Audio Audio
Guess the exact
correct pitch  
 
*/5&3"$5
Vocal Feedback 
Proposed system
Visual Feedback 
Counterpart
Pitch Test 

View Slide

19
Imagine the pitch
Visual
Feedback 
Listen the exact
correct pitch
Move to next
Bleep
sound
Visual
Vocal
Feedback 
Audio Audio
Guess the exact
correct pitch  
 
*/5&3"$5
Vocal Feedback 
Proposed system
Visual Feedback 
Counterpart
Pitch Test 

View Slide

20
Imagine the pitch
Visual
Feedback 
Listen the exact
correct pitch
Move to next
Bleep
sound
Visual
Vocal
Feedback 
Audio Audio
Guess the exact
correct pitch  
 
*/5&3"$5
Vocal Feedback 
Proposed system
Visual Feedback 
Counterpart
Pitch Test 
Implementation Visual feedback training

View Slide

21
Imagine the pitch
Visual
Feedback 
Listen the exact
correct pitch
Move to next
Bleep
sound
Visual
Vocal
Feedback 
Audio Audio
Guess the exact
correct pitch  
 
*/5&3"$5
Vocal Feedback 
Proposed system
Visual Feedback 
Counterpart
Pitch Test 

View Slide

22
Imagine the pitch
Visual
Feedback 
Listen the exact
correct pitch
Move to next
Bleep
sound
Visual
Vocal
Feedback 
Audio Audio
Guess the exact
correct pitch  
 
*/5&3"$5
Vocal Feedback 
Proposed system
Visual Feedback 
Counterpart
Pitch Test 
Oscillator Sound
Visual Feedback
Bleep
Staff notation & note name
System User
d

Listen
See

View Slide

23
Imagine the pitch
Visual
Feedback 
Listen the exact
correct pitch
Move to next
Bleep
sound
Visual
Vocal
Feedback 
Audio Audio
Guess the exact
correct pitch  
 
*/5&3"$5
Vocal Feedback 
Proposed system
Visual Feedback 
Counterpart
Pitch Test 
Implementation Absolute pitch test

View Slide

24
Imagine the pitch
Visual
Feedback 
Listen the exact
correct pitch
Move to next
Bleep
sound
Visual
Vocal
Feedback 
Audio Audio
Guess the exact
correct pitch  
 
*/5&3"$5
Vocal Feedback 
Proposed system
Visual Feedback 
Counterpart
Pitch Test 

View Slide

25
Imagine the pitch
Visual
Feedback 
Listen the exact
correct pitch
Move to next
Bleep
sound
Visual
Vocal
Feedback 
Audio Audio
Guess the exact
correct pitch  
 
*/5&3"$5
Vocal Feedback 
Proposed system
Visual Feedback 
Counterpart
Pitch Test 
Oscillator Sound
Bleep
System User
d

Listen

View Slide

26
Imagine the pitch
Visual
Feedback 
Listen the exact
correct pitch
Move to next
Bleep
sound
Visual
Vocal
Feedback 
Audio Audio
Guess the exact
correct pitch  
 
*/5&3"$5
Vocal Feedback 
Proposed system
Visual Feedback 
Counterpart
Pitch Test 

View Slide

27
Imagine the pitch
Visual
Feedback 
Listen the exact
correct pitch
Move to next
Bleep
sound
Visual
Vocal
Feedback 
Audio Audio
Guess the exact
correct pitch  
 
*/5&3"$5
Vocal Feedback 
Proposed system
Visual Feedback 
Counterpart
Pitch Test 

View Slide

28
Imagine the pitch
Visual
Feedback 
Listen the exact
correct pitch
Move to next
Bleep
sound
Visual
Vocal
Feedback 
Audio Audio
Guess the exact
correct pitch  
 
*/5&3"$5
Vocal Feedback 
Proposed system
Visual Feedback 
Counterpart
Pitch Test 

View Slide

29
Implementation Hardware set up */5&3"$5
 
System
User
Mouse
Headphone

View Slide

30
*/5&3"$5
 
Hypothesis
<
Visual
Feedback
Vocal
Feedback
in pitch training

View Slide

31
*/5&3"$5
 
Experiment Overview
Pattern
# of
participants
Pre-test Main Test
1
5 people
Absolute
pitch qui
z

(10 question)
C: 
Test Tas
k

(10 question)
A: 
Training Tas
k

Vocal Feedbac
k

(5 min)
C: 
Test Tas
k

(10 question)
Interva
l

(5 - 10 min)
C: 
Test Tas
k

(10 question)
B: 
Training Tas
k

Visual Feedbac
k

(5 min)
C: 
Test Tas
k

(10 question)
2
5 people
B: 
Training Tas
k

Visual Feedbac
k

(5 min)
A: 
Training Tas
k

Vocal Feedbac
k

(5 min)
*We prepared two patterns to reduce possible bias caused by the training order. Patterns are randomly assigned to be half number.
Start End
*Additionally, questionnaires was conducted before and after the experiment to survey user’s attributes, the experiment process and the usability of the system
.

View Slide

32
Experiment Preliminary Test */5&3"$5
 
For grouping purpose…
Perfect score (10/10)
Non-perfect score
Potential absolute pitch
Non-potential absolute pitch
Pitch quiz app  
Ear Training Myu-Tre: https://apps.apple.com/jp/app/id1015269208

View Slide

33
*/5&3"$5
Improve value: –0.09
Potential AP → Neither systems showed signi
fi
cant improvement
*Error means the difference between the correct pitch and user’s answer.

→ For example, if the correct pitch is C, but user answered it as C#, the error will be 1.0
.

*The average error is the average difference between the correct pitch and input pitch from the 10 questions in test task C.

*The improvement value refers to the difference between the average error before and after training.
Result Main Test: Potential absolute pitch
Error Error
Participants ID

View Slide

34
*/5&3"$5
Improve value: 0.65
Non-potential AP → Only proposed system showed signi
fi
cant improvement
*Error means the difference between the correct pitch and user’s answer.

→ For example, if the correct pitch is C, but user answered it as C#, the error will be 1.0
.

*The average error is the average difference between the correct pitch and input pitch from the 10 questions in test task C.

*The improvement value refers to the difference between the average error before and after training.
Result Main Test: Non-potential absolute pitch
Error Error
Participants ID

View Slide

35
Result Statistical Hypothesis Testing */5&3"$5
 
Method: Wilcoxon signed-rank tes
t

H0: There would be no difference in the average error before and after training with the proposed/existing syste
m

H1: There would be a difference in the average error before and after training with the proposed/existing system.
Signi
fi
cance only when non-potential AP with vocal feedback

View Slide

36
Discussion Difference between potential and non-potential participants */5&3"$5
 
The potential absolute pitch score before
training was already high, so the score
after training was unlikely to change.
Small improvement value
Small improvement value
because

View Slide

Discussion Self-evaluation of the pitch */5&3"$5
 
Little in
fl
uence on their self-evaluation of their absolute pitch

View Slide

38
Discussion User feedback */5&3"$5
 
80% prefer the vocal feedback system for long-term training

View Slide

39
Discussion improvements */5&3"$5
 
Do# → ‘Do sharp’
Visual feedback Vocal feedback
Redundancy of ‘sharp’ in vocal feedback

View Slide

40
Discussion improvements */5&3"$5
 
Do# → ‘Cis’
Visual feedback Vocal feedback
Redundancy of ‘sharp’ in vocal feedback
German note names

View Slide

41
Application Other examples of system input */5&3"$5
 
Non-GUI Application
by limiting the system to auditory system only instead of slide bar on GUI
Voice Instrument

View Slide

42
Application Other Examples of pitch training */5&3"$5
 
Relative Pitch training
https://hellomusictheory.com/learn/relative-pitch/
By giving the reference tone sound

View Slide

Auditory-centered Vocal Feedback System Using
Solmization for Training Absolute Pitch without GU
I

Nozomu Yoshida1, Kosaku Namikawa1, Yusuke Koroyasu1, Yoshiki Nagatani2, Yoichi Ochiai1, 2
1University of Tsukuba, 2Pixie Dust Technologies, Inc.
Proceedings session: Interaction techniques
2021-09-01 @ 19:25 – 19:45 Asia/Tokyo (UTC+09:00)
Contact: [email protected]
Thank you

View Slide

2. Why vocal feedback system adopted a GUI in the experiment?
3. Could the participants keep the improvement after the experiment?
1. Which tools or frameworks did you used for system implementation?
— In this research experiment, a GUI-based slide bar was used as the system input to
unify the experimental conditions between vocal and visual feedback.
— We don’t track the improvement after the experiment. It may require continuous
training for keeping their pitch improvement.
— MaxMSP for the application implementation.
FAQ */5&3"$5
 
4. Could the participants memorized the location of the pitch in slide bar?
— The slider’s pitch change rate per width is random for each quiz to prevent the user
from memorizing the note according to slider position.

View Slide

Auditory-centered Vocal Feedback System Using Solmization for Training Absolute Pitch without GUI

Auditory-centered Vocal Feedback System Using Solmization for Training Absolute Pitch without GUI

Digital Nature Group

More Decks by Digital Nature Group

Other Decks in Technology

Featured

Transcript