VAE-based Phoneme Alignment Using Gradient Annealing and SSL Acoustic Features

Transcript

1. VAE-based Phoneme Alignment Using Gradient Annealing and SSL Acoustic Features Interspeech 2024, A5-O7.2, Sept. 4, 2024 Tomoki Koriyama (CyberAgent AI Lab)
2. VAE-based Phoneme Alignment Using Gradient Annealing and SSL Acoustic Features Abstract • Task: phoneme alignment for speech analysis and modeling • Propose a novel alignment model that extends One TTS Alignment (OTA) ◦ • Incorporate VAE, SSL acoustic input, sub-phone-level input, and gradient annealing The proposed model predicted more accurate phoneme boundaries than conventional OTA, Montreal forced aligner, and CTC segmentation 2
3. VAE-based Phoneme Alignment Using Gradient Annealing and SSL Acoustic Features Background • Phoneme alignment ◦ • Assign which phonemes are uttered at which time Applications ◦ Speech analysis ▪ ◦ Extract phoneme- / word-level acoustic features Speech synthesis modeling ▪ ▪ ▪ Represent the relationship between a text and a speech waveform Train phoneme duration model Help fast convergence in training 3
4. VAE-based Phoneme Alignment Using Gradient Annealing and SSL Acoustic Features Purpose • Enhance the phoneme alignment accuracy • Issues to be solved ◦ High accuracy ▪ ◦ Gradient-based training ▪ ◦ i.e., predicted boundaries should be close to manually annotated ones Can train simultaneously with other models Unsupervised training ▪ ▪ We do not have ”true” boundary data for supervised training Needs to be trained only with speech and phoneme (text) sequence w/o the annotated boundary data so that it can be applied to diverse languages and speaking styles 4
5. VAE-based Phoneme Alignment Using Gradient Annealing and SSL Acoustic Features Conventional methods • GMM-HMM ◦ ◦ ◦ Traditional method Many tools (ex. Montreal forced aligner (MFA)) Constraints on fixed acoustic and linguistic features ▪ • Triphone, MFCC, etc. CTC segmentation [Kürzinger20] ◦ ◦ ◦ Use CTC-trained ASR models directly Recently used in torchaudio, WhisperX Insufficient alignment accuracy ▪ ▪ CTC loss does not take time information into account Very often, we observe one-frame phonemes (characters) 5
6. VAE-based Phoneme Alignment Using Gradient Annealing and SSL Acoustic Features Conventional method: One TTS Alignment (1) • One TTS Alignment (OTA) to rule them all [Shih19, Badlani22] ◦ • Proposed for fast convergence in TTS training Method ◦ ◦ ◦ Encode both acoustic and linguistic embeddings The similarity between embeddings is used for alignment Encoders are trained using ForwardSum loss Acoustic feature sequence Alignment Loss ForwardSum Alignment path Acoustic embedding Linguistic embedding Acoustic encoder Linguistic encoder Acoustic feature sequence Linguistic feature sequence TTS model Linguistic feature sequence 6
7. VAE-based Phoneme Alignment Using Gradient Annealing and SSL Acoustic Features Conventional method: One TTS Alignment (2) • Advantage ◦ ◦ • Gradient-based training, which enables simultaneous training with TTS Less constraints in embeddings, different from GMM-HMM Disadvantage ◦ ◦ Low accuracy for analysis TTS does not always accurate alignment Alignment Loss ForwardSum Alignment path Acoustic embedding Linguistic embedding Acoustic encoder Linguistic encoder Acoustic feature sequence Linguistic feature sequence Acoustic feature sequence TTS model Linguistic feature sequence 7
8. 01 Introduction 02 Proposed alignment model 03 Experiments 04 Conclusions 8
9. VAE-based Phoneme Alignment Using Gradient Annealing and SSL Acoustic Features Proposed alignment model • Extend OTA model • Incorporate ◦ ◦ ◦ ◦ VAE architecture SSL-based acoustic features State-level linguistic features Gradient annealing Acoustic decoder Forward-sum Linguistic decoder Acoustic embedding Linguistic embedding Acoustic encoder Linguistic encoder Acoustic feature sequence Linguistic feature sequence 9
10. VAE-based Phoneme Alignment Using Gradient Annealing and SSL Acoustic Features VAE-based Acoustic and Linguistic Embeddings • OTA often fails to overfit the alignment loss ◦ • Embeddings often lose acoustic or linguistic features Use VAE to ensure that embeddings have acoustic and linguistic information Acoustic decoder Forward-sum Linguistic decoder Acoustic embedding Linguistic embedding Acoustic encoder Linguistic encoder Acoustic feature sequence Linguistic feature sequence 10
11. VAE-based Phoneme Alignment Using Gradient Annealing and SSL Acoustic Features SSL Acoustic Features • SSL-based acoustic features ◦ ◦ ◦ Recent studies showed the effectiveness of pretrained features of self-supervised learning (SSL) (ex., wav2vec2.0, HuBERT, WavLM) Pick up L-th layer as input (L=0, 8, …) We can utilize rich SSL acoustic features SSL L-th layer Acoustic input CNN 11
12. VAE-based Phoneme Alignment Using Gradient Annealing and SSL Acoustic Features State-level Linguistic Features • State-level modeling ◦ ◦ ◦ ◦ Divide one phoneme to multiple states to consider sub-phone characteristics (e.g., silence and burst in a /t/ sound) GMM-HMM uses multiple states OTA and CTC uses one state assuming ”one phoneme has only one type of acoustic features” As implementation, we input /t1/ , /t2/, and /t3/ instead of just using /t/ ・State-level ・Phoneme-level /s/ Linguistic embedding Linguistic embedding Encoder Encoder /a/ /t/ /s1/ /s2/ /s3/ /a1/ /a2/ /a3/ /t1/ /t2/ /t3/ 12
13. VAE-based Phoneme Alignment Using Gradient Annealing and SSL Acoustic Features Gradient Annealing on Training • Problem: alignment path converges in the early stage of training • Solution: Anneal intermediate gradient [Gepperth21] of “state occupation prob” to adjacent states Alignment loss: annealing Intermediate gradient: encoders encoders 13
14. 01 Introduction 02 Proposed alignment model 03 Experiments 04 Conclusions 14
15. VAE-based Phoneme Alignment Using Gradient Annealing and SSL Acoustic Features Experimental Conditions • Dataset ◦ The Corpus of Spontaneous Japanese (CSJ) ▪ ◦ Training data ▪ ◦ 33.2 h / 2.49 k utterances / 1.58 M phonemes Test data ▪ • Manually annotated phoneme boundaries in the core data 1.3 h / 896 utterances Model ◦ ◦ 6-layer 1-d CNN for acoustic/linguistic encoders/decoders 64-dim embeddings for alignment 15
16. VAE-based Phoneme Alignment Using Gradient Annealing and SSL Acoustic Features Experimental Conditions • Linguistic features ◦ ◦ • Phonemes (52 entities) 3 states for each phoneme for the state-level modeling Acoustic features ◦ ◦ MFCC / Melspectrogram SSL features ▪ ▪ ▪ ◦ HuBERT (ja) / HuBERT (en) ... trained by Japanese / English database XLS-R … wav2vec2.0 trained by multiple languages w2v2-ASR … fine-tuned XLS-R for Japanese ASR Whisper … Whisper encoder 16
17. VAE-based Phoneme Alignment Using Gradient Annealing and SSL Acoustic Features Experimental Conditions • Metrics ▪ ◦ ◦ MAE … mean absolute error Median … median of absolute error ▪ ◦ ◦ Errors of predicted phoneme boundaries against manual annotation Less sensitive to outliers 20 ms tolerance 50 ms tolerance ▪ Ratio of phoneme boundaries whose errors are higher than 20 ms or 50 ms 17
18. VAE-based Phoneme Alignment Using Gradient Annealing and SSL Acoustic Features Results with conventional methods MAE [ms] Median [ms] 20 ms tol [%] 50 ms tol [%] MFA 16.46 10.45 23.1 4.31 CTC segmentation [Kürzinger20] 40.81 34.63 70.9 30.87 OTA [Shih19, Badlani22] 23.10 14.55 36.1 7.28 Proposed 12.91 8.25 16.1 2.59 – The proposed method predicted more accurate phoneme boundaries. – CTC segmentation had large error rates. 18
19. VAE-based Phoneme Alignment Using Gradient Annealing and SSL Acoustic Features Results: SSL acoustic features 25.00 MAE [ms] 20.00 15.00 10.00 XLS-R L24 L16 L8 L0 L24 L16 L8 L0 L12 L8 L4 HuBERT (en) w2v2-ASR Whisper HuBERT (ja) L0 L12 L8 L4 L0 Melspec 0.00 MFCC 5.00 – SSL features (HuBERT and XLS-R) yielded lower errors than MFCC – ASR-fine-tuned model gave higher errors 19
20. VAE-based Phoneme Alignment Using Gradient Annealing and SSL Acoustic Features Ablation Study MAE [ms] Median [ms] 20 ms tol [%] 50 ms tol [%] Proposed 12.91 8.25 16.1 2.59 w/o VAE 14.96 10.41 23.1 2.90 w/o gradient annealing 13.26 10.45 16.8 2.89 Phoneme-level linguistic unit 16.09 10.29 22.9 4.22 – VAE, gradient annealing, and state-level modeling reduced alignment errors 20
21. VAE-based Phoneme Alignment Using Gradient Annealing and SSL Acoustic Features Alignment results “A few years later the dome fell in.” 21
22. VAE-based Phoneme Alignment Using Gradient Annealing and SSL Acoustic Features Conclusions • Proposed alignment model by extending OTA ◦ ◦ ◦ ◦ VAE architecture SSL-based acoustic features State-level linguistic features Gradient annealing • Reduced phoneme boundary errors compared with conventional MFA, CTC, and OTA • Future work ◦ ◦ ◦ Apply it to other languages including low-resource ones Use character input instead of phonemes Incorporates it into ASR and TTS modeling 22
23. VAE-based Phoneme Alignment Using Gradient Annealing and SSL Acoustic Features 23
24. VAE-based Phoneme Alignment Using Gradient Annealing and SSL Acoustic Features Acoustic and Linguistic Features • SSL-based acoustic features ◦ ◦ ◦ • Recent studies showed the effectiveness of pretrained features of self-supervised learning (SSL) (ex., wav2vec2.0, HuBERT, WavLM) Pick up L-th layer as input (L=0, 8, …) We can utilize rich SSL acoustic features State-level linguistic features ◦ ◦ ◦ ◦ Divide one phoneme to multiple states to consider sub-phone characteristics (e.g., silence and burst in a /t/ sound) GMM-HMM uses multiple states OTA and CTC uses one state assuming ”one phoneme has only one type of acoustic features” As implementation, we input /t1/ , /t2/, and /t3/ instead of just using /t/ 30