Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Parallel WaveGAN: Fast and High-Quality GPU Text-to-Speech

Parallel WaveGAN: Fast and High-Quality GPU Text-to-Speech

Eebedc2ee7ff95ffb9d9102c6d4a065c?s=128

LINE DevDay 2020

November 25, 2020
Tweet

Transcript

  1. None
  2. Ryuichi Yamamoto › Joined LINE at 2018 › Work location

    › Kyoto office › NAVER green factory › Kyoto office (now) › Speech synthesis R&D @ Voice team › Likes open-source software r9y9
  3. Agenda › What is “TTS” ? › Trade off: Quality

    vs. Speed › About Parallel WaveGAN (ICASSP 2020)
  4. What is “TTS” ? to Speech Text Hello

  5. TTS in LINE

  6. None
  7. Order Answer ASR TTS LINE AiCall

  8. Overview of a TTS system Acoustic model Vocoder  Text

    processing
  9. Overview of a TTS system Acoustic model Vocoder [ a-

    ri^ ga- to- o- ] Text processing Text  Linguistic features
  10. Overview of a TTS system Acoustic model Vocoder Text processing

    Acoustic features  [ a- ri^ ga- to- o- ] Linguistic features
  11. Overview of a TTS system Acoustic model Vocoder Text processing

    Acoustic features Waveform  [ a- ri^ ga- to- o- ]
  12. Overview of a TTS system Acoustic model Text processing Acoustic

    features Waveform  [ a- ri^ ga- to- o- ] Today’s focus Neural Vocoder
  13. WaveNet [1] A. van den Oord et al., “WaveNet: A

    generative model for raw audio,” arXiv preprint arXiv:1609.03499, 2016 ! " = $ %&' ( !(*% |*' , … , *%.' ) Autoregressive modeling 16,000 samples / sec
  14. WaveNet https://deepmind.com/blog/article/wavenet-generative-model-raw-audio 3.86 3.67 4.21 4.55 3.47 3.79 4.08 4.21

    3 3.5 4 4.5 5 Concatenative Parametric WaveNet Human Speech US English Mandarin Chinese
  15. Trade off: Quality vs. Speed Autoregressive Generation Non-autoregressive Generation WaveNet

    [1] Parallel WaveNet [2] ClariNet [3] [1] A. van den Oord et al., “WaveNet: A generative model for raw audio,” arXiv preprint arXiv:1609.03499, 2016. [2] A. van den Oord, et al., “Parallel WaveNet: Fast high-fidelity speech synthesis,” in Proc. ICML, 2018. [3] W. Ping, et al., “ClariNet: Parallel wave generation in end-to-end text-to-speech,” in Proc. ICLR, 2019. AR !" !# !# !$ !%&# !' AR AR (% %)# ' Feed-forward model !% %)# '
  16. Trade off: Quality vs. Speed Autoregressive Generation Non-autoregressive Generation WaveNet

    [1] Parallel WaveNet [2] ClariNet [3] [1] A. van den Oord et al., “WaveNet: A generative model for raw audio,” arXiv preprint arXiv:1609.03499, 2016. [2] A. van den Oord, et al., “Parallel WaveNet: Fast high-fidelity speech synthesis,” in Proc. ICML, 2018. [3] W. Ping, et al., “ClariNet: Parallel wave generation in end-to-end text-to-speech,” in Proc. ICLR, 2019.
  17. Trade off: Quality vs. Speed Autoregressive Generation Non-autoregressive Generation WaveNet

    [1] Parallel WaveNet [2] ClariNet [3] [1] A. van den Oord et al., “WaveNet: A generative model for raw audio,” arXiv preprint arXiv:1609.03499, 2016. [2] A. van den Oord, et al., “Parallel WaveNet: Fast high-fidelity speech synthesis,” in Proc. ICML, 2018. [3] W. Ping, et al., “ClariNet: Parallel wave generation in end-to-end text-to-speech,” in Proc. ICLR, 2019. 180 0.03
  18. Trade off: Quality vs. Speed Speed L J Autoregressive Generation

    Non-autoregressive Generation
  19. Trade off: Quality vs. Speed Speed L J Quality J

    L Autoregressive Generation Non-autoregressive Generation
  20. Speed L J Quality J L Training J L Trade

    off: Quality vs. Speed Autoregressive Generation Non-autoregressive Generation
  21. Speed L J Quality J J Training J J Trade

    off: Quality vs. Speed Autoregressive Generation Our approach
  22. Fast Parallel High-quality GAN Efficient WaveNet Multi-resolution STFT loss Parallel

    WaveGAN [4] R. Yamamoto et al., “Parallel WaveGAN: A fast waveform generation model based on generative adversarial networks with multi-resolution spectrogram,” in Proc. ICASSP, 2020, pp. 703.
  23. Agenda › › › About Parallel WaveGAN (ICASSP 2020) ›

    Technical details
  24. Parallel WaveGAN: WaveNet-based waveform generator WaveNet [1] Waveform Samples Parallel

    WaveGAN Random Noise Input Convolution Output Deep residual block Convolution Previous Samples !":$%" Current Samples !$ Previous Samples !":$ Current Samples !$&"
  25. Learn to discriminate more correctly min $ max ' (

    log ,(.) + ( log(1 − ,(3(.)) Random Noise Generated object Ground Truth Which image is a real? Discriminator Generator Generative Adversarial Networks (GANs) min $ max ' ( log ,(.) + ( log(1 − ,(3(.)) min $ max ' ( log ,(.) + ( log(1 − ,(3(.)) Learn to generate more real
  26. Effects of Auxiliary loss for GANs Ground Truth Adversarial loss

    [5] J. Zhu et al., “Unpaired Image-to-Image Translation using Cycle-Consistent Adversarial Networks,” in Proc. ICCV, 2017, pp. 2223--2232. + Auxiliary loss
  27. Time Frequency Magnitude Magnitude (dB) Short-time Fourier transform (STFT)

  28. !"# = STFT ( − |STFT + ( | ,

    |STFT ( | , |STFT + ( | | STFT ( − STFT + ( | STFT ( [6] S. O. Arik, et al, “Fast Spectrogram Inversion using Multi-head Convolutional Neural Networks,” IEEE Signal Procees. Letters, 2019. STFT loss: Spectral convergence (SC) (1/2)
  29. !"#$ = 1 ' log STFT . − log |STFT

    1 . | 2 | log STFT . − log |STFT 1 . || log STFT . log |STFT 1 . | ' : number of elements in the STFT magnitude STFT loss: Log-scale STFT magnitude loss (2/2) [6] S. O. Arik, et al, “Fast Spectrogram Inversion using Multi-head Convolutional Neural Networks,” IEEE Signal Procees. Letters, 2019.
  30. 512 / 240 / 50 1024 / 600 / 120

    2048 / 1200 / 240 Higher frequency resolution Higher temporal resolution Balanced !"#$ (&) = 1 * + ,-. / !0 , (&) !0 (&) = 12~4 2 ,6~47898 [!0; <, = < + !?"@ <, = < ] * : number of STFT losses FFT size / window size / shift Multi-resolution STFT loss
  31. !" ($) Parallel WaveGAN: Training process Random noise & '

    Natural speech !()' !(*+ ,(*+ !- !. …… is Real? Acoustic features / 0 ' 1 $ ' !" (2) !" (1) Generator Discriminator STFT loss (1st ) STFT loss (2nd ) STFT loss (Mth ) Adversarial loss Discriminator loss 3 3 0 ' Generated speech Parameter update Parameter update
  32. Agenda › › › About Parallel WaveGAN (ICASSP 2020) ›

    Performance Evaluations
  33. Acoustic model Text processing Acoustic features Waveform  [ a-

    ri^ ga- to- o- ] Neural Vocoder Text Linguistic features Performance Evaluations
  34. Evaluations: Baseline systems WaveNet [1] ClariNet [3] AR !" !#

    !# !$ !%&# !' AR AR (% %)# ' Feed-forward model !% %)# ' [1] A. van den Oord et al., “WaveNet: A generative model for raw audio,” arXiv preprint arXiv:1609.03499, 2016. [3] W. Ping, et al., “ClariNet: Parallel wave generation in end-to-end text-to-speech,” in Proc. ICLR, 2019.
  35. Evaluations: Database 11,449 250 250 Training / validation / test

    (# of utts.) 23 0.5 0.5 (hours)
  36. 1.36 4.46 4.06 1 1.5 2 2.5 3 3.5 4

    4.5 5 Parallel WaveGAN Single STFT Loss Three STFT Loss Using multi-resolution STFT loss significantly improved perceptual quality Effects of multi-resolution STFT loss Reference
  37. Speed: Inference time WaveNet Parallel WaveGAN x 10,000 Faster J

    vs. 321 0.03
  38. Speed: Training time WaveNet Parallel WaveGAN vs. vs. ClariNet Non-autoregressive

    Non-autoregressive 12.7 2.8 7.4 x 4 Faster J
  39. Acoustic model Text processing Acoustic features Waveform  [ a-

    ri^ ga- to- o- ] Neural Vocoder Text Linguistic features Performance Evaluations
  40. Our model achieved 4.16 MOS competitive to ClariNet [3]. Naturalness

    evaluation in TTS 3.33 4 4.16 4.46 2 2.5 3 3.5 4 4.5 5 WaveNet ClariNet Parallel WaveGAN (Ours) Reference = High Quality J
  41. New TTS sample

  42. Related work 2019.10.8 MelGAN [7] 2019.10.25 Parallel WaveGAN [4] 2020.5.11

    Multi-band MelGAN [8] 2020.7.30 VocGAN [9] 2020.5.18 Quasi-periodic Parallel WaveGAN [11] 2020.7.8 FastSpeech 2s [12] 2020.10.12 HiFi-GAN [10] 2020.9.3 HiFiSinger [13]
  43. 3.33 4 4.16 4.46 2 2.5 3 3.5 4 4.5

    5 WaveNet ClariNet Parallel WaveGAN (Ours) Reference High Quality J Fast J Conclusion vs. ClariNet Parallel WaveGAN 0.03 2.8 vs. 12.7 321 WaveNet Parallel WaveGAN
  44. Thank you

  45. References [1] A. van den Oord et al., “WaveNet: A

    generative model for raw audio,” arXiv preprint arXiv:1609.03499, 2016. [2] A. van den Oord, et al., “Parallel WaveNet: fast high-fidelity speech synthesis,” in Proc. ICML, 2018. [3] W. Ping, et al., “ClariNet: Parallel wave generation in end-to-end text-to-speech,” in Proc. ICLR, 2019. [4] R. Yamamoto et al., “Parallel WaveGAN: A fast waveform generation model based on generative adversarial networks with multi- resolution spectrogram,” in Proc. ICASSP, 2020, pp. 703. [5] J. Zhu et al., “Unpaired image-to-image translation using cycle-consistent adversarial networks,” in Proc. ICCV, 2017, pp. 2223--2232. [6] S. O. Arik, et al, “Fast spectrogram inversion using multi-head convolutional neural networks,” IEEE Signal Procees. Letters, 2019. 2017. [7] K. Kumar, et al, “MelGAN: Generative adversarial networks for conditional waveform synthesis,” in Proc. NeurIPS, 2019, pp. 14881–14892. [8] Y. Geng, et al, “Multi-band MelGAN: Faster waveform generation for high-quality text-to-speech,” arXiv preprint arXiv:2005.05106, 2020. [9] J. Yang, et al., “VocGAN: A high-fidelity real-time vocoder with a hierarchically-nested adversarial network,” in Proc. INTERSPEECH, 2020. [10] J. Kong, et al, “HiFi-GAN: Generative adversarial networks for efficient and high fidelity speech synthesis,” in Proc. NeurIPS, 2020. [11] Y. Wo, et al., “Quasi-Periodic Parallel WaveGAN vocoder: A non-autoregressive pitch-dependent dilated convolution model for parametric speech generation,” in Proc. INTERSPEECH, 2020. [12] Y. Ren et al., “FastSpeech 2: Fast and high-quality end-to-end text to speech,” arXiv preprint arXiv:2006.04558, 2020. [13] J. Chen et al., “HiFiSinger: Towards high-fidelity neural singing voice synthesis,” arXiv preprint arXiv:2009.01776, 2020. Icon made by Pixel perfect from www.flaticon.com