Tutorial: Foundations of Blind Source Separation and Its Advances in Spatial Self-Supervised Learning

A self-contained (& slightly updated) version of our Interspeech tutorial: https://speakerdeck.com/yoshipon/interspeech2023-t5-part4-bando

Demo pages:
Neural FCA: https://ybando.jp/projects/spl2021/
TV-Neural FCA: https://ybando.jp/projects/spl2023/
Neural FCASA: https://ybando.jp/projects/neural-fcasa/

Transcript

1. Foundations of Blind Source Separation and Its Advances in Spatial Self-Supervised Learning Yoshiaki Bando National Institute of Advanced Industrial Science and Technology (AIST), Japan Center for Advanced Intelligent Project (AIP), RIKEN, Japan
2. Blind Source Separation (BSS) Sound source separation forms the basis of machine listening systems. • Such systems are often required to work in diverse environments. • This calls for BSS, which can work adaptively for the target environment. Distant speech recognition (DSR) [Watanabe+ 2020, Baker+ 2018] Sound event detection (SED) [Turpault+ 2020, Denton+ 2022] Foundations of Blind Source Separation and Its Advances in Spatial Self-Supervised Learning /46 2
3. Foundation of Modern BSS Methods Probabilistic generative models of multichannel mixture signals. • The generative model consists of a source model and a spatial model Source model ⋯ 𝑠𝑠𝑛𝑛𝑛𝑛𝑛𝑛 ∼ 𝒩𝒩ℂ 0, λ𝑛𝑛𝑛𝑛𝑛𝑛 𝑓𝑓 𝑡𝑡 𝑓𝑓 𝑡𝑡 Observed mixture 𝑓𝑓 𝑡𝑡 𝑚𝑚 Spatial model ⋯ 𝐱𝐱𝑛𝑛𝑛𝑛𝑛𝑛 ∼ 𝒩𝒩ℂ 0, λ𝑛𝑛𝑛𝑛𝑛𝑛 𝐇𝐇𝑛𝑛𝑛𝑛 𝑓𝑓 𝑡𝑡 𝑓𝑓 𝑡𝑡 𝑚𝑚 𝑚𝑚 𝑠𝑠1𝑓𝑓𝑓𝑓 𝐱𝐱𝑓𝑓𝑓𝑓 ∼ 𝒩𝒩ℂ 0, ∑𝑛𝑛 λ𝑛𝑛𝑛𝑛𝑛𝑛 𝐇𝐇𝑛𝑛𝑓𝑓 𝑠𝑠𝑁𝑁𝑓𝑓𝑓𝑓 𝐱𝐱1𝑓𝑓𝑓𝑓 𝐱𝐱𝑁𝑁𝑁𝑁𝑁𝑁 𝐱𝐱𝑓𝑓𝑓𝑓 ∈ ℝ𝑀𝑀 Foundations of Blind Source Separation and Its Advances in Spatial Self-Supervised Learning /46 3
4. Geometric Interpretation of Multichannel Generative Models Multivariate Gaussian representation of source images 𝐱𝐱𝑛𝑛𝑛𝑛𝑛𝑛 ∈ ℂ𝑀𝑀 𝐱𝐱𝑛𝑛𝑛𝑛𝑛𝑛 ∼ 𝒩𝒩ℂ 0, λ𝑛𝑛𝑛𝑛𝑛𝑛 𝐇𝐇𝑛𝑛𝑛𝑛 • Spatial covariance matrices (SCMs) 𝐇𝐇𝑛𝑛𝑛𝑛 ∈ 𝕊𝕊+ 𝑀𝑀×𝑀𝑀: “shape” of the ellipse • Power spectral density (PSD) 𝜆𝜆𝑛𝑛𝑛𝑛𝑛𝑛 ∈ ℝ+ : “size” of the ellipse こ んにちは！ Hello! Late Early 𝑛𝑛 = 1 𝑚𝑚1 𝑚𝑚2 𝜆𝜆1𝑓𝑓𝑓𝑓 𝐇𝐇1𝑓𝑓 𝑛𝑛 = 2 𝜆𝜆2𝑓𝑓𝑓𝑓 𝐇𝐇2𝑓𝑓 Foundations of Blind Source Separation and Its Advances in Spatial Self-Supervised Learning /46 4
5. Spatial Models for Blind Source Separation Rank-1 spatial model: 𝐇𝐇𝑛𝑛𝑛𝑛 = 𝐚𝐚𝑛𝑛𝑛𝑛 𝐚𝐚𝑛𝑛𝑛𝑛 H Fast and stable by the IP [Ono+ 2011] or ISS [Sheibler+ 2020] algorithm Weak against reverberations and diffuse noise. Full-rank spatial model: 𝐇𝐇𝑛𝑛𝑛𝑛 ∈ 𝕊𝕊𝑀𝑀×𝑀𝑀 Robust against reverberations and diffuse noise. Computationally expensive due to its EM or MU algorithm. Jointly-diagonalizable (JD) spatial model: 𝐇𝐇𝑛𝑛𝑛𝑛 ≜ 𝐐𝐐𝑓𝑓 −1 diag 𝐰𝐰𝑛𝑛 𝐐𝐐𝑓𝑓 −H Still robust against reverberations and diffuse noise. Moderately fast by IP or ISS algorithm. 𝑚𝑚1 𝑚𝑚2 can be considered as ∑𝑚𝑚 𝑤𝑤𝑛𝑛𝑛𝑛 𝐚𝐚𝑓𝑓𝑓𝑓 𝐚𝐚𝑓𝑓𝑓𝑓 H 𝑚𝑚1 𝑚𝑚2 [Duong+ 2010] [Yoshii+ 2013] Foundations of Blind Source Separation and Its Advances in Spatial Self-Supervised Learning /46 5
6. Source Model Based on Low-Rank Approximation Source power spectral density (PSD) often has low-rank structures. • Source PSD is estimated by non-negative matrix factorization (NMF) [Ozerov+ 2009]. • Its inference is fast and does not require supervised pre-training. 𝑠𝑠𝑓𝑓𝑓𝑓 ∼ 𝒩𝒩ℂ 0, ∑𝑘𝑘 𝑢𝑢𝑓𝑓𝑓𝑓 𝑣𝑣𝑘𝑘𝑘𝑘 Is there a more powerful representation of source spectra? × ∼ 𝑠𝑠𝑓𝑓𝑓𝑓 𝜆𝜆𝑓𝑓𝑓𝑓 𝑢𝑢𝑓𝑓𝑓𝑓 𝑣𝑣𝑘𝑘𝑘𝑘 Source PSD Source signal Bases Activations Foundations of Blind Source Separation and Its Advances in Spatial Self-Supervised Learning /46 6
7. Source Model Based on Deep Generative Model Source spectra are represented with low-dim. latent feature vectors. • A DNN is used to generate source power spectral density (PSD) precisely. • Freq.-independent latent features helps us to solve freq. permutation ambiguity. ∼ DNN Latent features Source PSD Source signal 𝑠𝑠𝑓𝑓𝑓𝑓 𝜆𝜆𝑓𝑓𝑓𝑓 𝑧𝑧𝑡𝑡𝑡𝑡 𝑔𝑔𝜃𝜃,𝑓𝑓 𝑠𝑠𝑓𝑓𝑓𝑓 ∣ 𝐳𝐳𝑡𝑡 ∼ 𝒩𝒩ℂ 0, 𝑔𝑔𝜃𝜃,𝑓𝑓 𝐳𝐳𝑡𝑡 Y. Bando, et al. "Statistical speech enhancement based on probabilistic integration of variational autoencoder and non- negative matrix factorization." IEEE ICASSP, pp. 716-720, 2018. 𝑧𝑧𝑡𝑡𝑡𝑡 ∼ 𝒩𝒩 0, 1 Foundations of Blind Source Separation and Its Advances in Spatial Self-Supervised Learning /46 7
8. Various BSS methods Combining Spatial and Source Models ※ In this talk, we often call GMVAE as MVAE. Rank-1 spatial model Independent Low-Rank Matrix Analysis (ILRMA) [Kitamura+ 2016] Multi-channel NMF (MNMF) [Sawada+ 2013] FastMNMF [Ito+ 2019, Sekiguchi+2019] Full-rank spatial model JD spatial model NMF source model VAE source model (Supervised) Generalized MVAE (GMVAE) [Seki + 2019] Multi-channel VAE (MVAE) [Kameoka+ 2018] Neural Full-rank Spatial Covariance Analysis (Neural FCA) [Bando+ 2021] Neural FastFCA [Bando+ 2023] VAE source model (Unsupervised) Foundations of Blind Source Separation and Its Advances in Spatial Self-Supervised Learning /46 8
9. Contents Two applications of deep source generative models. 1. Semi-supervised speech enhancement • We enhance speech signals by training on only clean speech signal. • Combination of a deep speech model and low-rank noise models 2. Self-supervised source separation • We train neural source separation model only from multichannel mixtures • The joint training of the source generative model and its inference model 2. Extensions of self-supervised training for real-world understanding • Handling moving sources / speeding up training & inference • Application for joint speech separation and diarization Foundations of Blind Source Separation and Its Advances in Spatial Self-Supervised Learning /46 9
10. Part 1: Multichannel Speech Enhancement Based on Supervised Deep Source Model • K. Sekiguchi, Y. Bando, A. A. Nugraha, K. Yoshii, T. Kawahara, “Semi-supervised Multichannel Speech Enhancement with a Deep Speech Prior,” IEEE/ACM TASLP, 2019 • K. Sekiguchi, A. A. Nugraha, Y. Bando, K. Yoshii, “Fast Multichannel Source Separation Based on Jointly Diagonalizable Spatial Covariance Matrices,” EUSIPCO, 2019 • Y. Bando, M. Mimura, K. Itoyama, K. Yoshii, T. Kawahara, “Statistical Speech Enhancement Based on Probabilistic Integration of Variational Autoencoder and Nonnegative Matrix Factorization,” IEEE ICASSP, 2018 Foundations of Blind Source Separation and Its Advances in Spatial Self-Supervised Learning / 46 10
11. Speech Enhancement A task to extract speech signals from a mixture of speech and noise • Various applications such as DSR, search-and-rescue, and hearing aids. Robustness against various acoustic environment is essential. • It is often difficult to assume the environment where they are used. Hey, Siri… CC0: https://pxhere.com/ja/photo/1234569 CC0: https://pxhere.com/ja/photo/742585 Foundations of Blind Source Separation and Its Advances in Spatial Self-Supervised Learning /46 11
12. Semi-Supervised Enhancement With Deep Speech Prior A hybrid method of deep speech model and statistical noise model • We can use many speech corpus  deep speech prior • Noise training data are often few  statistical noise prior w/ low-rank model + ≈ Observed noisy speech Deep speech prior Statistical noise prior Speech corpus Pre-training Estimated on the fly Foundations of Blind Source Separation and Its Advances in Spatial Self-Supervised Learning /46 12
13. Supervised Training of Deep Speech Prior (DP) The training based on a variational autoencoder (VAE) [Kingma+ 2013] • An encoder 𝑞𝑞𝜙𝜙 𝐙𝐙 𝐒𝐒 is introduced to estimate latent features from clean speech. The objective function is the evidence lower bound (ELBO) ℒ𝜃𝜃,𝜙𝜙 ℒ𝜃𝜃,𝜙𝜙 = 𝔼𝔼𝑞𝑞𝜙𝜙 log 𝑝𝑝𝜃𝜃 𝐒𝐒 𝐙𝐙 − 𝒟𝒟KL 𝑞𝑞𝜙𝜙 𝐙𝐙|𝐒𝐒 𝑝𝑝 𝐙𝐙 Reconstructed speech Latent features 𝐙𝐙 Observed speech Reconstruction term (IS-div.) Regularization term (KL-div.) Encoder 𝑞𝑞𝜙𝜙 𝐙𝐙 𝐒𝐒 Decoder 𝑝𝑝𝜃𝜃 𝐒𝐒 𝐙𝐙 The training is performed by making the reconstruction closer to the observation. Foundations of Blind Source Separation and Its Advances in Spatial Self-Supervised Learning /46 13
14. FastMNMF with a Deep Speech Prior (FastMNMF-DP) A unified generative model combining the VAE-based source model, NMF- based noise model, and jointly-diagonalizable (JD) spatial model. VAE-based speech model DNN 𝑧𝑧𝑑𝑑𝑑𝑑 𝜆𝜆1𝑓𝑓𝑓𝑓 NMF-based noise model × 𝑁𝑁 × JD spatial model SCM 𝐇𝐇𝑛𝑛𝑛𝑛 JD spatial model SCM 𝐇𝐇1𝑓𝑓 𝑚𝑚1 𝑚𝑚2 𝑚𝑚1 𝑚𝑚2 𝜆𝜆𝑛𝑛𝑛𝑛𝑛𝑛 Latent features Speech PSD Noise PSDs Activations 𝑣𝑣𝑘𝑘𝑘𝑘 Bases 𝑢𝑢𝑘𝑘𝑘𝑘 Speech image Noise images Noisy observation 𝐱𝐱𝑛𝑛𝑛𝑛𝑛𝑛 𝐱𝐱1𝑓𝑓𝑓𝑓 𝐱𝐱𝑓𝑓𝑓𝑓 JD SCMs 𝐇𝐇𝑛𝑛𝑛𝑛 = 𝐐𝐐𝑓𝑓 diag 𝐠𝐠𝑛𝑛𝑛𝑛 𝐐𝐐𝑓𝑓  Foundations of Blind Source Separation and Its Advances in Spatial Self-Supervised Learning /46 14
15. Monte-Carlo Expectation-Maximization (MC-EM) Inference Speech and noise are separated by estimating the model parameters. Speech signal is finally obtained by multichannel Wiener filtering. E-step samples latent features from its posterior 𝐳𝐳𝑡𝑡 ∼ 𝑝𝑝 𝐳𝐳𝑡𝑡 𝐗𝐗 • Metropolis-Hasting sampling is utilized due to its intractability. M-step updates the other parameters to maximize log 𝑝𝑝 𝐗𝐗 𝐐𝐐, � 𝐇𝐇, 𝐔𝐔, 𝐕𝐕 • 𝐐𝐐 is updated by the iterative-projection (IP) algorithm [Ono+ 2011]. • � 𝐇𝐇, 𝐔𝐔, 𝐕𝐕 are updated by multiplicative-update (MU) algorithm [Nakano+ 2010]. 1) domain transformation 2) TF masking 3) projection back Foundations of Blind Source Separation and Its Advances in Spatial Self-Supervised Learning /46 15
16. Experimental Condition We evaluated with a part of the CHiME-3 noisy speech dataset • 100 utterances from the CHiME-3 evaluation set • Each utterance was recorded by a 6-channel* mic. array on a tablet device. • The CHiME-3 dataset includes four noise environments: Evaluation metrics: • Source-to-distortion ratio (SDR) [dB] for evaluating enhancement performance • Computational time [msec] for evaluating the efficiency of the method. On a bus In a cafeteria In a pedestrian area On a street junction http://spandh.dcs.shef.ac.uk/chime_challenge/CHiME4/data.html *We emitted one microphone on the back of the tablet Foundations of Blind Source Separation and Its Advances in Spatial Self-Supervised Learning /46 16
17. Enhancement Performance in SDRs DP successively improved SDRs for FastMNMF and MNMF. • The JD full-rank model was better than full-rank and rank-1 models. Method Source model Spatial model FastMNMF-DP DP + NMF JD full-rank FastMNMF NMF JD full-rank MNMF-DP DP + NMF Full-rank MNMF NMF Full-rank ILRMA NMF Rank-1 [Sekiguchi+ 2019] [Sekiguchi+ 2019] [Sawada+ 2013] [Kitamura+ 2016] 15.1 13.2 18.6 16.8 18.9 12 13 14 15 16 17 18 19 20 [Sekiguchi+ 2019] Average SDR [dB] over 100 utterances Foundations of Blind Source Separation and Its Advances in Spatial Self-Supervised Learning /46 17
18. Computational Times for Speech Enhancement Although DP slightly increased computational cost, FastMNMF-DP was much faster than MNMF. Method Source model Spatial model FastMNMF-DP DP + NMF JD full-rank FastMNMF NMF JD full-rank MNMF-DP DP + NMF Full-rank MNMF NMF Full-rank ILRMA NMF Rank-1 [Sekiguchi+ 2019] [Sekiguchi+ 2019] [Sawada+ 2013] [Kitamura+ 2016] 10 660 710 40 78 0 100 200 300 400 500 600 700 800 [Sekiguchi+ 2019] Computational time [ms] for an 8-second signal *Evaluation is performed with NVIDIA TITAN RTX Foundations of Blind Source Separation and Its Advances in Spatial Self-Supervised Learning /46 18
19. Excerpts of Enhancement Results Observation Clean speech ILRMA FastMNMF-DP Foundations of Blind Source Separation and Its Advances in Spatial Self-Supervised Learning /46 19
20. Part 2: Self-Supervised Learning of Deep Source Generative Model and Its Inference Model • Y. Bando, K. Sekiguchi, Y. Masuyama, A. A. Nugraha, M. Fontaine, K. Yoshii, “Neural full-rank spatial covariance analysis for blind source separation,” IEEE SP Letters, 2021 Foundations of Blind Source Separation and Its Advances in Spatial Self-Supervised Learning / 46 20
21. Source Separation Based on Multichannel VAEs (MVAEs) Deep source generative models achieved excellent performance. • 𝐳𝐳𝑛𝑛𝑛𝑛 and 𝐇𝐇𝑓𝑓𝑓𝑓 are estimated to maximize the likelihood function at the inference Can the deep source models be trained only from mixture signals? Generative model Multichannel reconstruction ⋯ Latent source features ⋯ × × × ⋯ SCM Source PSD [Kameoka+ 2018, Seki+ 2019] Foundations of Blind Source Separation and Its Advances in Spatial Self-Supervised Learning /46 21
22. Self-Supervised Training of Deep Source Model The generative model is trained jointly with its inference model. • We train the models regarding them as a “large VAE” for a multichannel mixture. The training is performed to make the reconstruction closer to the observation. Inference model Generative model Multichannel mixture Multichannel reconstruction ⋯ ⋯ Latent source features ⋯ × × × ⋯ SCM Source PSD Foundations of Blind Source Separation and Its Advances in Spatial Self-Supervised Learning /46 22
23. Training Based on Autoencoding Variational Bayes As in the training of the VAE, the ELBO ℒ is maximized by using SGD. • Our training can be considered as BSS for all the training mixtures. Generative model Multichannel mixture Multichannel reconstruction ⋯ ⋯ Inference model Latent source features ⋯ Minimize 𝒟𝒟𝐾𝐾𝐾𝐾 𝑞𝑞 𝐙𝐙 𝐗𝐗 𝑝𝑝 𝐙𝐙 𝐗𝐗, 𝐇𝐇 Maximize 𝑝𝑝 𝐗𝐗 𝐇𝐇 EM update rule Foundations of Blind Source Separation and Its Advances in Spatial Self-Supervised Learning /46 23
24. Solving Frequency Permutation Ambiguity We solve the ambiguity by making latent vectors 𝐳𝐳1𝑡𝑡 , … , 𝐳𝐳𝑁𝑁𝑁𝑁 independent.  Each source shares the same content  Latent vectors have a LARGE correlation The KL term weight 𝛽𝛽 is set to a large value for first several epochs. • approaches to the std. Gaussian dist. (no correlation between sources). • Disentanglement of the latent features by β-VAE.  Each source has a different content  Latent vectors have a SMALL correlation 𝑓𝑓 𝑡𝑡 𝑓𝑓 𝑡𝑡 𝑓𝑓 𝑡𝑡 𝑓𝑓 𝑡𝑡 Source 1 Source 2 Source 1 Source 2 Foundations of Blind Source Separation and Its Advances in Spatial Self-Supervised Learning /46 24
25. Relations Between Neural FCA and Existing Methods Neural FCA is a NEURAL & BLIND source separation method • Self-supervised training of the deep source generative model Linear BLIND Source Separation NEURAL (Semi-)supervised Source Separation MNMF [Ozerov+ 2009, Sawada+ 2013] ILRMA [Kitamura+ 2015] FastMNMF [Sekiguchi+ 2019, Ito+ 2019] IVA [Ono+ 2011] MVAE [Kameoka+ 2018] FastMNMF-DP [Sekiguchi+ 2018, Leglaive+ 2019] IDLMA [Mogami+ 2018] DNN-MSS [Nugraha+ 2016] Neural FCA (proposed) NF-IVA [Nugraha+ 2020] NF-FastMNMF [Nugraha+ 2022] Neural spatial models Neural source model NEURAL BLIND Source Separation Foundations of Blind Source Separation and Its Advances in Spatial Self-Supervised Learning /46 25
26. Experimental Condition Evaluation with the spatialized WSJ0-2mix dataset • 4-ch mixture signals of two speech sources with RT60 = 200–600 ms • All mixture signals were dereverberated in advance by using WPE. Method Brief description Permutation solver cACGMM [Ito+ 2016] Conventional linear BSS methods (for determined conditions) Required FCA [Duong+ 2010] Required FastMNMF2 [Sekiguchi+ 2020] Free Pseudo supervised [Togami+ 2020] DNN imitates the MWF of BSS (FCA) results Required Neural cACGMM [Drude+ 2019] DNN is trained to maximize the log-marginal likelihood of the cACGMM Required MVAE[Seki+ 2019] The supervised version of our neural FCA – Neural FCA (proposed) Our neural blind source separation method Free Foundations of Blind Source Separation and Its Advances in Spatial Self-Supervised Learning /46 26
27. Experimental Results With SDRs Neural FCA outperformed conventional BSS methods and neural unsupervised methods and was comparable to the supervised MVAE. 15.2 2.9 15.2 12.4 14.7 13.0 12.7 10.8 0 2 4 6 8 10 12 14 16 cACGMM FCA FastMNMF2 Pseudo supervised Neural cACGMM Neural FCA MVAE (random init.) MVAE (FCA init.) SDR (higher is better) [dB] Foundations of Blind Source Separation and Its Advances in Spatial Self-Supervised Learning /46 27
28. Excerpts of Separation Results Neural FCA *More separation examples: https://ybando.jp/projects/spl2021 FastMNMF MVAE (supervised) Mixture input Foundations of Blind Source Separation and Its Advances in Spatial Self-Supervised Learning /46 28
29. Part 3: Toward Real-World Understanding via Spatial Self-Supervised Learning • H. Munakata, Y. Bando, R. Takeda, K. Komatani, M. Onishi, “Joint Separation and Localization of Moving Sound Sources Based on Neural Full-Rank Spatial Covariance Analysis,” IEEE SP Letters, 2023 • Y. Bando, Y. Masuyama, A. A. Nugraha, K. Yoshii, “Neural Fast Full-Rank Spatial Covariance Analysis for Blind Source Separation,” EUSIPCO, 2023 • Y. Bando, T. Nakamura, S. Watanabe, “Neural blind source separation and diarization for distant speech recognition,” accepted to INTERSPEECH 2024 Foundations of Blind Source Separation and Its Advances in Spatial Self-Supervised Learning / 46 29
30. Extension 1: Separation of Moving Sound Sources BSS methods usually assume that sources are (almost) stationary. • Many daily sound sources move (e.g., walking persons, natural habitats, cars, …) • All sources relatively move if the microphone moves (e.g., mobile robots). Woo-hoo! Broom! Chirp, chip Foundations of Blind Source Separation and Its Advances in Spatial Self-Supervised Learning /46 30
31. Time-Varying (TV) Neural FCA Joint source localization and separation for tracking moving sources. • The localization results are constrained to be smooth by moving average. • SCMs are then constrained by the time-varying smoothed localization results. Generative model Inference model 𝐇𝐇0𝑛𝑛𝑛𝑛 𝐇𝐇1𝑛𝑛𝑛𝑛 𝐇𝐇𝑁𝑁𝑛𝑛𝑛𝑛 𝐮1𝑛𝑛 𝐮𝑁𝑁𝑛𝑛 Time-varying SCMs Latent spectral features Time-varying DoAs Regularize Separation Localization SCM Source PSD Multichannel mixture Multichannel reconstruction 𝑔𝑔𝜃𝜃,𝑛𝑛 𝐳𝐳0𝑛𝑛 𝑔𝑔𝜃𝜃,𝑛𝑛 𝐳𝐳𝑁𝑁𝑛𝑛 𝑔𝑔𝜃𝜃,𝑛𝑛 𝐳𝐳1𝑛𝑛 Foundations of Blind Source Separation and Its Advances in Spatial Self-Supervised Learning /46 31
32. Training on Mixtures of Two Moving Speech Sources TV Neural FCA performed well regardless of source velocity. • FastMNMF2 and Neural FCA drastically degraded when sources move fast. • TV-Neural FCA can improved avg. SDR 4.2dB from that of DoA-HMM [Higuchi+ 2014] SDR [dB] 0 2 4 6 8 10 12 14 Average 0-15°/s 15-30°/s 30-45°/s TV-Neural FCA Neural FCA FastMNMF DOA-HMM Foundations of Blind Source Separation and Its Advances in Spatial Self-Supervised Learning /46 32
33. Separation Results of Moving Sound Sources Our method can be trained from mixtures of moving sources. • Robustness against real audio recordings was improved. Stationary condition Moving condition FastMNMF FastMNMF TV Neural FCA TV Neural FCA Foundations of Blind Source Separation and Its Advances in Spatial Self-Supervised Learning /46 33
34. ⋯ Multichannel reconstruction Extension 2: Speeding Up Neural FCA Iterative estimation of full-rank SCMs is computationally demanding. • Training for 30 hours of 8-ch data (WSJ0-2mix) requires 400 GPU hours @ NVIDIA V100 Inference model Multichannel mixture ⋯ ⋯ × × ⋯ Generative model Latent source features × SCM Source PSD The neural models are jointly trained to maximize the likelihood for training mixtures. Estimated by a heavy EM algorithm Foundations of Blind Source Separation and Its Advances in Spatial Self-Supervised Learning /46 34
35. Deep Source Model + JD Spatial Model  Neural FastFCA Speeding up neural FCA with a JD spatial model and the ISS algorithm. We utilize the ISS algorithm in the inference model to quickly estimate SCMs. Inference model Multichannel mixture Multichannel reconstruction ⋯ Latent source features ⋯ ⋯ × Source PSD × × ⋯ SCM Generative model DNN ISS JD SCM parameters [Scheibler+ 2021] Foundations of Blind Source Separation and Its Advances in Spatial Self-Supervised Learning /46 35
36. A multichannel mixture is generated by a local Gaussian model w/ JD SCMs. • This likelihood can be simplified with � 𝐱𝐱𝑓𝑓𝑓𝑓 ≜ 𝐐𝐐𝑓𝑓 𝐱𝐱𝑓𝑓𝑓𝑓 and � 𝑦𝑦𝑓𝑓𝑓𝑓𝑓𝑓 ≜ ∑𝑛𝑛 𝑤𝑤𝑛𝑛𝑛𝑛 𝑔𝑔𝜃𝜃,𝑓𝑓 𝐳𝐳𝑛𝑛𝑛𝑛 as: ⋯ Multichannel reconstruction Generative Model of Mixture Signals Based on a JD Spatial Model Generative model × Source PSD JD SCM 𝐐𝐐𝑓𝑓 −1 diag 𝐰𝐰1 𝐐𝐐𝑓𝑓 −H ⋯ × × ⋯ 𝐐𝐐𝑓𝑓 −1 diag 𝐰𝐰2 𝐐𝐐𝑓𝑓 −H 𝐐𝐐𝑓𝑓 −1 diag 𝐰𝐰𝑁𝑁 𝐐𝐐𝑓𝑓 −H 𝐱𝐱𝑓𝑓𝑓𝑓 ∼ 𝒩𝒩ℂ 0, 𝐐𝐐𝑓𝑓 −1 ∑𝑛𝑛 𝑔𝑔𝜃𝜃,𝑓𝑓 𝐳𝐳𝑛𝑛𝑛𝑛 diag 𝐰𝐰𝑛𝑛 𝐐𝐐𝑓𝑓 −H 𝑝𝑝𝜃𝜃 𝐗𝐗 𝐐𝐐, 𝐖𝐖, 𝐙𝐙 = 2𝑇𝑇 ∑𝑓𝑓 log |𝐐𝐐𝑓𝑓 | − ∑𝑓𝑓,𝑡𝑡,𝑚𝑚 log � 𝑦𝑦𝑓𝑓𝑓𝑓𝑓𝑓 + � 𝑥𝑥𝑓𝑓𝑓𝑓𝑓𝑓 2 � 𝑦𝑦𝑓𝑓𝑓𝑓𝑓𝑓 Foundations of Blind Source Separation and Its Advances in Spatial Self-Supervised Learning /46 36
37. Inference Model Integrating DNN and ISS-Based Blocks The inference model estimates the params. of the generative model. • The ISS algorithm is involved to quickly estimate 𝐐𝐐𝑓𝑓 from 𝐱𝐱𝑓𝑓𝑓𝑓 and mask 𝒎𝒎𝜙𝜙,𝑓𝑓𝑓𝑓. • Each DNN utilizes an intermediate diagonalization result for its estimate. DNN(1) ISS(1) 𝐡 𝜙𝜙,𝑛𝑛 (1) 𝐐𝐐 𝑛𝑛 (1) 𝐦 𝜙𝜙,𝑛𝑛𝑛𝑛 (1) DNN(0) 𝐐𝐐 𝑛𝑛 (0) 𝐡 𝜙𝜙,𝑛𝑛 (0) 𝐦 𝜙𝜙,𝑛𝑛𝑛𝑛 (0) 𝐱𝐱𝑛𝑛𝑛𝑛 𝐱𝐱 � 𝑛𝑛𝑛𝑛 (1) DNN(𝐵) ISS(B) 1 × 1 Conv 𝐱𝐱 � 𝑛𝑛𝑛𝑛 (𝐵) 𝐡 𝜙𝜙,𝑛𝑛 (𝐵) 𝝎𝜙𝜙,𝑛𝑛𝑛𝑛𝑛𝑛 𝝁𝜙𝜙,𝑛𝑛𝑛𝑛 𝝈𝜙𝜙,𝑛𝑛𝑛𝑛 2 𝐐𝐐 𝑛𝑛 (𝐵) 1st blocks 𝐵-th blocks 1st blocks B-th blocks DNN(0) DNN(1) DNN(B) ISS(B) ISS(1) 1×1 Conv Foundations of Blind Source Separation and Its Advances in Spatial Self-Supervised Learning /46 37
38. Experimental Condition: Speech Separation Evaluation was performed with simulated 8-ch speech mixtures • The simulation was almost the same as the spatialized WSJ0-mix dataset. • The main difference is that # of srcs. was randomly drawn between 2 and 4. • All the mixtures were dereverberated in advance by using the WPE method. All the methods are performed by specifying a fixed # (5) of sources. • We show that our method can work with only specifying the max. # of sources. Method Brief description # of iters. MNMF [Sawada+ 2013] Conventional linear BSS methods that have ability to solve frequency permutation ambiguity 200 ILRMA [Kitamura+ 2016] FastMNMF [Sekiguchi+ 2020] Neural FCA [Bando+ 2021] The conventional neural BSS method 200 Neural FastFCA (Proposed) The proposed neural BSS method Iteration free Foundations of Blind Source Separation and Its Advances in Spatial Self-Supervised Learning /46 38
39. Experimental Results: Average Separation Performance Our method outperformed the conventional BSS methods in all the metrics and slightly better than neural FCA in SDR and STOI. 7.5 7 9.3 8.9 11.1 11.6 6 7 8 9 10 11 12 SDR 1.49 1.43 1.6 1.71 1.88 1.85 1.32 1.42 1.52 1.62 1.72 1.82 PESQ 0.76 0.76 0.8 0.79 0.84 0.85 0.74 0.76 0.78 0.8 0.82 0.84 0.86 STOI ▪ MNMF ▪ ILRMA ▪ FastMNMF ▪ Neural FCA (fix z) ▪ Neural FCA ▪ Neural FastFCA Foundations of Blind Source Separation and Its Advances in Spatial Self-Supervised Learning /46 39
40. Experimental Results: Elapsed Time for Inference On the other hand, the elapsed time was drastically improved from neural FCA thanks to the JD spatial model and ISS-based inference model. 0.09 4.77 2.67 1.81 1.36 2.07 0 0.5 1 1.5 2 2.5 3 3.5 4 4.5 5 Elapsed time for separating a 5-second mixture using NVIDIA V100 GPU [s] ▪ MNMF ▪ ILRMA ▪ FastMNMF ▪ Neural FCA (fix z) ▪ Neural FCA ▪ Neural FastFCA 53x faster Foundations of Blind Source Separation and Its Advances in Spatial Self-Supervised Learning /46 40
41. Extension 3: Front-End of Distant Speech Recognition (DSR) It is essential for DSR to extract speech sources from noisy mixture signals having dynamically changing numbers of active speakers. Single-speaker DSR (e.g., smart speakers) has achieved excellent performance. (e.g., CHiME-3, 4 Challenges) https://spandh.dcs.shef.ac.uk//chime_challenge/chime2015/overview.html (e.g., CHiME-5, 6, 7, 8 Challenges) Multi-speaker DSR (e.g., home parties) is still a challenging problem. https://spandh.dcs.shef.ac.uk//chime_challenge/CHiME5/overview.html Foundations of Blind Source Separation and Its Advances in Spatial Self-Supervised Learning /46 41
42. Proposed Method: Neural FCA With Speaker Activity (FCASA) Multi-task learning of self-supervised separation and supervised diarization. Generative model Latent source features Inference model ⋯ SCMs × Source PSD × × ⋯ SCM Multichannel mixture ⋯ Multichannel reconstruction ⋯ ⋯ Separation 𝑢𝑢1𝑡𝑡 𝑢𝑢2𝑡𝑡 𝑢𝑢𝑁𝑁𝑁𝑁 Source activity masking Diarization Foundations of Blind Source Separation and Its Advances in Spatial Self-Supervised Learning /46 42
43. Demo: Separation and Diarization of English Conversation Training only on 80 hours of 8-ch mixtures and diarization annotations (AMI)  Inference on a recording of our real chatting with our mic. array Foundations of Blind Source Separation and Its Advances in Spatial Self-Supervised Learning /46 43
44. Demo: Separation and Diarization of “Japanese” Conversation Since neural FCASA involves the ISS algorithm, it is reasonably robust against language mismatch between the training and inference data. Foundations of Blind Source Separation and Its Advances in Spatial Self-Supervised Learning /46 44
45. Conclusion Two applications of deep source generative models: 1. Semi-supervised speech enhancement  FastMNMF-DP 2. Self-supervised source separation  Neural FCA Future work: • Handling unknown # of sources • Training neural FCA on diverse real audio recordings ∼ DNN Latent features Source PSD Source signal 𝑠𝑠𝑓𝑓𝑓𝑓 𝜆𝜆𝑓𝑓𝑓𝑓 𝑧𝑧𝑑𝑑𝑑𝑑 𝑔𝑔𝜃𝜃,𝑓𝑓 𝑧𝑧𝑡𝑡𝑡𝑡 ∼ 𝒩𝒩 0, 1 Foundations of Blind Source Separation and Its Advances in Spatial Self-Supervised Learning /46 45