Inverse-Free Online Independent Vector Analysis With Flexible Iterative Source Steering

APSIPA 2022
WedAM1-8

Transcript

1. Inverse-free Online Independent Vector Analysis with Flexible Iterative Source Steering APSIPA ASC 2022 WedAM1-8 Taishi Nakashima Nobutaka Ono Tokyo Metropolitan University, Japan. 9 November, 2022
2. Outline 1/13 Problem • Online blind source separation � Essential for real-time applications Methods • Online auxiliary-function-based independent vector analysis (AuxIVA) � Good separation performance Purpose • To utilize iterative source steering (ISS) for online AuxIVA � To update one specific steering vector
3. Contents 2/13 Introduction Conventional methods Frequency-domain BSS Batch AuxIVA Online AuxIVA Iterative source steering Proposed method: Online AuxIVA-ISS Experiment Conclusion
4. Multichannel blind source separation (BSS) 2/13 Blind Source Separation • To recover source signals from observed signals
5. Multichannel blind source separation (BSS) 2/13 Blind Source Separation • To recover source signals from observed signals
6. Multichannel blind source separation (BSS) 2/13 Blind Source Separation • To recover source signals from observed signals
7. Multichannel blind source separation (BSS) 2/13 Blind Source Separation Observed • To recover source signals from observed signals
8. Multichannel blind source separation (BSS) 2/13 Observed Estimated Blind Source Separation • To recover source signals from observed signals
9. Multichannel blind source separation (BSS) 2/13 Observed Estimated Blind Source Separation • To recover source signals from observed signals 🎺🎸🎹
10. Contents 3/13 Introduction Conventional methods Frequency-domain BSS Batch AuxIVA Online AuxIVA Iterative source steering Proposed method: Online AuxIVA-ISS Experiment Conclusion
11. Frequency-domain BSS 3/13 . . . Mix. system Demix. system Estimated Observed Source freq. time src. . . . Mixing system Observed 𝒙𝑓𝑡 = Mixing matrix 𝑨𝑓 𝒔𝑓𝑡 Demixing system Estimated 𝒚𝑓𝑡 = Demixing matrix 𝑾𝑓 𝒙𝑓𝑡
12. Frequency-domain BSS 3/13 . . . Mix. system Demix. system Estimated Observed Source freq. time src. . . . Mixing system Observed 𝒙𝑓𝑡 = Mixing matrix 𝑨𝑓 𝒔𝑓𝑡 Demixing system Estimated 𝒚𝑓𝑡 = Demixing matrix 𝑾𝑓 𝒙𝑓𝑡 Goal To estimate 𝑾𝑓 such that 𝒚𝑓𝑡 approximates 𝒔𝑓𝑡
13. Batch vs online 4/13 Estimated Observed Blind Source Separation Batch BSS � High separation performance � Weak against dynamic environment � Inappropriate for real-time Online BSS � Robust against dynamic environment � Limited separation performance � Complex parameter tuning
14. Batch vs online 4/13 Estimated Observed Blind Source Separation Batch BSS � High separation performance � Weak against dynamic environment � Inappropriate for real-time Online BSS � Robust against dynamic environment � Limited separation performance � Complex parameter tuning
15. Batch vs online 4/13 Estimated Observed Blind Source Separation Batch BSS � High separation performance � Weak against dynamic environment � Inappropriate for real-time Online BSS � Robust against dynamic environment � Limited separation performance � Complex parameter tuning
16. Batch vs online 4/13 Estimated Observed Blind Source Separation Batch BSS � High separation performance � Weak against dynamic environment � Inappropriate for real-time Online BSS � Robust against dynamic environment � Limited separation performance � Complex parameter tuning
17. Batch vs online 4/13 Estimated Observed Blind Source Separation Batch BSS � High separation performance � Weak against dynamic environment � Inappropriate for real-time Online BSS � Robust against dynamic environment � Limited separation performance � Complex parameter tuning
18. Batch AuxIVA [Ono2011] 5/13 Auxiliary function for IVA min. 𝐽+({𝑾𝑓 }𝑓 ) = ∑ 𝑓 [−2 log|det Parameter 𝑾𝑓 | + ∑ 𝑘 𝒘H 𝑘𝑓 𝑼𝑘𝑓 Parameter 𝒘𝑘𝑓 ] where 𝑼𝑘𝑓 = 1 𝑇 𝑇 ∑ 𝑡=1 𝒙𝑓𝑡 𝒙H 𝑓𝑡 2𝑟𝑘𝑓𝑡
19. Batch AuxIVA [Ono2011] 5/13 Auxiliary function for IVA min. 𝐽+({𝑾𝑓 }𝑓 ) = ∑ 𝑓 [−2 log|det Parameter 𝑾𝑓 | + ∑ 𝑘 𝒘H 𝑘𝑓 𝑼𝑘𝑓 Parameter 𝒘𝑘𝑓 ] where 𝑼𝑘𝑓 = 1 𝑇 𝑇 ∑ 𝑡=1 𝒙𝑓𝑡 𝒙H 𝑓𝑡 2𝑟𝑘𝑓𝑡 Iterative projection (IP) [Ono2011] • To minimize 𝐽+ w.r.t. 𝑾𝑓 𝒘𝑘𝑓 ← (𝑾𝑓 𝑼𝑘𝑓 )−1𝒆𝑘 𝒘𝑘𝑓 ← 𝒘𝑘𝑓 √𝒘H 𝑘𝑓 𝑼𝑘𝑓 𝒘𝑘𝑓
20. Batch AuxIVA [Ono2011] 5/13 Auxiliary function for IVA min. 𝐽+({𝑾𝑓 }𝑓 ) = ∑ 𝑓 [−2 log|det 𝑾𝑓 | + ∑ 𝑘 𝒘H 𝑘𝑓 𝑼𝑘𝑓 𝒘𝑘𝑓 ] where 𝑼𝑘𝑓 = 1 𝑇 𝑇 ∑ 𝑡=1 𝒙𝑓𝑡 𝒙H 𝑓𝑡 2𝑟𝑘𝑓𝑡 Iterative projection (IP) [Ono2011] • To minimize 𝐽+ w.r.t. 𝑾𝑓 𝒘𝑘𝑓 ← (𝑾𝑓 𝑼𝑘𝑓 )−1𝒆𝑘 𝒘𝑘𝑓 ← 𝒘𝑘𝑓 √𝒘H 𝑘𝑓 𝑼𝑘𝑓 𝒘𝑘𝑓 � Requires observed signals 𝒙𝑓𝑡 for all 𝑡 to calculate 𝑼𝑘𝑓
21. Online AuxIVA [Taniguchi+2014] 6/13 Modification of signal model 𝒙𝑓𝑡 = Time-variant 𝑨𝑓𝑡 𝒔𝑓𝑡 𝒚𝑓𝑡 = 𝑾𝑓𝑡 𝒙𝑓𝑡 Incremental update of covariance matrices 𝑼𝑘𝑓𝑡 ← Forgetting factor (0 < 𝛼 ≤ 1) 𝛼 Past data 𝑼𝑘𝑓(𝑡−1) + (1 − 𝛼) Current data 1 2𝑟𝑘𝑡 𝒙𝑓𝑡 𝒙H 𝑓𝑡
22. Online AuxIVA [Taniguchi+2014] 6/13 Modification of signal model 𝒙𝑓𝑡 = Time-variant 𝑨𝑓𝑡 𝒔𝑓𝑡 𝒚𝑓𝑡 = 𝑾𝑓𝑡 𝒙𝑓𝑡 Incremental update of covariance matrices 𝑼𝑘𝑓𝑡 ← Forgetting factor (0 < 𝛼 ≤ 1) 𝛼 Past data 𝑼𝑘𝑓(𝑡−1) + (1 − 𝛼) Current data 1 2𝑟𝑘𝑡 𝒙𝑓𝑡 𝒙H 𝑓𝑡 Motivation • To update 𝑾𝑓𝑡 efficiently ∘ 𝑾𝑓𝑡 should converge after sufficient frames
23. Online AuxIVA [Taniguchi+2014] 6/13 Modification of signal model 𝒙𝑓𝑡 = Time-variant 𝑨𝑓𝑡 𝒔𝑓𝑡 𝒚𝑓𝑡 = 𝑾𝑓𝑡 𝒙𝑓𝑡 Incremental update of covariance matrices 𝑼𝑘𝑓𝑡 ← Forgetting factor (0 < 𝛼 ≤ 1) 𝛼 Past data 𝑼𝑘𝑓(𝑡−1) + (1 − 𝛼) Current data 1 2𝑟𝑘𝑡 𝒙𝑓𝑡 𝒙H 𝑓𝑡 Motivation • To update 𝑾𝑓𝑡 efficiently ∘ 𝑾𝑓𝑡 should converge after sufficient frames � Exploit property of iterative source steering
24. Iterative source steering (ISS) [Scheibler+2020] 7/13 • Update rule of 𝑾𝑓 using elementary row operations 𝑾𝑓 ← 𝑾𝑓 − 𝒗𝑘𝑓 𝒘H 𝑘𝑓 𝑣𝑚𝑘𝑓 = ⎧ { ⎨ { ⎩ 𝒘H 𝑚𝑓 𝑼𝑚𝑓 𝒘𝑘𝑓 𝒘H 𝑘𝑓 𝑼𝑚𝑓 𝒘 𝑘𝑓 (if 𝑚 ≠ 𝑘) 1 − (𝒘H 𝑘𝑓 𝑼𝑘𝑓 𝒘𝑘𝑓 )−1 2 (if 𝑚 = 𝑘) � Inverse-free
25. Intuitive interpretation of ISS 8/13 • Equivalent to updates of steering vectors 𝒂𝑘𝑓 [Scheibler+2020] 𝒂𝑘𝑓 ← 1 1−𝑣𝑘𝑘𝑓 (𝒂𝑘𝑓 + ∑ 𝑚≠𝑘 𝑣𝑚𝑘𝑓 𝒂𝑚𝑓 )
26. Intuitive interpretation of ISS 8/13 • Equivalent to updates of steering vectors 𝒂𝑘𝑓 [Scheibler+2020] 𝒂𝑘𝑓 ← 1 1−𝑣𝑘𝑘𝑓 (𝒂𝑘𝑓 + ∑ 𝑚≠𝑘 𝑣𝑚𝑘𝑓 𝒂𝑚𝑓 )
27. Intuitive interpretation of ISS 8/13 • Equivalent to updates of steering vectors 𝒂𝑘𝑓 [Scheibler+2020] 𝒂𝑘𝑓 ← 1 1−𝑣𝑘𝑘𝑓 (𝒂𝑘𝑓 + ∑ 𝑚≠𝑘 𝑣𝑚𝑘𝑓 𝒂𝑚𝑓 ) � Example: after source 𝑠1𝑓𝑡 moves...
28. Intuitive interpretation of ISS 8/13 • Equivalent to updates of steering vectors 𝒂𝑘𝑓 [Scheibler+2020] 𝒂𝑘𝑓 ← 1 1−𝑣𝑘𝑘𝑓 (𝒂𝑘𝑓 + ∑ 𝑚≠𝑘 𝑣𝑚𝑘𝑓 𝒂𝑚𝑓 ) � Example: after source 𝑠1𝑓𝑡 moves... � IP: update 𝒘2𝑓𝑡 , 𝒘3𝑓𝑡 , …
29. Intuitive interpretation of ISS 8/13 • Equivalent to updates of steering vectors 𝒂𝑘𝑓 [Scheibler+2020] 𝒂𝑘𝑓 ← 1 1−𝑣𝑘𝑘𝑓 (𝒂𝑘𝑓 + ∑ 𝑚≠𝑘 𝑣𝑚𝑘𝑓 𝒂𝑚𝑓 ) � Example: after source 𝑠1𝑓𝑡 moves... � IP: update 𝒘2𝑓𝑡 , 𝒘3𝑓𝑡 , … � ISS: update only 𝒂1𝑓𝑡
30. Contents 9/13 Introduction Conventional methods Frequency-domain BSS Batch AuxIVA Online AuxIVA Iterative source steering Proposed method: Online AuxIVA-ISS Experiment Conclusion
31. Online AuxIVA with ISSPROPOSED 9/13 for 𝑡 = 1, … , 𝑇 do 𝑾𝑓𝑡 ← 𝑾𝑓(𝑡−1) (∀𝑓) for it = 1, … , 𝑁it do for 𝑘 = 1, … , 𝐾 do 𝑟𝑘𝑡 ← √∑ 𝑓 |𝒘H 𝑘𝑓𝑡 𝒙𝑓𝑡 |2 𝑼𝑘𝑓𝑡 ← 𝛼𝑼𝑘𝑓(𝑡−1) + (1 − 𝛼) 1 2𝑟𝑘𝑡 𝒙𝑓𝑡 𝒙H 𝑓𝑡 (∀𝑓) for 𝑘 ∈ 𝒦 do for 𝑚 = 1, … , 𝐾 do 𝑣𝑚𝑘𝑓 ← ⎧ { ⎨ { ⎩ 𝒘H 𝑚𝑓 𝑼𝑚𝑓 𝒘𝑘𝑓 𝒘H 𝑘𝑓 𝑼𝑚𝑓 𝒘 𝑘𝑓 if 𝑚 ≠ 𝑘 1 − (𝒘H 𝑘𝑓 𝑼𝑘𝑓 𝒘𝑘𝑓 )−1 2 if 𝑚 = 𝑘 (∀𝑓) 𝑾𝑓𝑡 ← 𝑾𝑓𝑡 − 𝒗𝑘𝑓 𝒘H 𝑘𝑓𝑡 (∀𝑓) 𝒚𝑓𝑡 = 𝑾𝑓𝑡 𝒙𝑓𝑡 (∀𝑓)
32. Online AuxIVA with ISSPROPOSED 9/13 for 𝑡 = 1, … , 𝑇 do 𝑾𝑓𝑡 ← 𝑾𝑓(𝑡−1) (∀𝑓) for it = 1, … , 𝑁it do for 𝑘 = 1, … , 𝐾 do 𝑟𝑘𝑡 ← √∑ 𝑓 |𝒘H 𝑘𝑓𝑡 𝒙𝑓𝑡 |2 𝑼𝑘𝑓𝑡 ← 𝛼𝑼𝑘𝑓(𝑡−1) + (1 − 𝛼) 1 2𝑟𝑘𝑡 𝒙𝑓𝑡 𝒙H 𝑓𝑡 (∀𝑓) for Source index set to be udpated 𝑘 ∈ 𝒦 do for 𝑚 = 1, … , 𝐾 do 𝑣𝑚𝑘𝑓 ← ⎧ { ⎨ { ⎩ 𝒘H 𝑚𝑓 𝑼𝑚𝑓 𝒘𝑘𝑓 𝒘H 𝑘𝑓 𝑼𝑚𝑓 𝒘 𝑘𝑓 if 𝑚 ≠ 𝑘 1 − (𝒘H 𝑘𝑓 𝑼𝑘𝑓 𝒘𝑘𝑓 )−1 2 if 𝑚 = 𝑘 (∀𝑓) 𝑾𝑓𝑡 ← 𝑾𝑓𝑡 − 𝒗𝑘𝑓 𝒘H 𝑘𝑓𝑡 (∀𝑓) 𝒚𝑓𝑡 = 𝑾𝑓𝑡 𝒙𝑓𝑡 (∀𝑓)
33. Online AuxIVA with ISSPROPOSED 9/13 for 𝑡 = 1, … , 𝑇 do 𝑾𝑓𝑡 ← 𝑾𝑓(𝑡−1) (∀𝑓) for it = 1, … , 𝑁it do for 𝑘 = 1, … , 𝐾 do 𝑟𝑘𝑡 ← √∑ 𝑓 |𝒘H 𝑘𝑓𝑡 𝒙𝑓𝑡 |2 𝑼𝑘𝑓𝑡 ← 𝛼𝑼𝑘𝑓(𝑡−1) + (1 − 𝛼) 1 2𝑟𝑘𝑡 𝒙𝑓𝑡 𝒙H 𝑓𝑡 (∀𝑓) for Source index set to be udpated Example: Until 𝑾𝑓𝑡 converge 𝒦 = {1, … , 𝐾} After 𝑾𝑓𝑡 converge 𝒦 = ∅ When source 𝑙 moves 𝒦 = {𝑙} 𝑘 ∈ 𝒦 do for 𝑚 = 1, … , 𝐾 do 𝑣𝑚𝑘𝑓 ← ⎧ { ⎨ { ⎩ 𝒘H 𝑚𝑓 𝑼𝑚𝑓 𝒘𝑘𝑓 𝒘H 𝑘𝑓 𝑼𝑚𝑓 𝒘 𝑘𝑓 if 𝑚 ≠ 𝑘 1 − (𝒘H 𝑘𝑓 𝑼𝑘𝑓 𝒘𝑘𝑓 )−1 2 if 𝑚 = 𝑘 (∀𝑓) 𝑾𝑓𝑡 ← 𝑾𝑓𝑡 − 𝒗𝑘𝑓 𝒘H 𝑘𝑓𝑡 (∀𝑓) 𝒚𝑓𝑡 = 𝑾𝑓𝑡 𝒙𝑓𝑡 (∀𝑓)
34. Contents 10/13 Introduction Conventional methods Frequency-domain BSS Batch AuxIVA Online AuxIVA Iterative source steering Proposed method: Online AuxIVA-ISS Experiment Conclusion
35. Setup 10/13 Room layout Width = �.� m Depth = �.� m � � � Height = �.�� m Microphones Source (fixed) Source (moved)
36. Setup 10/13 Room layout Width = �.� m Depth = �.� m � � � �` Height = �.�� m Microphones Source (fixed) Source (moved) 1. Copy source 3 to source 3′
37. Setup 10/13 Room layout Width = �.� m Depth = �.� m � � � �` Height = �.�� m Microphones Source (fixed) Source (moved) 1. Copy source 3 to source 3′ 2. Cut second half of source 3 Cut first half of source 3′
38. Setup 10/13 Room layout Width = �.� m Depth = �.� m � � � �` Height = �.�� m Microphones Source (fixed) Source (moved) 1. Copy source 3 to source 3′ 2. Cut second half of source 3 Cut first half of source 3′ � Instant movement of the source 3
39. Setup 10/13 Room layout Width = �.� m Depth = �.� m � � � �` Height = �.�� m Microphones Source (fixed) Source (moved) 1. Copy source 3 to source 3′ 2. Cut second half of source 3 Cut first half of source 3′ � Instant movement of the source 3 Methods Name Target source indices 𝒦 Before move After move 🆕 ISS-all {1, 2, 3} {1, 2, 3} 🆕 ISS-one {1, 2, 3} {3} IP-all {1, 2, 3} {1, 2, 3} IP-one {1, 2, 3} {3}
40. Setup 10/13 Room layout Width = �.� m Depth = �.� m � � � �` Height = �.�� m Microphones Source (fixed) Source (moved) 1. Copy source 3 to source 3′ 2. Cut second half of source 3 Cut first half of source 3′ � Instant movement of the source 3 Methods Name Target source indices 𝒦 Before move After move 🆕 ISS-all {1, 2, 3} {1, 2, 3} 🆕 ISS-one {1, 2, 3} {3} IP-all {1, 2, 3} {1, 2, 3} IP-one {1, 2, 3} {3} Note When and which source moves is known as an oracle in this experiment
41. Separation performance 11/13 0 20 ISS-all ISS-one IP-all IP-one 0 20 0 5 10 15 20 25 30 Segment index 0 20 SegSDR improvement (dB) $I
     $I
     $I
     $I
     $I
     $I
     • Segmental SDR improvements by channels
42. Separation performance 11/13 0 20 ISS-all ISS-one IP-all IP-one 0 20 0 5 10 15 20 25 30 Segment index 0 20 Source 3 moved SegSDR improvement (dB) $I
     $I
     $I
     $I
     $I
     $I
     • Vertical dashed line indicates movement of source 3
43. Separation performance 11/13 0 20 ISS-all ISS-one IP-all IP-one 0 20 0 5 10 15 20 25 30 Segment index 0 20 Source 3 moved SegSDR improvement (dB) $I
     $I
     $I
     $I
     $I
     $I
     • Before source 3 moved: update demixing matrices for all source indices
44. Separation performance 11/13 0 20 ISS-all ISS-one IP-all IP-one 0 20 0 5 10 15 20 25 30 Segment index 0 20 Source 3 moved SegSDR improvement (dB) $I
     $I
     $I
     $I
     $I
     $I
     • After source 3 moved: ISS-one updates only 𝒂3𝑓 , IP-one updates only 𝒘3𝑓
45. Separation performance 11/13 0 20 ISS-all ISS-one IP-all IP-one 0 20 0 5 10 15 20 25 30 Segment index 0 20 Source 3 moved SegSDR improvement (dB) $I
     $I
     $I
     $I
     $I
     $I
     • Degraded separation performance due to moving source
46. Separation performance 11/13 0 20 ISS-all ISS-one IP-all IP-one 0 20 0 5 10 15 20 25 30 Segment index 0 20 Source 3 moved SegSDR improvement (dB) $I
     $I
     $I
     $I
     $I
     $I
     • Steadily improved thanks to online updates
47. Separation performance 11/13 0 20 ISS-all ISS-one IP-all IP-one 0 20 0 5 10 15 20 25 30 Segment index 0 20 Source 3 moved SegSDR improvement (dB) $I
     $I
     $I
     $I
     $I
     $I
     � ISS-one : almost the same performance as ISS-all with less updates
48. Separation performance 11/13 0 20 ISS-all ISS-one IP-all IP-one 0 20 0 5 10 15 20 25 30 Segment index 0 20 Source 3 moved SegSDR improvement (dB) $I
     $I
     $I
     $I
     $I
     $I
     � ISS-one : higher performance than IP-one
49. Runtime 12/13 IP-all ISS-all ISS-one 0 2 4 6 8 10 12 Runtime (s) 11.54 11.00 8.66 � ISS-one was faster than IP-all
50. Contents 13/13 Introduction Conventional methods Frequency-domain BSS Batch AuxIVA Online AuxIVA Iterative source steering Proposed method: Online AuxIVA-ISS Experiment Conclusion
51. Conclusion 13/13 Summary • New online AuxIVA with iterative source steering (ISS) � Good separation performance under dynamic environment � Efficient update when the single source moves
52. Conclusion 13/13 Summary • New online AuxIVA with iterative source steering (ISS) � Good separation performance under dynamic environment � Efficient update when the single source moves Future work • Automatic detection of moveing sources • Efficient forgetting factor tuning • Efficient update rule of ISS
53. References i [Ono2011] N. Ono, “Stable and fast update rules for independent vector analysis based on auxiliary function technique,” in Proceedings of IEEE Workshop on Applications of Signal Processing to Audio and Acoustics (WASPAA), pp. 189–192, 2011. [Scheibler+2020] R. Scheibler and N. Ono, “Fast and stable blind source separation with rank-1 updates,” in Proceedings of IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), May 2020. [Taniguchi+2014] T. Taniguchi, N. Ono, A. Kawamura, and S. Sagayama, “An auxiliary-function approach to online independent vector analysis for real-time blind source separation,” in Proceedings of Hands-Free Speech Communication and Microphone Arrays (HSCMA), pp. 107–111, May 2014. Thank you!