$30 off During Our Annual Pro Sale. View Details »

Inverse-Free Online Independent Vector Analysis With Flexible Iterative Source Steering

Inverse-Free Online Independent Vector Analysis With Flexible Iterative Source Steering

APSIPA 2022
WedAM1-8

Taishi Nakashima

November 09, 2022
Tweet

More Decks by Taishi Nakashima

Other Decks in Research

Transcript

  1. Inverse-free Online Independent Vector Analysis with Flexible Iterative Source Steering

    APSIPA ASC 2022 WedAM1-8 Taishi Nakashima Nobutaka Ono Tokyo Metropolitan University, Japan. 9 November, 2022
  2. Outline 1/13 Problem • Online blind source separation � Essential

    for real-time applications Methods • Online auxiliary-function-based independent vector analysis (AuxIVA) � Good separation performance Purpose • To utilize iterative source steering (ISS) for online AuxIVA � To update one specific steering vector
  3. Contents 2/13 Introduction Conventional methods Frequency-domain BSS Batch AuxIVA Online

    AuxIVA Iterative source steering Proposed method: Online AuxIVA-ISS Experiment Conclusion
  4. Multichannel blind source separation (BSS) 2/13 Blind Source Separation •

    To recover source signals from observed signals
  5. Multichannel blind source separation (BSS) 2/13 Blind Source Separation •

    To recover source signals from observed signals
  6. Multichannel blind source separation (BSS) 2/13 Blind Source Separation •

    To recover source signals from observed signals
  7. Multichannel blind source separation (BSS) 2/13 Blind Source Separation Observed

    • To recover source signals from observed signals
  8. Multichannel blind source separation (BSS) 2/13 Observed Estimated Blind Source

    Separation • To recover source signals from observed signals
  9. Multichannel blind source separation (BSS) 2/13 Observed Estimated Blind Source

    Separation • To recover source signals from observed signals 🎺🎸🎹
  10. Contents 3/13 Introduction Conventional methods Frequency-domain BSS Batch AuxIVA Online

    AuxIVA Iterative source steering Proposed method: Online AuxIVA-ISS Experiment Conclusion
  11. Frequency-domain BSS 3/13 . . . Mix. system Demix. system

    Estimated Observed Source freq. time src. . . . Mixing system Observed 𝒙𝑓𝑡 = Mixing matrix 𝑨𝑓 𝒔𝑓𝑡 Demixing system Estimated 𝒚𝑓𝑡 = Demixing matrix 𝑾𝑓 𝒙𝑓𝑡
  12. Frequency-domain BSS 3/13 . . . Mix. system Demix. system

    Estimated Observed Source freq. time src. . . . Mixing system Observed 𝒙𝑓𝑡 = Mixing matrix 𝑨𝑓 𝒔𝑓𝑡 Demixing system Estimated 𝒚𝑓𝑡 = Demixing matrix 𝑾𝑓 𝒙𝑓𝑡 Goal To estimate 𝑾𝑓 such that 𝒚𝑓𝑡 approximates 𝒔𝑓𝑡
  13. Batch vs online 4/13 Estimated Observed Blind Source Separation Batch

    BSS � High separation performance � Weak against dynamic environment � Inappropriate for real-time Online BSS � Robust against dynamic environment � Limited separation performance � Complex parameter tuning
  14. Batch vs online 4/13 Estimated Observed Blind Source Separation Batch

    BSS � High separation performance � Weak against dynamic environment � Inappropriate for real-time Online BSS � Robust against dynamic environment � Limited separation performance � Complex parameter tuning
  15. Batch vs online 4/13 Estimated Observed Blind Source Separation Batch

    BSS � High separation performance � Weak against dynamic environment � Inappropriate for real-time Online BSS � Robust against dynamic environment � Limited separation performance � Complex parameter tuning
  16. Batch vs online 4/13 Estimated Observed Blind Source Separation Batch

    BSS � High separation performance � Weak against dynamic environment � Inappropriate for real-time Online BSS � Robust against dynamic environment � Limited separation performance � Complex parameter tuning
  17. Batch vs online 4/13 Estimated Observed Blind Source Separation Batch

    BSS � High separation performance � Weak against dynamic environment � Inappropriate for real-time Online BSS � Robust against dynamic environment � Limited separation performance � Complex parameter tuning
  18. Batch AuxIVA [Ono2011] 5/13 Auxiliary function for IVA min. 𝐽+({𝑾𝑓

    }𝑓 ) = ∑ 𝑓 [−2 log|det Parameter 𝑾𝑓 | + ∑ 𝑘 𝒘H 𝑘𝑓 𝑼𝑘𝑓 Parameter 𝒘𝑘𝑓 ] where 𝑼𝑘𝑓 = 1 𝑇 𝑇 ∑ 𝑡=1 𝒙𝑓𝑡 𝒙H 𝑓𝑡 2𝑟𝑘𝑓𝑡
  19. Batch AuxIVA [Ono2011] 5/13 Auxiliary function for IVA min. 𝐽+({𝑾𝑓

    }𝑓 ) = ∑ 𝑓 [−2 log|det Parameter 𝑾𝑓 | + ∑ 𝑘 𝒘H 𝑘𝑓 𝑼𝑘𝑓 Parameter 𝒘𝑘𝑓 ] where 𝑼𝑘𝑓 = 1 𝑇 𝑇 ∑ 𝑡=1 𝒙𝑓𝑡 𝒙H 𝑓𝑡 2𝑟𝑘𝑓𝑡 Iterative projection (IP) [Ono2011] • To minimize 𝐽+ w.r.t. 𝑾𝑓 𝒘𝑘𝑓 ← (𝑾𝑓 𝑼𝑘𝑓 )−1𝒆𝑘 𝒘𝑘𝑓 ← 𝒘𝑘𝑓 √𝒘H 𝑘𝑓 𝑼𝑘𝑓 𝒘𝑘𝑓
  20. Batch AuxIVA [Ono2011] 5/13 Auxiliary function for IVA min. 𝐽+({𝑾𝑓

    }𝑓 ) = ∑ 𝑓 [−2 log|det 𝑾𝑓 | + ∑ 𝑘 𝒘H 𝑘𝑓 𝑼𝑘𝑓 𝒘𝑘𝑓 ] where 𝑼𝑘𝑓 = 1 𝑇 𝑇 ∑ 𝑡=1 𝒙𝑓𝑡 𝒙H 𝑓𝑡 2𝑟𝑘𝑓𝑡 Iterative projection (IP) [Ono2011] • To minimize 𝐽+ w.r.t. 𝑾𝑓 𝒘𝑘𝑓 ← (𝑾𝑓 𝑼𝑘𝑓 )−1𝒆𝑘 𝒘𝑘𝑓 ← 𝒘𝑘𝑓 √𝒘H 𝑘𝑓 𝑼𝑘𝑓 𝒘𝑘𝑓 � Requires observed signals 𝒙𝑓𝑡 for all 𝑡 to calculate 𝑼𝑘𝑓
  21. Online AuxIVA [Taniguchi+2014] 6/13 Modification of signal model 𝒙𝑓𝑡 =

    Time-variant 𝑨𝑓𝑡 𝒔𝑓𝑡 𝒚𝑓𝑡 = 𝑾𝑓𝑡 𝒙𝑓𝑡 Incremental update of covariance matrices 𝑼𝑘𝑓𝑡 ← Forgetting factor (0 < 𝛼 ≤ 1) 𝛼 Past data 𝑼𝑘𝑓(𝑡−1) + (1 − 𝛼) Current data 1 2𝑟𝑘𝑡 𝒙𝑓𝑡 𝒙H 𝑓𝑡
  22. Online AuxIVA [Taniguchi+2014] 6/13 Modification of signal model 𝒙𝑓𝑡 =

    Time-variant 𝑨𝑓𝑡 𝒔𝑓𝑡 𝒚𝑓𝑡 = 𝑾𝑓𝑡 𝒙𝑓𝑡 Incremental update of covariance matrices 𝑼𝑘𝑓𝑡 ← Forgetting factor (0 < 𝛼 ≤ 1) 𝛼 Past data 𝑼𝑘𝑓(𝑡−1) + (1 − 𝛼) Current data 1 2𝑟𝑘𝑡 𝒙𝑓𝑡 𝒙H 𝑓𝑡 Motivation • To update 𝑾𝑓𝑡 efficiently ∘ 𝑾𝑓𝑡 should converge after sufficient frames
  23. Online AuxIVA [Taniguchi+2014] 6/13 Modification of signal model 𝒙𝑓𝑡 =

    Time-variant 𝑨𝑓𝑡 𝒔𝑓𝑡 𝒚𝑓𝑡 = 𝑾𝑓𝑡 𝒙𝑓𝑡 Incremental update of covariance matrices 𝑼𝑘𝑓𝑡 ← Forgetting factor (0 < 𝛼 ≤ 1) 𝛼 Past data 𝑼𝑘𝑓(𝑡−1) + (1 − 𝛼) Current data 1 2𝑟𝑘𝑡 𝒙𝑓𝑡 𝒙H 𝑓𝑡 Motivation • To update 𝑾𝑓𝑡 efficiently ∘ 𝑾𝑓𝑡 should converge after sufficient frames � Exploit property of iterative source steering
  24. Iterative source steering (ISS) [Scheibler+2020] 7/13 • Update rule of

    𝑾𝑓 using elementary row operations 𝑾𝑓 ← 𝑾𝑓 − 𝒗𝑘𝑓 𝒘H 𝑘𝑓 𝑣𝑚𝑘𝑓 = ⎧ { ⎨ { ⎩ 𝒘H 𝑚𝑓 𝑼𝑚𝑓 𝒘𝑘𝑓 𝒘H 𝑘𝑓 𝑼𝑚𝑓 𝒘 𝑘𝑓 (if 𝑚 ≠ 𝑘) 1 − (𝒘H 𝑘𝑓 𝑼𝑘𝑓 𝒘𝑘𝑓 )−1 2 (if 𝑚 = 𝑘) � Inverse-free
  25. Intuitive interpretation of ISS 8/13 • Equivalent to updates of

    steering vectors 𝒂𝑘𝑓 [Scheibler+2020] 𝒂𝑘𝑓 ← 1 1−𝑣𝑘𝑘𝑓 (𝒂𝑘𝑓 + ∑ 𝑚≠𝑘 𝑣𝑚𝑘𝑓 𝒂𝑚𝑓 )
  26. Intuitive interpretation of ISS 8/13 • Equivalent to updates of

    steering vectors 𝒂𝑘𝑓 [Scheibler+2020] 𝒂𝑘𝑓 ← 1 1−𝑣𝑘𝑘𝑓 (𝒂𝑘𝑓 + ∑ 𝑚≠𝑘 𝑣𝑚𝑘𝑓 𝒂𝑚𝑓 )
  27. Intuitive interpretation of ISS 8/13 • Equivalent to updates of

    steering vectors 𝒂𝑘𝑓 [Scheibler+2020] 𝒂𝑘𝑓 ← 1 1−𝑣𝑘𝑘𝑓 (𝒂𝑘𝑓 + ∑ 𝑚≠𝑘 𝑣𝑚𝑘𝑓 𝒂𝑚𝑓 ) � Example: after source 𝑠1𝑓𝑡 moves...
  28. Intuitive interpretation of ISS 8/13 • Equivalent to updates of

    steering vectors 𝒂𝑘𝑓 [Scheibler+2020] 𝒂𝑘𝑓 ← 1 1−𝑣𝑘𝑘𝑓 (𝒂𝑘𝑓 + ∑ 𝑚≠𝑘 𝑣𝑚𝑘𝑓 𝒂𝑚𝑓 ) � Example: after source 𝑠1𝑓𝑡 moves... � IP: update 𝒘2𝑓𝑡 , 𝒘3𝑓𝑡 , …
  29. Intuitive interpretation of ISS 8/13 • Equivalent to updates of

    steering vectors 𝒂𝑘𝑓 [Scheibler+2020] 𝒂𝑘𝑓 ← 1 1−𝑣𝑘𝑘𝑓 (𝒂𝑘𝑓 + ∑ 𝑚≠𝑘 𝑣𝑚𝑘𝑓 𝒂𝑚𝑓 ) � Example: after source 𝑠1𝑓𝑡 moves... � IP: update 𝒘2𝑓𝑡 , 𝒘3𝑓𝑡 , … � ISS: update only 𝒂1𝑓𝑡
  30. Contents 9/13 Introduction Conventional methods Frequency-domain BSS Batch AuxIVA Online

    AuxIVA Iterative source steering Proposed method: Online AuxIVA-ISS Experiment Conclusion
  31. Online AuxIVA with ISSPROPOSED 9/13 for 𝑡 = 1, …

    , 𝑇 do 𝑾𝑓𝑡 ← 𝑾𝑓(𝑡−1) (∀𝑓) for it = 1, … , 𝑁it do for 𝑘 = 1, … , 𝐾 do 𝑟𝑘𝑡 ← √∑ 𝑓 |𝒘H 𝑘𝑓𝑡 𝒙𝑓𝑡 |2 𝑼𝑘𝑓𝑡 ← 𝛼𝑼𝑘𝑓(𝑡−1) + (1 − 𝛼) 1 2𝑟𝑘𝑡 𝒙𝑓𝑡 𝒙H 𝑓𝑡 (∀𝑓) for 𝑘 ∈ 𝒦 do for 𝑚 = 1, … , 𝐾 do 𝑣𝑚𝑘𝑓 ← ⎧ { ⎨ { ⎩ 𝒘H 𝑚𝑓 𝑼𝑚𝑓 𝒘𝑘𝑓 𝒘H 𝑘𝑓 𝑼𝑚𝑓 𝒘 𝑘𝑓 if 𝑚 ≠ 𝑘 1 − (𝒘H 𝑘𝑓 𝑼𝑘𝑓 𝒘𝑘𝑓 )−1 2 if 𝑚 = 𝑘 (∀𝑓) 𝑾𝑓𝑡 ← 𝑾𝑓𝑡 − 𝒗𝑘𝑓 𝒘H 𝑘𝑓𝑡 (∀𝑓) 𝒚𝑓𝑡 = 𝑾𝑓𝑡 𝒙𝑓𝑡 (∀𝑓)
  32. Online AuxIVA with ISSPROPOSED 9/13 for 𝑡 = 1, …

    , 𝑇 do 𝑾𝑓𝑡 ← 𝑾𝑓(𝑡−1) (∀𝑓) for it = 1, … , 𝑁it do for 𝑘 = 1, … , 𝐾 do 𝑟𝑘𝑡 ← √∑ 𝑓 |𝒘H 𝑘𝑓𝑡 𝒙𝑓𝑡 |2 𝑼𝑘𝑓𝑡 ← 𝛼𝑼𝑘𝑓(𝑡−1) + (1 − 𝛼) 1 2𝑟𝑘𝑡 𝒙𝑓𝑡 𝒙H 𝑓𝑡 (∀𝑓) for Source index set to be udpated 𝑘 ∈ 𝒦 do for 𝑚 = 1, … , 𝐾 do 𝑣𝑚𝑘𝑓 ← ⎧ { ⎨ { ⎩ 𝒘H 𝑚𝑓 𝑼𝑚𝑓 𝒘𝑘𝑓 𝒘H 𝑘𝑓 𝑼𝑚𝑓 𝒘 𝑘𝑓 if 𝑚 ≠ 𝑘 1 − (𝒘H 𝑘𝑓 𝑼𝑘𝑓 𝒘𝑘𝑓 )−1 2 if 𝑚 = 𝑘 (∀𝑓) 𝑾𝑓𝑡 ← 𝑾𝑓𝑡 − 𝒗𝑘𝑓 𝒘H 𝑘𝑓𝑡 (∀𝑓) 𝒚𝑓𝑡 = 𝑾𝑓𝑡 𝒙𝑓𝑡 (∀𝑓)
  33. Online AuxIVA with ISSPROPOSED 9/13 for 𝑡 = 1, …

    , 𝑇 do 𝑾𝑓𝑡 ← 𝑾𝑓(𝑡−1) (∀𝑓) for it = 1, … , 𝑁it do for 𝑘 = 1, … , 𝐾 do 𝑟𝑘𝑡 ← √∑ 𝑓 |𝒘H 𝑘𝑓𝑡 𝒙𝑓𝑡 |2 𝑼𝑘𝑓𝑡 ← 𝛼𝑼𝑘𝑓(𝑡−1) + (1 − 𝛼) 1 2𝑟𝑘𝑡 𝒙𝑓𝑡 𝒙H 𝑓𝑡 (∀𝑓) for Source index set to be udpated Example: Until 𝑾𝑓𝑡 converge 𝒦 = {1, … , 𝐾} After 𝑾𝑓𝑡 converge 𝒦 = ∅ When source 𝑙 moves 𝒦 = {𝑙} 𝑘 ∈ 𝒦 do for 𝑚 = 1, … , 𝐾 do 𝑣𝑚𝑘𝑓 ← ⎧ { ⎨ { ⎩ 𝒘H 𝑚𝑓 𝑼𝑚𝑓 𝒘𝑘𝑓 𝒘H 𝑘𝑓 𝑼𝑚𝑓 𝒘 𝑘𝑓 if 𝑚 ≠ 𝑘 1 − (𝒘H 𝑘𝑓 𝑼𝑘𝑓 𝒘𝑘𝑓 )−1 2 if 𝑚 = 𝑘 (∀𝑓) 𝑾𝑓𝑡 ← 𝑾𝑓𝑡 − 𝒗𝑘𝑓 𝒘H 𝑘𝑓𝑡 (∀𝑓) 𝒚𝑓𝑡 = 𝑾𝑓𝑡 𝒙𝑓𝑡 (∀𝑓)
  34. Contents 10/13 Introduction Conventional methods Frequency-domain BSS Batch AuxIVA Online

    AuxIVA Iterative source steering Proposed method: Online AuxIVA-ISS Experiment Conclusion
  35. Setup 10/13 Room layout Width = �.� m Depth =

    �.� m � � � Height = �.�� m Microphones Source (fixed) Source (moved)
  36. Setup 10/13 Room layout Width = �.� m Depth =

    �.� m � � � �` Height = �.�� m Microphones Source (fixed) Source (moved) 1. Copy source 3 to source 3′
  37. Setup 10/13 Room layout Width = �.� m Depth =

    �.� m � � � �` Height = �.�� m Microphones Source (fixed) Source (moved) 1. Copy source 3 to source 3′ 2. Cut second half of source 3 Cut first half of source 3′
  38. Setup 10/13 Room layout Width = �.� m Depth =

    �.� m � � � �` Height = �.�� m Microphones Source (fixed) Source (moved) 1. Copy source 3 to source 3′ 2. Cut second half of source 3 Cut first half of source 3′ � Instant movement of the source 3
  39. Setup 10/13 Room layout Width = �.� m Depth =

    �.� m � � � �` Height = �.�� m Microphones Source (fixed) Source (moved) 1. Copy source 3 to source 3′ 2. Cut second half of source 3 Cut first half of source 3′ � Instant movement of the source 3 Methods Name Target source indices 𝒦 Before move After move 🆕 ISS-all {1, 2, 3} {1, 2, 3} 🆕 ISS-one {1, 2, 3} {3} IP-all {1, 2, 3} {1, 2, 3} IP-one {1, 2, 3} {3}
  40. Setup 10/13 Room layout Width = �.� m Depth =

    �.� m � � � �` Height = �.�� m Microphones Source (fixed) Source (moved) 1. Copy source 3 to source 3′ 2. Cut second half of source 3 Cut first half of source 3′ � Instant movement of the source 3 Methods Name Target source indices 𝒦 Before move After move 🆕 ISS-all {1, 2, 3} {1, 2, 3} 🆕 ISS-one {1, 2, 3} {3} IP-all {1, 2, 3} {1, 2, 3} IP-one {1, 2, 3} {3} Note When and which source moves is known as an oracle in this experiment
  41. Separation performance 11/13 0 20 ISS-all ISS-one IP-all IP-one 0

    20 0 5 10 15 20 25 30 Segment index 0 20 SegSDR improvement (dB) $I $I $I $I $I $I • Segmental SDR improvements by channels
  42. Separation performance 11/13 0 20 ISS-all ISS-one IP-all IP-one 0

    20 0 5 10 15 20 25 30 Segment index 0 20 Source 3 moved SegSDR improvement (dB) $I $I $I $I $I $I • Vertical dashed line indicates movement of source 3
  43. Separation performance 11/13 0 20 ISS-all ISS-one IP-all IP-one 0

    20 0 5 10 15 20 25 30 Segment index 0 20 Source 3 moved SegSDR improvement (dB) $I $I $I $I $I $I • Before source 3 moved: update demixing matrices for all source indices
  44. Separation performance 11/13 0 20 ISS-all ISS-one IP-all IP-one 0

    20 0 5 10 15 20 25 30 Segment index 0 20 Source 3 moved SegSDR improvement (dB) $I $I $I $I $I $I • After source 3 moved: ISS-one updates only 𝒂3𝑓 , IP-one updates only 𝒘3𝑓
  45. Separation performance 11/13 0 20 ISS-all ISS-one IP-all IP-one 0

    20 0 5 10 15 20 25 30 Segment index 0 20 Source 3 moved SegSDR improvement (dB) $I $I $I $I $I $I • Degraded separation performance due to moving source
  46. Separation performance 11/13 0 20 ISS-all ISS-one IP-all IP-one 0

    20 0 5 10 15 20 25 30 Segment index 0 20 Source 3 moved SegSDR improvement (dB) $I $I $I $I $I $I • Steadily improved thanks to online updates
  47. Separation performance 11/13 0 20 ISS-all ISS-one IP-all IP-one 0

    20 0 5 10 15 20 25 30 Segment index 0 20 Source 3 moved SegSDR improvement (dB) $I $I $I $I $I $I � ISS-one : almost the same performance as ISS-all with less updates
  48. Separation performance 11/13 0 20 ISS-all ISS-one IP-all IP-one 0

    20 0 5 10 15 20 25 30 Segment index 0 20 Source 3 moved SegSDR improvement (dB) $I $I $I $I $I $I � ISS-one : higher performance than IP-one
  49. Runtime 12/13 IP-all ISS-all ISS-one 0 2 4 6 8

    10 12 Runtime (s) 11.54 11.00 8.66 � ISS-one was faster than IP-all
  50. Contents 13/13 Introduction Conventional methods Frequency-domain BSS Batch AuxIVA Online

    AuxIVA Iterative source steering Proposed method: Online AuxIVA-ISS Experiment Conclusion
  51. Conclusion 13/13 Summary • New online AuxIVA with iterative source

    steering (ISS) � Good separation performance under dynamic environment � Efficient update when the single source moves
  52. Conclusion 13/13 Summary • New online AuxIVA with iterative source

    steering (ISS) � Good separation performance under dynamic environment � Efficient update when the single source moves Future work • Automatic detection of moveing sources • Efficient forgetting factor tuning • Efficient update rule of ISS
  53. References i [Ono2011] N. Ono, “Stable and fast update rules

    for independent vector analysis based on auxiliary function technique,” in Proceedings of IEEE Workshop on Applications of Signal Processing to Audio and Acoustics (WASPAA), pp. 189–192, 2011. [Scheibler+2020] R. Scheibler and N. Ono, “Fast and stable blind source separation with rank-1 updates,” in Proceedings of IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), May 2020. [Taniguchi+2014] T. Taniguchi, N. Ono, A. Kawamura, and S. Sagayama, “An auxiliary-function approach to online independent vector analysis for real-time blind source separation,” in Proceedings of Hands-Free Speech Communication and Microphone Arrays (HSCMA), pp. 107–111, May 2014. Thank you!