$30 off During Our Annual Pro Sale. View Details »

Clémence Prévost

S³ Seminar
February 11, 2022

Clémence Prévost

(CRIStAL - Univ. Lille, CNRS, Centrale Lille)

https://s3-seminar.github.io/seminars/clemence-prevost/

Title — Multimodal data fusion by tensor low-rank approximations - applications in remote sensing

Abstract — Due to the large availability of raw data, the appeal for data fusion has been steadily growing in the signal processing community. Hence the design of coupled models, that exploit shared information between several observations. It is thus expected from data fusion that it provides a better estimation of the parameters of interest rather than separate processing of the datasets.

In remote sensing, hyperspectral images have been thoroughly exploited in, e.g., spectral unmixing, image classification or target detection. The natural 3-dimensional format of these images allows them to be mathematically represented as 3-dimensional tensors.

In this presentation, I will introduce some of my recent results regarding multimodal data fusion using low-rank tensor decompositions, applied on hyperspectral images. I will focus on the hyperspectral super-resolution and spectral unmixing problems accounting for inter-image variability. While the first one addresses image reconstruction, the second one falls under the scope of source separation. A major difficulty lies in the presence of inter-image variability, that reinforces the ill-posedness of the problems. I will introduce two algorithms to solve the problems at hand. Then, I will showcase their performance for image fusion and spectral unmixing on a set of real hyperspectral data accounting for spectral variability.

Biography — Clémence Prévost is currently a post-doctoral fellow in CRIStAL, University of Lille, under the supervision of Pierre Chainais and Rémy Boyer. She received the PhD degree in signal processing in 2021 from CRAN, University of Nancy. Her main research interests include multimodal data fusion, tensor decompositions and solving ill-posed inverse problems.

S³ Seminar

February 11, 2022
Tweet

More Decks by S³ Seminar

Other Decks in Research

Transcript

  1. Multimodal data fusion by low-rank tensor approximations Applications in remote

    sensing S3 Seminar February 11th, 2022 Univ. Lille, CNRS, Centrale Lille, UMR 9189 CRIStAL 1/36
  2. General introduction

  3. General introduction Data fusion

  4. What is multimodality ? Vocabulary Modality Signal Phenomenon of interest

    acquires contains Several datasets → Multimodality. 2/36
  5. What is multimodality ? Vocabulary Modality Signal Phenomenon of interest

    acquires contains Several datasets → Multimodality. fMRI EEG Brain activity 2/36
  6. What is multimodality ? Vocabulary Modality Signal Phenomenon of interest

    acquires contains Several datasets → Multimodality. fMRI EEG Brain activity Hyperspectral Airborne scene Multispectral Pictures are courtesy of (Datcu et. al, 2005) 2/36
  7. Multimodal data fusion Definition (Lahat et al., 2015) Data fusion

    is the analysis of several datasets such that [they] can interact with and inform each other. → (Kanatsoulis et al., 2018), (Biessmann et al., 2011), (Rivet et al., 2014), (Betoule et al., 2014),... Main issues: • Sizes, resolutions, noise contaminations; • Various fusion strategies, links between modalities; • Uncertainties in shared information. 3/36
  8. Multimodal data fusion Definition (Lahat et al., 2015) Data fusion

    is the analysis of several datasets such that [they] can interact with and inform each other. → (Kanatsoulis et al., 2018), (Biessmann et al., 2011), (Rivet et al., 2014), (Betoule et al., 2014),... Main issues: • Sizes, resolutions, noise contaminations; • Various fusion strategies, links between modalities; • Uncertainties in shared information. → In this work: • True fusion: symmetric roles for modalities; • A specific problem: hyperspectral super-resolution. 3/36
  9. General introduction The super-resolution problem

  10. Principle of spectral imaging Water Soil Vegetation Spectral unmixing Image

    classification Target detection ... 4/36
  11. Tradeoff in resolutions Hyperspectral image (HSI) Multispectral image (MSI) 

    Hundreds of spectral bands;  Low spatial resolution;  Few spectral bands;  High spatial resolution. 5/36
  12. Hyperspectral super-resolution 6/36

  13. Remote sensing problems • Pansharpening: (Vivone et al., 2014); (Loncan

    et al., 2015) ; Hyperspectral image Panchromatic image Super-resolution image FROM RECOVER 7/36
  14. Remote sensing problems • Pansharpening: (Vivone et al., 2014); (Loncan

    et al., 2015) ; Hyperspectral image Panchromatic image Super-resolution image FROM RECOVER • Unmixing: (Parente et al., 2010); (Bioucas-Dias et al., 2012); (Qian et al., 2016) ; 7/36
  15. General introduction Observation model

  16. From matrix models ... → Component substitution (Laben et al.,

    2000); → Multi-resolution analysis (Aiazzi et al., 2006); → Unmixing (Yokoya et al. 2011); → Bayesian (Wei et al., 2015).  I-fold, J-fold diversities (Sidiropoulos et al., 2000) ;  Good model fitting and interpretability;  Non-unique; → additional diversities (Dohono et. al, 2004), (Comon, 1994) ;  Higher-dimensional observations. 8/36
  17. ... Towards tensor models Tensor: array of p dimensions (p

    ≥ 3).  I-fold, J-fold and K-fold diversities;  Structure-preserving for high-dimensional data;  Some low-rank decompositions are unique. → Suited for the super-resolution problem. 9/36
  18. Tensor algebra (1) Mode product Tensor unfoldings 10/36

  19. Tensor algebra (2) • Canonical polyadic decomposition: • Tucker decomposition:

    • Block-term decomposition: 11/36
  20. Tensor observation model YH = Y •1 P1 •2 P2

    +EH , YM = Y •3 P3 +EM . • P1 , P2 : Gaussian blurring + downsampling (Wald et al., 1997); • P3 : spectral response functions. 12/36
  21. Tensor observation model YH = Y •1 P1 •2 P2

    +EH , YM = Y •3 P3 +EM . • P1 , P2 : Gaussian blurring + downsampling (Wald et al., 1997); • P3 : spectral response functions. Basic optimization problem minimize low-rank Y YH −Y • 1 P1 • 2 P2 2 F +λ YM −Y • 3 P3 2 F . 12/36
  22. Ill-posedness Aim Recover IJK entries from IJKM +IH JH K

    observations. COMPLEXITY Matrix LL1-BTD CPD Tucker (IJ +K −R)R ((I +J −L)L+(K −1))R (I +J +K −2)N IR1 +JR2 +KR3 + 3 i=1 Ri − 3 i=1 R2 i • I = J = K = 100, IH = JH = 50, KM = 10; • N = LR, R1 = R2 = LR, R3 = R. 5 10 15 20 103 104 105 Number of unknown parameters 5 10 15 20 103 104 105 106 5 10 15 20 104 105 13/36
  23. Variations of the model 14/36

  24. Variations of the model 14/36

  25. Summary 15/36

  26. LL1-BTD for joint fusion and unmixing of the unknown SRI

  27. LL1-BTD for joint fusion and unmixing of the unknown SRI

    Problem statement
  28. Inter-image variability Principle Few satellites with bothsensors Different acquisition times

    Seasonal, atmospheric, illumination variations Inter-image variability (Hilker et al., 2009; Emelyanova et al., 2013) 16/36
  29. A more flexible model (Borsoi, Prévost et al., 2021) SRI

    SRI HSI MSI Variability tensor + = YH = Y •1 P1 •2 P2 +EH , YM = Y •3 P3 +EM = (Y +Ψ)•3 P3 +EM . Very ambiguous; Ψ: General (spatial and spectral) variability. → Low-rank decompositions with small ranks. 17/36
  30. Linear mixing model and LL1 LL1-BTD as linear mixing model

    (Shivappa, 2010) Y = R r=1 ArBT r ⊗cr = R r=1 Sr ⊗cr ⇒Y(3) = SCT, • cr : spectral signatures, • Sr : abundance maps with low-rank L: Sr = ArBT r ; Non-negative factors. 18/36
  31. Spectral variability Simple model C = ψ+C, • ψ: spectral

    variability factor;  Explicit in 3rd dimension;  Equivalent to multiplicative model (Borsoi et al., 2018); Less general variability, less restrictive recovery conditions. Y = R r=1 ArBT r ⊗cr, Ψ = R r=1 ArBT r ⊗ψr, Y = R r=1 ArBT r ⊗cr = R r=1 ArBT r ⊗(cr +ψr). 19/36
  32. LL1-BTD for joint fusion and unmixing of the unknown SRI

    Proposed approach
  33. State-of-the-art  Tensor approach;  Fusion + unmixing;  Exact

    recovery without additional constraints. 20/36
  34. A procedure for joint fusion and unmixing Goal Recover uniquely

    Y and Ψ•3 P3 and the non-negative LL1 factors. Unknown Spatial degradation Spectral degradation Observations Unmixing HSR LL1-BTD Low-rank Variability model 21/36
  35. Exact recovery Hypotheses: noiseless case, YM = Y •3 P3

    , YH = Y •1 P1 •2 P2 . Generic theorem The SRI Y and degraded SRI Y •3 P3 are uniquely recovered by Y = R r=1 (ArBT r )⊗cr, Y •3 P3 = R r=1 (ArBT r )•3 P3cr, if IH JH ≥ LR, IJ ≥ L2R and min I L ,R +min J L ,R +min(KM ,R) ≥ 2R+2. Only Ψ•3 P3 ; C, P3C, S: unique up to permutation and scaling; → Holds for unmixing part of the problem. 22/36
  36. An approach for fusion only Main idea minimize A,B,C,CM J

    = YH − R r=1 (P1Ar(P2Br)T)⊗cr 2 F +λ YM − R r=1 (ArBT r )⊗cM,r 2 F . BTD-Var Input: YH , YM , B, C, CM , P1 , P2 , P3 ; R, L; Output: Y, Ψ•3 P3 ; While stopping criterion not met, do 1. Normalize columns of C,CM with unit norm; 2. A,B,C,CM ← minimize J (alternating procedure); 3. S ← ...,vec{ArBT r },... ; End 4. Y(3) ← SCT, Ψ•3 P3 ← YM −Y •3 P3 .  Fusion;  Unmixing. 23/36
  37. A constrained algorithm Constrained optimization problem minimize A,B,{Sr}R r=1 ,C,CM

    J s. to {Sr = ArBT r }R r=1 ≥ 0,C ≥ 0,CM ≥ 0. CNN-BTD-Var Input: YH , YM , B, C, CM , P1 , P2 , P3 ; R, L; Output: Y, Ψ•3 P3 , {Sr}R r=1 ,C,CM ; While stopping criterion not met, do 1. Normalize columns of C,CM with unit norm; 2. A,B,C,CM ← minimize J (ADMM procedure + non-negativity); 3. S ← ADMM procedure: low-rank + non-negativity; End 4. Y(3) ← SCT, Ψ•3 P3 ← YM −Y •3 P3 .  Fusion + unmixing of Y. 24/36
  38. LL1-BTD for joint fusion and unmixing of the unknown SRI

    Simulations
  39. Fusion setup • real SRI Y (AVIRIS) and MSI YM

    (Sentinel 2-A); Y → YH (decimation factor d = 4, filter with unit variance); • EH and EM : white Gaussian noise, 30dB SNR; → normalized spectral bands; • comparison to matrix and tensor methods + FuVar (matrix + variability) and CB-STAR (tensor + localized changes) (Borsoi, Prévost et al., 2021). • Dataset: Lockwood, acquired on 2018-08-20 (SRI) and 2018-10-19 (MSI); Y ∈ R80×100×173. SRI MSI . 25/36
  40. Fusion performance Algorithm R-SNR CC SAM ERGAS Time CNMF 18.7829

    0.89063 2.9768 6.7014 4.353 HySure 14.125 0.8633 4.4044 11.6 6.9823 FuVar 12.2703 0.7297 3.7313 6.7951 724.91 STEREO 6.552 0.80196 27.3623 25.1749 1.8835 SCOTT 2.2276 0.79276 28.5771 45.9608 0.2228 BTD-Var 20.1273 0.918432 2.92921 6.35566 5.46272 CNN-BTD-Var 19.4882 0.906525 3.0299 6.29101 4.11573 CB-STAR 19.0751 0.89445 3.3707 7.2926 68.0282 Reference BTD-Var CNN-BTD-Var SCOTT CNMF Reference BTD-Var CNN-BTD-Var CT-STAR CB-STAR 26/36
  41. Unmixing setup • Unmixing of the SRI Y; • Matrix

    approach: CNMF (Yokoya et al., 2012); • Two-step procedures: CB-STAR + MU-Acc (Gillis et al., 2012), BMDR-ADMM (Nus et al., 2018). • Lake Tahoe: Acquired on 2014-10-04 (SRI) and 2017-10-24 (MSI); Y ∈ R80×100×173. SRI MSI 27/36
  42. Unmixing performance 0 0.1 0.2 Water Ref. CNN-BTD-Var CNMF MU-Acc

    BMDR-ADMM 0.05 0.1 Soil 0 0.05 0.1 Vegetation Ref. CNN-BTD-Var CNMF Mu-Acc BMDR-ADMM 28/36
  43. Retrieving the variability factor 0 5 10 0 0.2 0.4

    0.6 Water Ref. Est. 0 5 10 -0.2 0 0.2 0 5 10 0.2 0.3 0.4 Soil 0 5 10 -0.4 -0.2 0 0.2 0 5 10 0.2 0.4 0.6 Vegetation 0 5 10 -0.5 0 0.5 • Water→5th band→blue wavelengths; • Soil→10th band→orange to infrared wavelengths; • Vegetation→7th band→green wavelengths. 29/36
  44. Partial conclusion • Flexible tensor model: inter-image variability; • A

    joint solution for fusion and unmixing of the super-resolution image; • Noiseless recovery guarantees: link with statistical identifiability. 30/36
  45. Partial conclusion • Flexible tensor model: inter-image variability; • A

    joint solution for fusion and unmixing of the super-resolution image; • Noiseless recovery guarantees: link with statistical identifiability. • Are the algorithms efficient ? → Performance analysis for BTD-Var. → A new constrained bound accounting for uncertainties. 30/36
  46. Performance bounds for coupled tensor models

  47. Deriving bounds for coupled tensor models 1. Choose parameters: low-rank

    factors ? Reconstructed tensor ? 2. Identify the constraints; 3. Apply formula according to scenario: uncoupled, partially-coupled, fully-coupled; 4. Evaluate performance of estimator. Contributions → Standard CCRB for coupled CP model: performance of STEREO and Blind-STEREO; → Randomly-constrained CRB: application to coupled LL1 models with random variability. 31/36
  48. General coupled model Y1 ∼ fY1;ω and Y2 ∼ fY2;ω,

    g(ω) = 0. Assumption: Model statistically identifiable. Constrained Cramér-Rao bound (CCRB) [Stoica et al.,1998] CCRB(ω) = U UTFU −1 UT, with U a basis of ker(G). Non informative when g(ω) depends on a random parameter; → random parameter θr . 32/36
  49. Randomly constrained CRB (RCCRB) First step: • ω ω y

    : locally unbiased cond. to θr ; Conditional CCRB CCRBθr (ω) = U θr (ω) UT θr (ω)CRB−1 θr (ω)U θr (ω) −1 UT θr (ω). RCCRB RCCRB(ω) = Eθr;ω CCRBθr (ω) , Ey|ω (ω−ω)(ω−ω)T ≥ RCCRB(ω). 33/36
  50. LL1 estimation LL1-ALS inspired from (De Lathauwer) min A2,B2,C1 Y1

    − R r=1 P1(A2)r(P2(B2)r)T ⊗(c1)r 2 F +λ Y2 − R r=1 (A2)r(B2)T r ⊗P3(c1)r 2 F . → Fully-coupled model; → Ignores the variability; 34/36
  51. LL1 estimation LL1-ALS inspired from (De Lathauwer) min A2,B2,C1 Y1

    − R r=1 P1(A2)r(P2(B2)r)T ⊗(c1)r 2 F +λ Y2 − R r=1 (A2)r(B2)T r ⊗P3(c1)r 2 F . → Fully-coupled model; → Ignores the variability; BTD-Var designed in Chap. 3 min A2,B2,C1,C2 Y1 − R r=1 P1(A2)r(P2(B2)r)T ⊗(c1)r 2 F +λ Y2 − R r=1 (A2)r(B2)T r ⊗(c2)r 2 F . → Blind in the third dimension; → Random variability hidden in C2 ; 34/36
  52. Performance of BTD-Var 5 10 15 20 25 30 35

    40 45 50 55 60 100 MSE Trace Performance for image reconstruction 5 10 15 20 100 101 MSE Trace 30 40 50 60 0.025 0.03 0.035 0.04 0.045 0.05 MSE Trace  Gain w.r.t. model without variability;  Robust to a lack of knowledge of the phenomenon;  Asymptotically efficient ? 35/36
  53. Conclusion

  54. Conclusion • Better performance than state-of-art; • Does not reach

    the RCCRB; • Still room for a clairvoyant algorithm ! https://cprevost4.github.io Thank you for your attention. 36/36