Thomas Oberlin

(ISAE-SUPAERO, ANITI, IRIT)

https://s3-seminar.github.io/seminars/thomas-oberlin

Title — Regularization via deep generative models: an analysis point of view

Abstract — In this talk, we present a new way of regularizing an inverse problem in imaging (e.g., deblurring or inpainting) by means of a deep generative neural network. Compared to end-to-end models, such approaches seem particularly interesting since the same network can be used for many different problems and experimental conditions, as soon as the generative model is suited to the data. Previous works proposed to use a synthesis framework, where the estimation is performed on the latent vector, the solution being obtained afterwards via the decoder. Instead, we propose an analysis formulation where we directly optimize the image itself and penalize the latent vector. We illustrate the interest of such a formulation by running experiments of inpainting, deblurring and super-resolution. In many cases our technique achieves a clear improvement of the performance and seems to be more robust, in particular with respect to initialization.

Biography — Thomas Oberlin holds a Ph.D. in applied mathematics from the University of Grenoble. In 2014, he was a postdoctoral fellow in signal processing and medical imaging at Inria Rennes, before joining as an Assistant Professor INP Toulouse - ENSEEIHT and the IRIT Laboratory, at Université de Toulouse. Since 2019, he is an Assistant/Associate Professor of Image Processing and Machine Learning at ISAE-SUPAERO and member of IA institute ANITI. His research interests include hyperspectral/spectral imaging, latent factor models, data-driven regularization of inverse problems, and time-frequency representations.

Transcript

1. Regularization via deep generative models: an
   analysis point of view
   Thomas Oberlin and Mathieu Verm
   ISAE-SUPAERO, Université de Toulouse, ANITI, IRIT
   Séminaire L2S
   June 25th, 2021
   T. Oberlin Regularization via deep generative models 1 / 28
   

   View Slide

2. Outline
   1. Introduction : inverse problems
   2. Deep neural networks for inverse problems
   3. Analysis vs synthesis
   4. Experiments
   5. Conclusion
   T. Oberlin Regularization via deep generative models 2 / 28
   

   View Slide

3. Inverse problems in imaging
   Forward model of image reconstruction :
   y = Ax∗ + n, (1)
   Inverse problem (variational formulation)
   ˆ
   x = arg min
   x
   1
   2
   Ax − y 2
   2
   + λϕ(x) (2)
   Examples
   Inpainting (A is a mask)
   Deblurring (A is Toeplitz)
   Tomography (A computes radial projections)
   Compressed sensing (A satisfies conditions such as RIP)
   etc
   T. Oberlin Regularization via deep generative models 3 / 28
   

   View Slide

4. Priors and regularizers
   Standard penalties or priors
   Sparsity : 0
   , 1
   , greedy algorithms, etc
   Sparsity in a transform domain : wavelets, TV, etc
   Low-rankness : nuclear norm
   Structured sparsity : 12
   (group-lasso)
   Bayesian priors −→ MMSE, MAP
   Data-driven
   Dictionary learning
   Neural networks
   T. Oberlin Regularization via deep generative models 4 / 28
   

   View Slide

5. Example : fast acquisition of EELS spectrum-images
   T. Oberlin Regularization via deep generative models 5 / 28
   

   View Slide

6. Example : Fusion imagery/spectroscopy for the JWST
   [Guilloteau et al 2020]
   Problem similar to super-resolution
   Need for spatial and spectral regularizations
   T. Oberlin Regularization via deep generative models 6 / 28
   

   View Slide

7. Outline
   1. Introduction : inverse problems
   2. Deep neural networks for inverse problems
   Regression DNNs
   Unrolling
   Plug and play
   Generative models
   3. Analysis vs synthesis
   4. Experiments
   5. Conclusion
   T. Oberlin Regularization via deep generative models 7 / 28
   

   View Slide

8. Regression DNNs
   Examples in single-image super-resolution :
   SRCNN [Dong et al., TPAMI 2015]
   VDSR [Kim et al, CVPR 2016]
   T. Oberlin Regularization via deep generative models 8 / 28
   

   View Slide

9. Unrolling/unfolding
   Learns some parameters or building blocks of an iterative algorithm
   (ADMM, proximal gradient, etc) −→ can include some information about
   model or prior
   [Yang et al. NeurIPS 2016]
   T. Oberlin Regularization via deep generative models 9 / 28
   

   View Slide

10. Plug and play
    Decouple the regularization and the degradation model −→ the
    same network can help to solve different inverse problems
    Early works : standard denoising algorithms such as BM3D
    [Venkatakrishnan et al., 2013]
    More recently : denoising neural networks
    Limitation : hard to tune the strength of regularization (related to
    the noise level used for training the network)
    T. Oberlin Regularization via deep generative models 10 / 28
    

    View Slide

11. Generative models
    Generative neural networks as priors
    1. Learn a generative NN
    2. Use it as a prior in a model-based formulation [Bora et al., ICML
    2017]
    Pros and cons
    (+) Well-posed formulation (variational/MAP)
    (+) Tunable regularization
    (+) Generic : same network for any inverse problem
    (-) Performance can be lower than with regression
    (-) Do not generalize well for ood examples
    T. Oberlin Regularization via deep generative models 11 / 28
    

    View Slide

12. Outline
    1. Introduction : inverse problems
    2. Deep neural networks for inverse problems
    3. Analysis vs synthesis
    Generative NNs
    Invertible Neural Networks
    Analysis vs Synthesis
    Related works
    4. Experiments
    5. Conclusion
    T. Oberlin Regularization via deep generative models 12 / 28
    

    View Slide

13. Generative neural networks : Variational auto-encoder
    (VAE)
    Generative model
    z ∼ p(z) “prior”
    x ∼ pθ
    (x|z) = N(Dθ
    (z), ηI), avec Dθ
    decoder/generator network
    Maximum likelihood :
    pθ
    (x) = pθ
    (x|z)p(z) dz.
    Intractable −→ variational inference
    log pθ
    (x) = Ez∼qφ
    (z|x)
    log pθ
    (x)
    ≥ Ez∼qφ
    (z|x)
    log
    pθ
    (x, z)
    qφ
    (z|x)
    termed ELBO or VLB
    T. Oberlin Regularization via deep generative models 13 / 28
    

    View Slide

14. Learning a VAE
    Loss = ELBO :
    Lθ,φ
    (x) = Ez∼qφ
    (z|x)
    log pθ
    (x|z)
    Reconstruction error
    + Ez∼qφ
    (z|x)
    log
    p(z)
    qφ
    (z|x)
    Regularization
    Reconstruction error :
    log pθ
    (x|z) = −
    1
    2η
    x − Dθ
    (z) 2
    2
    Regularization :
    log
    p(z)
    qφ
    (z|x)
    = −DKL
    (qφ
    (z|x)||p(z))
    Reparametrization of qφ
    :
    qφ
    (z|x) = N(µ(x), σ(x)I),
    where µ and σ are parameterized by encoder E.
    T. Oberlin Regularization via deep generative models 14 / 28
    

    View Slide

15. Illustration of a VAE
    [credit : Lilian Weng]
    T. Oberlin Regularization via deep generative models 15 / 28
    

    View Slide

16. Invertible Neural Networks (INNs)
    Same generative model
    z ∼ p(z) prior
    x ∼ pθ
    (x|z) = N(Dθ
    (z), ηI) decoder / generator
    INN : design Dθ
    so as to allow for exact inference.
    Main ideas :
    Invertible (bijective) layers −→ no convolution
    Triangular Jacobian for tractable gradient backpropagation
    Simple layers and deep networks
    T. Oberlin Regularization via deep generative models 16 / 28
    

    View Slide

17. Glow [Kingma & Dhariwal, NeurIPS 2018]
    Building blocks : realNVP [Dinh et al 2016] and 1 × 1 convolution
    Actnorm : channel-wise affine transformation
    Invertible 1 × 1 convolution (i.e., invertible linear transformation in
    the channel dimension)
    Affine coupling layer : split + affine transformation (computed from
    an auxiliary NN)
    [Kingma & Dhariwal 2018]
    T. Oberlin Regularization via deep generative models 17 / 28
    

    View Slide

18. Synthesis formulation
    1. Learn a generative NN
    2. Use it as a prior in a synthesis formulation [Bora et al., ICML 2017]
    [Asim et al, 2020]
    ˆ
    x = D arg min
    z
    1
    2
    AD(z) − y 2
    2
    + λ z 2
    2
    . (3)
    Limitations
    Hard to initialize
    Sensisitive to any bias in the prior
    Bad generalization ability
    T. Oberlin Regularization via deep generative models 18 / 28
    

    View Slide

19. Analysis regularization
    Our proposal :
    ˆ
    x = arg min
    x
    1
    2
    Ax − y 2
    2
    + λ E(x) 2
    2
    . (4)
    Similar to analysis vs synthesis for sparsity in a dictionary
    Not a MAP (no Jacobian)
    Intractable for VAEs
    Well suited to INNs
    Generalizes well
    T. Oberlin Regularization via deep generative models 19 / 28
    

    View Slide

20. Related works
    Joint Posterior MAP [González, Almanda and Tan 2021]
    Focused on MAPs with VAEs
    Joint MAP in x and z with alternate optimization
    Add noise during VAE training to improve generalization
    Bayesian estimation of z [Holden, Pereyra and Zygalakis 2021]
    Bayesian computation in the manifold given by a VAE or GAN
    (“synthesis”)
    Sample the posterior with MCMC
    T. Oberlin Regularization via deep generative models 20 / 28
    

    View Slide

21. Outline
    1. Introduction : inverse problems
    2. Deep neural networks for inverse problems
    3. Analysis vs synthesis
    4. Experiments
    5. Conclusion
    T. Oberlin Regularization via deep generative models 21 / 28
    

    View Slide

22. Experimental setting
    Experimental setting similar to [Asim et al 2020]
    Dataset : CelebA, face images of size 64 x 64
    Network : Glow with 32 steps of flow, trained with Adam
    Parameter λ : tuned manually
    Initialization : z0 = 0 and x0 = D(0)
    Inverse problems :
    Inpainting with random mask (60% of missing pixels) or structured
    (squared of 10 x 10)
    Deblurring with 7 x 7 uniform filter
    Super-resolution with factor 2 and 4 and uniform filters
    T. Oberlin Regularization via deep generative models 22 / 28
    

    View Slide

23. Deblurring and super-resolution
    True
    Observed
    Synthesis
    Analysis
    Deblurring 2x super-resolution
    T. Oberlin Regularization via deep generative models 23 / 28
    

    View Slide

24. Inpainting
    True
    Observed
    Synthesis
    Analysis
    Random mask Structured mask
    T. Oberlin Regularization via deep generative models 24 / 28
    

    View Slide

25. Scores
    Task PSNR (synthesis) PSNR (analysis) SSIM (synthesis) SSIM (analysis)
    Deblurring 23.38 ±2.04 32.16 ±1.56 0.74 ±0.09 0.94 ±0.01
    Super-res. (x2) 22.26 ±4.21 31.19 ±1.33 0.76 ±0.12 0.93 ±0.01
    Super-res. (x4) 18.94 ±2.81 24.12 ±1.21 0.61 ±0.11 0.76 ±0.03
    Inp. (random mask) 21.84 ±3.57 27.89 ±2.24 0.71 ±0.14 0.87 ±0.05
    Inp. (struct. mask) 30.40 ±2.53 27.50 ±3.26 0.94 ±0.02 0.91 ±0.03
    Average performance over 50 images ± standard deviation. Best score
    between analysis and synthesis highlighted in bold, for both metrics.
    T. Oberlin Regularization via deep generative models 25 / 28
    

    View Slide

26. Out of distribution examples
    Synthesis
    Analysis
    Deblurring Inpainting (r) Inpainting (s)
    T. Oberlin Regularization via deep generative models 26 / 28
    

    View Slide

27. Out of distribution examples
    Synthesis
    Analysis
    Deblurring Inpainting (r) Inpainting (s)
    T. Oberlin Regularization via deep generative models 27 / 28
    

    View Slide

28. Conclusion & perspectives
    Summary
    Regularization with DNNs : use data-driven priors within a
    model-based inversion −→ generic and well grounded
    Analysis regularization : generalizes better, less sensitive to bias or
    mismatch in the prior
    Perspectives
    Short-term perspectives : MAP with Glow ; Analysis with VAE or
    other non-invertible DNNs
    Long-term perspective : close the gap between toy datasets and real
    applications
    T. Oberlin Regularization via deep generative models 28 / 28
    

    View Slide