Regularization via deep generative models: an analysis point of view Thomas Oberlin and Mathieu Verm ISAE-SUPAERO, Université de Toulouse, ANITI, IRIT Séminaire L2S June 25th, 2021 T. Oberlin Regularization via deep generative models 1 / 28

Outline 1. Introduction : inverse problems 2. Deep neural networks for inverse problems 3. Analysis vs synthesis 4. Experiments 5. Conclusion T. Oberlin Regularization via deep generative models 2 / 28

Inverse problems in imaging Forward model of image reconstruction : y = Ax∗ + n, (1) Inverse problem (variational formulation) ˆ x = arg min x 1 2 Ax − y 2 2 + λϕ(x) (2) Examples Inpainting (A is a mask) Deblurring (A is Toeplitz) Tomography (A computes radial projections) Compressed sensing (A satisfies conditions such as RIP) etc T. Oberlin Regularization via deep generative models 3 / 28

Priors and regularizers Standard penalties or priors Sparsity : 0 , 1 , greedy algorithms, etc Sparsity in a transform domain : wavelets, TV, etc Low-rankness : nuclear norm Structured sparsity : 12 (group-lasso) Bayesian priors −→ MMSE, MAP Data-driven Dictionary learning Neural networks T. Oberlin Regularization via deep generative models 4 / 28

Example : fast acquisition of EELS spectrum-images T. Oberlin Regularization via deep generative models 5 / 28

Example : Fusion imagery/spectroscopy for the JWST [Guilloteau et al 2020] Problem similar to super-resolution Need for spatial and spectral regularizations T. Oberlin Regularization via deep generative models 6 / 28

Outline 1. Introduction : inverse problems 2. Deep neural networks for inverse problems Regression DNNs Unrolling Plug and play Generative models 3. Analysis vs synthesis 4. Experiments 5. Conclusion T. Oberlin Regularization via deep generative models 7 / 28

Regression DNNs Examples in single-image super-resolution : SRCNN [Dong et al., TPAMI 2015] VDSR [Kim et al, CVPR 2016] T. Oberlin Regularization via deep generative models 8 / 28

Unrolling/unfolding Learns some parameters or building blocks of an iterative algorithm (ADMM, proximal gradient, etc) −→ can include some information about model or prior [Yang et al. NeurIPS 2016] T. Oberlin Regularization via deep generative models 9 / 28

Plug and play Decouple the regularization and the degradation model −→ the same network can help to solve different inverse problems Early works : standard denoising algorithms such as BM3D [Venkatakrishnan et al., 2013] More recently : denoising neural networks Limitation : hard to tune the strength of regularization (related to the noise level used for training the network) T. Oberlin Regularization via deep generative models 10 / 28

Generative models Generative neural networks as priors 1. Learn a generative NN 2. Use it as a prior in a model-based formulation [Bora et al., ICML 2017] Pros and cons (+) Well-posed formulation (variational/MAP) (+) Tunable regularization (+) Generic : same network for any inverse problem (-) Performance can be lower than with regression (-) Do not generalize well for ood examples T. Oberlin Regularization via deep generative models 11 / 28

Outline 1. Introduction : inverse problems 2. Deep neural networks for inverse problems 3. Analysis vs synthesis Generative NNs Invertible Neural Networks Analysis vs Synthesis Related works 4. Experiments 5. Conclusion T. Oberlin Regularization via deep generative models 12 / 28

Generative neural networks : Variational auto-encoder (VAE) Generative model z ∼ p(z) “prior” x ∼ pθ (x|z) = N(Dθ (z), ηI), avec Dθ decoder/generator network Maximum likelihood : pθ (x) = pθ (x|z)p(z) dz. Intractable −→ variational inference log pθ (x) = Ez∼qφ (z|x) log pθ (x) ≥ Ez∼qφ (z|x) log pθ (x, z) qφ (z|x) termed ELBO or VLB T. Oberlin Regularization via deep generative models 13 / 28

Learning a VAE Loss = ELBO : Lθ,φ (x) = Ez∼qφ (z|x) log pθ (x|z) Reconstruction error + Ez∼qφ (z|x) log p(z) qφ (z|x) Regularization Reconstruction error : log pθ (x|z) = − 1 2η x − Dθ (z) 2 2 Regularization : log p(z) qφ (z|x) = −DKL (qφ (z|x)||p(z)) Reparametrization of qφ : qφ (z|x) = N(µ(x), σ(x)I), where µ and σ are parameterized by encoder E. T. Oberlin Regularization via deep generative models 14 / 28

Illustration of a VAE [credit : Lilian Weng] T. Oberlin Regularization via deep generative models 15 / 28

Invertible Neural Networks (INNs) Same generative model z ∼ p(z) prior x ∼ pθ (x|z) = N(Dθ (z), ηI) decoder / generator INN : design Dθ so as to allow for exact inference. Main ideas : Invertible (bijective) layers −→ no convolution Triangular Jacobian for tractable gradient backpropagation Simple layers and deep networks T. Oberlin Regularization via deep generative models 16 / 28

Glow [Kingma & Dhariwal, NeurIPS 2018] Building blocks : realNVP [Dinh et al 2016] and 1 × 1 convolution Actnorm : channel-wise affine transformation Invertible 1 × 1 convolution (i.e., invertible linear transformation in the channel dimension) Affine coupling layer : split + affine transformation (computed from an auxiliary NN) [Kingma & Dhariwal 2018] T. Oberlin Regularization via deep generative models 17 / 28

Synthesis formulation 1. Learn a generative NN 2. Use it as a prior in a synthesis formulation [Bora et al., ICML 2017] [Asim et al, 2020] ˆ x = D arg min z 1 2 AD(z) − y 2 2 + λ z 2 2 . (3) Limitations Hard to initialize Sensisitive to any bias in the prior Bad generalization ability T. Oberlin Regularization via deep generative models 18 / 28

Analysis regularization Our proposal : ˆ x = arg min x 1 2 Ax − y 2 2 + λ E(x) 2 2 . (4) Similar to analysis vs synthesis for sparsity in a dictionary Not a MAP (no Jacobian) Intractable for VAEs Well suited to INNs Generalizes well T. Oberlin Regularization via deep generative models 19 / 28

Related works Joint Posterior MAP [González, Almanda and Tan 2021] Focused on MAPs with VAEs Joint MAP in x and z with alternate optimization Add noise during VAE training to improve generalization Bayesian estimation of z [Holden, Pereyra and Zygalakis 2021] Bayesian computation in the manifold given by a VAE or GAN (“synthesis”) Sample the posterior with MCMC T. Oberlin Regularization via deep generative models 20 / 28

Outline 1. Introduction : inverse problems 2. Deep neural networks for inverse problems 3. Analysis vs synthesis 4. Experiments 5. Conclusion T. Oberlin Regularization via deep generative models 21 / 28

Experimental setting Experimental setting similar to [Asim et al 2020] Dataset : CelebA, face images of size 64 x 64 Network : Glow with 32 steps of flow, trained with Adam Parameter λ : tuned manually Initialization : z0 = 0 and x0 = D(0) Inverse problems : Inpainting with random mask (60% of missing pixels) or structured (squared of 10 x 10) Deblurring with 7 x 7 uniform filter Super-resolution with factor 2 and 4 and uniform filters T. Oberlin Regularization via deep generative models 22 / 28

Deblurring and super-resolution True Observed Synthesis Analysis Deblurring 2x super-resolution T. Oberlin Regularization via deep generative models 23 / 28

Inpainting True Observed Synthesis Analysis Random mask Structured mask T. Oberlin Regularization via deep generative models 24 / 28

Scores Task PSNR (synthesis) PSNR (analysis) SSIM (synthesis) SSIM (analysis) Deblurring 23.38 ±2.04 32.16 ±1.56 0.74 ±0.09 0.94 ±0.01 Super-res. (x2) 22.26 ±4.21 31.19 ±1.33 0.76 ±0.12 0.93 ±0.01 Super-res. (x4) 18.94 ±2.81 24.12 ±1.21 0.61 ±0.11 0.76 ±0.03 Inp. (random mask) 21.84 ±3.57 27.89 ±2.24 0.71 ±0.14 0.87 ±0.05 Inp. (struct. mask) 30.40 ±2.53 27.50 ±3.26 0.94 ±0.02 0.91 ±0.03 Average performance over 50 images ± standard deviation. Best score between analysis and synthesis highlighted in bold, for both metrics. T. Oberlin Regularization via deep generative models 25 / 28

Out of distribution examples Synthesis Analysis Deblurring Inpainting (r) Inpainting (s) T. Oberlin Regularization via deep generative models 26 / 28

Out of distribution examples Synthesis Analysis Deblurring Inpainting (r) Inpainting (s) T. Oberlin Regularization via deep generative models 27 / 28

Conclusion & perspectives Summary Regularization with DNNs : use data-driven priors within a model-based inversion −→ generic and well grounded Analysis regularization : generalizes better, less sensitive to bias or mismatch in the prior Perspectives Short-term perspectives : MAP with Glow ; Analysis with VAE or other non-invertible DNNs Long-term perspective : close the gap between toy datasets and real applications T. Oberlin Regularization via deep generative models 28 / 28