Recovery Guarantees for Low Complexity Models Samuel Vaiter CEREMADE, Univ. Paris-Dauphine October 17, 2013 Séminaire Image, IMB, Bordeaux I 

People & Papers • V., M. Golbabaee, M. J. Fadili et G. Peyré, Model Selection with Piecewise Regular Gauges, Tech. report, http://arxiv.org/abs/1307.2342 , 2013 • J. Fadili, V. and G. Peyré, Linear Convergence Rates for Gauge Regularization, ongoing work 

Our work • Study of models, not algorithms • Impact of the noise • Understand the geometry 3 parts of this talk 1. Defining our framework of interest 2. Noise robustness results 3. Some examples 

Linear Inverse Problem denoising inpainting deblurring 

Linear Inverse Problem : Forward Model y = Φ x0 + w y ∈ RQ observations Φ ∈ RQ×N linear operator x0 ∈ RN unknown vector w ∈ RQ realization of a noise (bounded here) 

Linear Inverse Problem : Forward Model y = Φ x0 + w y ∈ RQ observations Φ ∈ RQ×N linear operator x0 ∈ RN unknown vector w ∈ RQ realization of a noise (bounded here) Objective: Recover x0 from y. 

The Variational Approach x ∈ argmin x∈RN F(y, x) + λ J(x) (Pλ(y)) Trade-off between data fidelity and prior regularization 

The Variational Approach x ∈ argmin x∈RN F(y, x) + λ J(x) (Pλ(y)) Trade-off between data fidelity and prior regularization • Data fidelity: 2 loss, logistic, etc. F(y, x) = 1 2 ||y − Φx||2 2 

The Variational Approach x ∈ argmin x∈RN F(y, x) + λ J(x) (Pλ(y)) Trade-off between data fidelity and prior regularization • Data fidelity: 2 loss, logistic, etc. F(y, x) = 1 2 ||y − Φx||2 2 • Parameter: By hand or automatic like SURE. 

The Variational Approach x ∈ argmin x∈RN F(y, x) + λ J(x) (Pλ(y)) Trade-off between data fidelity and prior regularization • Data fidelity: 2 loss, logistic, etc. F(y, x) = 1 2 ||y − Φx||2 2 • Parameter: By hand or automatic like SURE. • Regularization: ? 

A Zoo ... ? Block Sparsity Nuclear Norm Trace Lasso Polyhedral Antisparsity Total Variation 

Relations to Previous Works • Fuchs, J. J. (2004). On sparse representations in arbitrary redundant bases. • Tropp, J. A. (2006). Just relax: Convex programming methods for identifying sparse signals in noise. • Grasmair, M. and al. (2008). Sparse regularization with q penalty term. • Bach, F. R. (2008). Consistency of the group lasso and multiple kernel learning & Consistency of trace norm minimization. • V. and al. (2011). Robust sparse analysis regularization. • Grasmair, M. and al. (2011). Necessary and sufficient conditions for linear convergence of 1-regularization. • Grasmair, M. (2011). Linear convergence rates for Tikhonov regularization with positively homogeneous functionals. (and more !) 

First Part: Gauges and Model Space 

Back to the Source: Union of Linear Models 2 components signal 

Back to the Source: Union of Linear Models 2 components signal T0 0 is the only 0-sparse vector 

Back to the Source: Union of Linear Models 2 components signal Te1 Te2 Axis points are 1-sparse (except 0) 

Back to the Source: Union of Linear Models 2 components signal The whole space points minus the axis are 2-sparse 

0 to 1 Combinatorial penalty associated to the previous union of model J(x) = ||x||0 = | {i : xi = 0} | 

0 to 1 Combinatorial penalty associated to the previous union of model J(x) = ||x||0 = | {i : xi = 0} | → non-convex → no regularity → NP-hard regularization Encode Union of Model in a good functional 

0 to 1 Combinatorial penalty associated to the previous union of model J(x) = ||x||0 = | {i : xi = 0} | → non-convex → no regularity → NP-hard regularization Encode Union of Model in a good functional x ||x||0 1 ||x||1 

Union of Linear Models to Regularizations Union of Model Gauges Combinatorial world Functional world 

Gauge J(x) 0 J(λx) = λJ(x) for λ 0 J convex x → J(x) 1 C = {x : J(x) 1} C a convex set 

Gauge J(x) 0 J(λx) = λJ(x) for λ 0 J convex x → J(x) 1 C = {x : J(x) 1} C a convex set 

Subdifferential t f (t) (t, t2) 

Subdifferential t f (t) (t, t2) 

Subdifferential t f (t) 

Subdifferential t f (t) ∂f (t) = η : f (t ) f (t) + η, t − t 

The Model Linear Space 0 x 

The Model Linear Space 0 x ∂J(x) 

The Model Linear Space 0 x ∂J(x) 

The Model Linear Space 0 x ∂J(x) Tx Tx = VectHull(∂J(x))⊥ 

The Model Linear Space 0 x ∂J(x) Tx ex Tx = VectHull(∂J(x))⊥ ex = PTx (∂J(x)) 

Special cases Sparsity Tx = {η : supp(η) ⊆ supp(x)} ex = sign(x) 

Special cases Sparsity Tx = {η : supp(η) ⊆ supp(x)} ex = sign(x) (Aniso/Iso)tropic Total Variation Tx = {η : supp(∇η) ⊆ supp(∇x)} ex = sign(∇x) or ex = ∇x ||∇x|| 

Special cases Sparsity Tx = {η : supp(η) ⊆ supp(x)} ex = sign(x) (Aniso/Iso)tropic Total Variation Tx = {η : supp(∇η) ⊆ supp(∇x)} ex = sign(∇x) or ex = ∇x ||∇x|| Trace Norm SVD: x = UΛV ∗ Tx = {η : U∗ ⊥ ηV⊥ = 0} ex = UV ∗ 

Algebraic Stability Composition by a linear operator • ||∇ · ||1 — Anisotropic TV • ||∇ · ||1,2 — Istotropic TV • ||Udiag(·)||∗ — Trace Lasso 

Algebraic Stability Composition by a linear operator • ||∇ · ||1 — Anisotropic TV • ||∇ · ||1,2 — Istotropic TV • ||Udiag(·)||∗ — Trace Lasso Sum of gauges (Composite priors) • || · ||1 + || · ||1 — sparse TV • || · ||1 + || · ||2 — Elastic net • || · ||1 + || · ||∗ — Sparse + Low-rank 

Second Part: 2 Robustness and Model Selection 

Certificate x ∈ argmin Φx=Φx0 J(x) (P0(y)) x Φx = Φx0 

Certificate x ∈ argmin Φx=Φx0 J(x) (P0(y)) ∂J(x) x Φx = Φx0 η Dual certificates: D = Im Φ∗ ∩ ∂J(x0) 

Certificate x ∈ argmin Φx=Φx0 J(x) (P0(y)) ∂J(x) x Φx = Φx0 η Dual certificates: D = Im Φ∗ ∩ ∂J(x0) Proposition ∃η ∈ D ⇔ x0 solution de (P0(y)) 

Tight Certificate and Restricted Injectivity Tight dual certificates ¯ D = Im Φ∗ ∩ ri ∂J(x) 

Tight Certificate and Restricted Injectivity Tight dual certificates ¯ D = Im Φ∗ ∩ ri ∂J(x) Restricted Injectivity Ker Φ ∩ Tx = {0} (RICx ) 

Tight Certificate and Restricted Injectivity Tight dual certificates ¯ D = Im Φ∗ ∩ ri ∂J(x) Restricted Injectivity Ker Φ ∩ Tx = {0} (RICx ) Proposition ∃η ∈ ¯ D ∧ (RICx ) ⇒ x unique solution of (Pλ(y)) 

2 stability Theorem If ∃η ∈ ¯ D ∧ (RICx ), then λ ∼ ||w|| ⇒ ||x − x || = O(||w||) 

Minimal-norm Certificate η ∈ D ⇐⇒ η = Φ∗q, ηT = e and J◦(η) 1 

Minimal-norm Certificate η ∈ D ⇐⇒ η = Φ∗q, ηT = e and J◦(η) 1 Minimal-norm precertificate η0 = argmin η=Φ∗q ηT =e ||q|| 

Minimal-norm Certificate η ∈ D ⇐⇒ η = Φ∗q, ηT = e and J◦(η) 1 Minimal-norm precertificate η0 = argmin η=Φ∗q ηT =e ||q|| Proposition If (RICx ), then η0 = (Φ+ T Φ)∗e 

Model Selection Theorem If η0 ∈ ¯ D, the noise-to-signal ratio is low enough and λ ∼ ||w||, the unique solution x of (Pλ(y)) satifies Tx = Tx0 and ||x − x || = O(||w||) 

A Better Certificate ? • With model selection: no 

A Better Certificate ? • With model selection: no • Without: ongoing works Duval-Peyré: Sparse Deconvolution Dossal: Sparse Tomography 

Third Part: Some Examples 

Sparse Spike Deconvolution (Dossal, 2005) x0 

Sparse Spike Deconvolution (Dossal, 2005) Φx = i xi ϕ(· − ∆i) J(x) = ||x||1 x0 γ Φx0 

Sparse Spike Deconvolution (Dossal, 2005) Φx = i xi ϕ(· − ∆i) J(x) = ||x||1 x0 γ Φx0 η0 ∈ ¯ D ⇔ ||Φ+,∗ Ic ΦI s||∞ < 1 γ ||η0,Ic ||∞ γcrit 1 

TV Denoising (V. et al., 2011) Φ = Id J(x) = ||∇x||1 i xi i xi 

TV Denoising (V. et al., 2011) Φ = Id J(x) = ||∇x||1 i xi i xi k mk k mk +1 −1 

TV Denoising (V. et al., 2011) Φ = Id J(x) = ||∇x||1 i xi i xi k mk k mk +1 −1 Support stability No support stability 

TV Denoising (V. et al., 2011) Φ = Id J(x) = ||∇x||1 i xi i xi k mk k mk +1 −1 Support stability No support stability Both are 2-stable 

Thanks for your attention !