Samuel Vaiter
November 13, 2014
72

# Low Complexity Regularizations: a Localization Result

SMATI, Télécom ParisTech, Paris, November 2014.

## Samuel Vaiter

November 13, 2014

## Transcript

1. ### Régularisations de faibles complexités : Un résultat de "localisation" Samuel

Vaiter 13 November 2014

8. ### Main Theorem: First Version For a large class of recovery

problems, under some assumptions to be precised, one observes that both “energy” recovery and “localization” recovery hold. Theorem
9. ### Linear Inverse Problem (in Finite Dimension) ৙ > ʇ৘1 ,

ৗ denoising inpainting deblurring
10. ### Variational Regularization ৘Գ ѵ bshnjo ৘ѵϓৎ ভ)৘- ৙* , ౠ঱)৘*

Data ﬁdelity Prior vs

12. ### Some Remarks Loss considered: square prediction-error No noise: use of

the constrained form
13. ### Some Remarks Loss considered: square prediction-error No noise: use of

the constrained form
14. ### Some Remarks Loss considered: square prediction-error No noise: use of

the constrained form
15. ### Choice of a Regularization Wavelet basis sparsity Total Variation Fused

Lasso Nuclear norm OSCAR Spread representation Group Sparsity Trace Lasso Sobolev TGV Weighted sparsity Elastic net Ridge Analysis sparsity
16. ### Models Models (Convex) Functions vector of ﬁxed support matrix of

ﬁxed rank vector with same saturation pattern vector with same jump set

19. ### Main Theorem: 2nd Version Theorem For a large class of

variational regularizations, under some assumptions to be precised, one observes that both : Ј ৘Գ ѵ Ϥ৘1 Ј }}৘Գ ѿ ৘1 }} is small

22. ### Subdifferential ে)৘* ৘ Ȣ ৘ ে) Ȣ ৘* ে)৘* Ӓ

ে) Ȣ ৘* , ܕѴে) Ȣ ৘*- ৘ ѿ Ȣ ৘ܖ
23. ### Subdifferential ে)৘* ৘ Ȣ ৘ ে) Ȣ ৘* ে)৘* Ӓ

ে) Ȣ ৘* , ܕৈ- ৘ ѿ Ȣ ৘ܖ
24. ### Subdifferential ে)৘* ৘ Ȣ ৘ ে) Ȣ ৘* |ৈ ң

ে)৘* Ӓ ে) Ȣ ৘* , ܕৈ- ৘ ѿ Ȣ ৘ܖ~ ౯ে) Ȣ ৘* >
25. ### First Order Conditions Ȣ ৘ ѵ bshnjo ে)৘* ܧ Ѵে)

Ȣ ৘* > 1 Euler equation for convex + smooth function
26. ### First Order Conditions Ȣ ৘ ѵ bshnjo ে)৘* ܧ Ѵে)

Ȣ ৘* > 1 Euler equation for convex + smooth function Ȣ ৘ ѵ bshnjo ে)৘* ܧ ౯ে) Ȣ ৘* Ѹ 1 Euler equation for convex + non-smooth function

঱) Ȣ ৘*~ Ȣ ৘

঱) Ȣ ৘ , e৘*

঱) Ȣ ৘ , e৘*
34. ### What’s Important in Sparsity ? Ϥ৘ Ȣ ৘ 1: }}

Բ }}2 restricted to Ϥ৘ is locally smooth

e৘*

e৘*

e৘*
38. ### What’s Important in Sparsity ? Ϥ৘ Ȣ ৘ 1: }}

Բ }}2 restricted to Ϥ৘ is locally smooth 2: }} Բ }}2 is sharp (non-differentiable) in the direction of ϤԒ ৘
39. ### What’s Important in Sparsity ? Ϥ৘ Ȣ ৘ 1: }}

Բ }}2 restricted to Ϥ৘ is locally smooth 2: }} Բ }}2 is sharp (non-differentiable) in the direction of ϤԒ ৘ 3: ౯}} Բ }}2 restricted to Ϥ৘ is locally continuous (constant here)
40. ### Partial Smoothness Deﬁnition ঱ is partly smooth at ৘ relative

to a C3 -manifold Ϥ if • Smoothness. ঱ restricted to Ϥ is C3 around ৘ • Sharpness. ѭυ ѵ )ਚϤ ৘*Ԓ , ৔ М ঱)৘ , ৔υ* is non-smooth at ৔ > 1. • Continuity. ౯঱ on Ϥ is continuous around ৘. Ϥ nby)1- }}৘}} ѿ 2*
41. ### Main Theorem: 3rd Version Theorem Ј ৘Գ ѵ Ϥ৘1 Ј

}}৘Գ ѿ ৘1 }} is small When ঱ is partly smooth at ৘1 relatively to Ϥ৘1 , under some assumptions to be precised, one observes that:
42. ### Examples and Algebraic Stability ৘ М }}৘}}2 - }}৘}}3 -

}}৘}}ҋ - }}৘}}2-3 - nby ৉ )৘৉ *, - any smooth function Ϳ
43. ### Examples and Algebraic Stability ৘ М }}৘}}2 - }}৘}}3 -

}}৘}}ҋ - }}৘}}2-3 - nby ৉ )৘৉ *, - any smooth function Ϳ ঱ and ম partly smooth ܦ ঱ , ম partly smooth }} Բ }}2 , ౰}} Բ }}3 3 partly smooth
44. ### Examples and Algebraic Stability ৘ М }}৘}}2 - }}৘}}3 -

}}৘}}ҋ - }}৘}}2-3 - nby ৉ )৘৉ *, - any smooth function Ϳ ঱ and ম partly smooth ܦ ঱ , ম partly smooth }} Բ }}2 , ౰}} Բ }}3 3 partly smooth ঱ partly smooth and ঳ linear operator ܦ ঱ ҅ ঳ partly smooth }}Ѵ Բ }}2-3 (isotropic TV) partly smooth
45. ### Examples and Algebraic Stability ৘ М }}৘}}2 - }}৘}}3 -

}}৘}}ҋ - }}৘}}2-3 - nby ৉ )৘৉ *, - any smooth function Ϳ ঱ and ম partly smooth ܦ ঱ , ম partly smooth }} Բ }}2 , ౰}} Բ }}3 3 partly smooth ঱ partly smooth and ঳ linear operator ܦ ঱ ҅ ঳ partly smooth }}Ѵ Բ }}2-3 (isotropic TV) partly smooth ঱ partly smooth ܦ ঱ ҅ ౨ partly smooth (spectral lift) }} Բ }}҄ (nuclear/trace norm) partly smooth

48. ### Main Ideas: 1) The Lagrangian problem is nothing more than

the constrained problem up to "something" that we are going to control (by duality)
49. ### Main Ideas: 2 ) To r e c o v

e r a g o o d estimation of the ground truth with noise, it should be identiﬁable without noise

55. ### From Primal to Dual … … and Back 2 3

}}৙ ѿ ʇ৘}}3 , ౠ঱)৘* Ј ৘৙-ౠ njo )ਗ৙-ౠ * ঱҄)ʇ҄ઐ* ѿ ܕ৙- ઐܖ , ౠ 3 }}ઐ}}3 Ј ઐ৙-ౠ njo )਑৙-ౠ * ৘৙-1 njo )ਗ৙-1 * І ঱)৘* s.t ʇ৘ > ৙ ઐ৙-1 njo )਑৙-1 * І ঱҄)ʇ҄ઐ* ѿ ܕ৙- ઐܖ
56. ### From Primal to Dual … … and Back 2 3

}}৙ ѿ ʇ৘}}3 , ౠ঱)৘* Ј ৘৙-ౠ njo )ਗ৙-ౠ * ঱҄)ʇ҄ઐ* ѿ ܕ৙- ઐܖ , ౠ 3 }}ઐ}}3 Ј ઐ৙-ౠ njo )਑৙-ౠ * ৘৙-1 njo )ਗ৙-1 * І ঱)৘* s.t ʇ৘ > ৙ ઐ৙-1 njo )਑৙-1 * І ঱҄)ʇ҄ઐ* ѿ ܕ৙- ઐܖ Primal Primal-Limit Dual Dual-Limit
57. ### From Primal to Dual … … and Back 2 3

}}৙ ѿ ʇ৘}}3 , ౠ঱)৘* Ј ৘৙-ౠ njo )ਗ৙-ౠ * ঱҄)ʇ҄ઐ* ѿ ܕ৙- ઐܖ , ౠ 3 }}ઐ}}3 Ј ઐ৙-ౠ njo )਑৙-ౠ * ৘৙-1 njo )ਗ৙-1 * І ঱)৘* s.t ʇ৘ > ৙ ઐ৙-1 njo )਑৙-1 * І ঱҄)ʇ҄ઐ* ѿ ܕ৙- ઐܖ Primal Primal-Limit Dual Dual-Limit
58. ### From Primal to Dual … … and Back 2 3

}}৙ ѿ ʇ৘}}3 , ౠ঱)৘* Ј ৘৙-ౠ njo )ਗ৙-ౠ * ঱҄)ʇ҄ઐ* ѿ ܕ৙- ઐܖ , ౠ 3 }}ઐ}}3 Ј ઐ৙-ౠ njo )਑৙-ౠ * ৘৙-1 njo )ਗ৙-1 * І ঱)৘* s.t ʇ৘ > ৙ ઐ৙-1 njo )਑৙-1 * І ঱҄)ʇ҄ઐ* ѿ ܕ৙- ઐܖ Primal Primal-Limit Dual Dual-Limit Dualization
59. ### From Primal to Dual … … and Back 2 3

}}৙ ѿ ʇ৘}}3 , ౠ঱)৘* Ј ৘৙-ౠ njo )ਗ৙-ౠ * ঱҄)ʇ҄ઐ* ѿ ܕ৙- ઐܖ , ౠ 3 }}ઐ}}3 Ј ઐ৙-ౠ njo )਑৙-ౠ * ৘৙-1 njo )ਗ৙-1 * І ঱)৘* s.t ʇ৘ > ৙ ઐ৙-1 njo )਑৙-1 * І ঱҄)ʇ҄ઐ* ѿ ܕ৙- ઐܖ Primal Primal-Limit Dual Dual-Limit
60. ### Dual Certiﬁcates Proposition • There exists a dual certiﬁcate ܧ

৘ is a solution of )ਗ৙-1 * • ʇઐ ѵ ౯঱)৘* ܧ ઐ solution of )਑৙-1 * Deﬁnition A dual certiﬁcate is a vector ઐ such that ʇ҄ઐ ѵ ౯঱)৘*
61. ### Tight Dual Certiﬁcates (Meet the Relative Interior) Deﬁnition relative boundary

relative interior
62. ### An Intermediate Result* Theorem Assume ʇ҄ઐ ѵ sj ౯঱)৘1 *

and Lfs ʇ Җ ঻৘1 > |1~/ Choosing ౠ > ৄ}}ৗ}}3, ৄ ? 1, for any minimizer ৘Գ of )ਗ৙-ౠ * }}৘Գ ѿ ৘1 }}3 ӑ প)ৄ- ઐ*}}ৗ}}3 / No model selection ! What is missing ? *Huge research topic in the German/Austrian school (Grassmair, Hatmaier, Scherzer, etc.)
63. ### How to Build a Certiﬁcate ? 2 3 }}৙ ѿ

ʇ৘}}3 , ౠ঱)৘* Ј ৘৙-ౠ njo )ਗ৙-ౠ * ʇ҄ઐ* ѿ ܕ৙- ઐܖ , ౠ 3 }}ઐ}}3 Ј ઐ৙-ౠ njo )਑৙-ౠ * ৘৙-1 njo )ਗ৙-1 * І ঱)৘* s.t ʇ৘ > ৙ ઐ৙-1 njo )਑৙-1 * І ঱҄)ʇ҄ઐ* ѿ ܕ৙- ઐܖ one solution several solutions
64. ### How to Build a Certiﬁcate ? 2 3 }}৙ ѿ

ʇ৘}}3 , ౠ঱)৘* Ј ৘৙-ౠ njo )ਗ৙-ౠ * ʇ҄ઐ* ѿ ܕ৙- ઐܖ , ౠ 3 }}ઐ}}3 Ј ઐ৙-ౠ njo )਑৙-ౠ * ৘৙-1 njo )ਗ৙-1 * І ঱)৘* s.t ʇ৘ > ৙ ઐ৙-1 njo )਑৙-1 * І ঱҄)ʇ҄ઐ* ѿ ܕ৙- ઐܖ one solution several solutions Convergence ?
65. ### Minimal-norm Certiﬁcate 2 3 }}৙ ѿ ʇ৘}}3 , ౠ঱)৘* Ј

৘৙-ౠ njo )ਗ৙-ౠ * ʇ҄ઐ* ѿ ܕ৙- ઐܖ , ౠ 3 }}ઐ}}3 Ј ઐ৙-ౠ njo )਑৙-ౠ * ৘৙-1 njo )ਗ৙-1 * І ঱)৘* s.t ʇ৘ > ৙ ઐ৙-1 njo )਑৙-1 * І ঱҄)ʇ҄ઐ* ѿ ܕ৙- ઐܖ Proposition with

67. ### Main Theorem Theorem Assume ঱ is partly smooth at ৘1

relative to Ϥ. Suppose ʇ҄ઐভ ѵ sj ౯঱)৘1 * and Lfs ʇ Җ ঻৘1 > |1~/ There exists প ? 1 such that if nby)ౠ- }}ৗ}}0ౠ* ӑ প- the unique solution ৘Գ of (ਗ৙-ౠ) satisﬁes ৘Գ ѵ Ϥ and }}৘Գ ѿ ৘1 }} > শ)}}ৗ}}*/
68. ### How Tight is This Result ? Theorem Missing part: relative

boundary of the subdifferential Wait for an example

rates

sparsity

sparsity

sparsity

77. ### 1D Total Variation Denoising (Staircasing is everywhere) stable jump unstable

jump w/ staircasing
78. ### Compressed Sensing with Nuclear Norm 1000 1500 2000 2500 0

0.5 1 1.5 2 2.5 3 3.5 4 5 10 15 20 0 0.5 1 1.5 Dashed line : Candes’ threshold Fixed rank Variable measurements Variable rank Fixed measurements

83. ### Take-away Message For a large class of recovery problems, under

some assumptions, one observes that both “energy” recovery and “localization” recovery hold. • Convex Analysis • Geometrical Structure Solutions build around the dual of the constrained problem Partial smoothness is the key
84. ### Thanks for your attention Want more ? Review book chapter:

V., G. Peyré, J. Fadili, Low Complexity Regularizations, LNCS, 2014 Preprint on model selection/consistency: V., G. Peyré, J. Fadili, Manifold Consistency with Partly Smooth Regularizers Special case for analysis sparsity: V., C. Dossal, G. Peyré, J. Fadili, Robust Sparse Analysis Regularization, TIT, 2013