Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Low Complexity Regularizations: A Localization Result

Samuel Vaiter
January 16, 2015

Low Complexity Regularizations: A Localization Result

GT Signal, IRISA, Rennes, January 2015.

Samuel Vaiter

January 16, 2015
Tweet

More Decks by Samuel Vaiter

Other Decks in Science

Transcript

  1. Main Theorem: First Version For a large class of recovery

    problems, under some assumptions to be precised, one observes that both “energy” recovery and “localization” recovery hold. Theorem
  2. Choice of a Regularization Wavelet basis sparsity Total Variation Fused

    Lasso Nuclear norm OSCAR Spread representation Group Sparsity Trace Lasso Sobolev TGV Weighted sparsity Elastic net Ridge Analysis sparsity
  3. Models Models (Convex) Functions vector of fixed support matrix of

    fixed rank vector with same saturation pattern vector with same jump set
  4. Main Theorem: 2nd Version Theorem For a large class of

    variational regularizations, under some assumptions to be precised, one observes that both : Ј ৘Գ ѵ Ϥ৘1 Ј }}৘Գ ѿ ৘1 }} is small
  5. Subdifferential ে)৘* ৘ Ȣ ৘ ে) Ȣ ৘* ে)৘* Ӓ

    ে) Ȣ ৘* , ܕѴে) Ȣ ৘*- ৘ ѿ Ȣ ৘ܖ
  6. Subdifferential ে)৘* ৘ Ȣ ৘ ে) Ȣ ৘* ে)৘* Ӓ

    ে) Ȣ ৘* , ܕৈ- ৘ ѿ Ȣ ৘ܖ
  7. Subdifferential ে)৘* ৘ Ȣ ৘ ে) Ȣ ৘* |ৈ ң

    ে)৘* Ӓ ে) Ȣ ৘* , ܕৈ- ৘ ѿ Ȣ ৘ܖ~ ౯ে) Ȣ ৘* >
  8. First Order Conditions Ȣ ৘ ѵ bshnjo ে)৘* ܧ Ѵে)

    Ȣ ৘* > 1 Euler equation for convex + smooth function
  9. First Order Conditions Ȣ ৘ ѵ bshnjo ে)৘* ܧ Ѵে)

    Ȣ ৘* > 1 Euler equation for convex + smooth function Ȣ ৘ ѵ bshnjo ে)৘* ܧ ౯ে) Ȣ ৘* Ѹ 1 Euler equation for convex + non-smooth function
  10. What’s Important in Sparsity ? Ϥ৘ Ȣ ৘ 1: }}

    Բ }}2 restricted to Ϥ৘ is locally smooth
  11. What’s Important in Sparsity ? Ϥ৘ Ȣ ৘ 1: }}

    Բ }}2 restricted to Ϥ৘ is locally smooth 2: }} Բ }}2 is sharp (non-differentiable) in the direction of ϤԒ ৘
  12. What’s Important in Sparsity ? Ϥ৘ Ȣ ৘ 1: }}

    Բ }}2 restricted to Ϥ৘ is locally smooth 2: }} Բ }}2 is sharp (non-differentiable) in the direction of ϤԒ ৘ 3: ౯}} Բ }}2 restricted to Ϥ৘ is locally continuous (constant here)
  13. Partial Smoothness Definition ঱ is partly smooth at ৘ relative

    to a C3 -manifold Ϥ if • Smoothness. ঱ restricted to Ϥ is C3 around ৘ • Sharpness. ѭυ ѵ )ਚϤ ৘*Ԓ , ৔ М ঱)৘ , ৔υ* is non-smooth at ৔ > 1. • Continuity. ౯঱ on Ϥ is continuous around ৘. Ϥ nby)1- }}৘}} ѿ 2*
  14. Main Theorem: 3rd Version Theorem Ј ৘Գ ѵ Ϥ৘1 Ј

    }}৘Գ ѿ ৘1 }} is small When ঱ is partly smooth at ৘1 relatively to Ϥ৘1 , under some assumptions to be precised, one observes that:
  15. Examples and Algebraic Stability ৘ М }}৘}}2 - }}৘}}3 -

    }}৘}}ҋ - }}৘}}2-3 - nby ৉ )৘৉ *, - any smooth function Ϳ
  16. Examples and Algebraic Stability ৘ М }}৘}}2 - }}৘}}3 -

    }}৘}}ҋ - }}৘}}2-3 - nby ৉ )৘৉ *, - any smooth function Ϳ ঱ and ম partly smooth ܦ ঱ , ম partly smooth }} Բ }}2 , ౰}} Բ }}3 3 partly smooth
  17. Examples and Algebraic Stability ৘ М }}৘}}2 - }}৘}}3 -

    }}৘}}ҋ - }}৘}}2-3 - nby ৉ )৘৉ *, - any smooth function Ϳ ঱ and ম partly smooth ܦ ঱ , ম partly smooth }} Բ }}2 , ౰}} Բ }}3 3 partly smooth ঱ partly smooth and ঳ linear operator ܦ ঱ ҅ ঳ partly smooth }}Ѵ Բ }}2-3 (isotropic TV) partly smooth
  18. Examples and Algebraic Stability ৘ М }}৘}}2 - }}৘}}3 -

    }}৘}}ҋ - }}৘}}2-3 - nby ৉ )৘৉ *, - any smooth function Ϳ ঱ and ম partly smooth ܦ ঱ , ম partly smooth }} Բ }}2 , ౰}} Բ }}3 3 partly smooth ঱ partly smooth and ঳ linear operator ܦ ঱ ҅ ঳ partly smooth }}Ѵ Բ }}2-3 (isotropic TV) partly smooth ঱ partly smooth ܦ ঱ ҅ ౨ partly smooth (spectral lift) }} Բ }}҄ (nuclear/trace norm) partly smooth
  19. Main Ideas: 1) The Lagrangian problem is nothing more than

    the constrained problem up to "something" that we are going to control (by duality)
  20. Main Ideas: 2 ) To r e c o v

    e r a g o o d estimation of the ground truth with noise, it should be identifiable without noise
  21. From Primal to Dual … … and Back 2 3

    }}৙ ѿ ʇ৘}}3 , ౠ঱)৘* Ј ৘৙-ౠ njo )ਗ৙-ౠ * ঱҄)ʇ҄ઐ* ѿ ܕ৙- ઐܖ , ౠ 3 }}ઐ}}3 Ј ઐ৙-ౠ njo )਑৙-ౠ * ৘৙-1 njo )ਗ৙-1 * І ঱)৘* s.t ʇ৘ > ৙ ઐ৙-1 njo )਑৙-1 * І ঱҄)ʇ҄ઐ* ѿ ܕ৙- ઐܖ
  22. From Primal to Dual … … and Back 2 3

    }}৙ ѿ ʇ৘}}3 , ౠ঱)৘* Ј ৘৙-ౠ njo )ਗ৙-ౠ * ঱҄)ʇ҄ઐ* ѿ ܕ৙- ઐܖ , ౠ 3 }}ઐ}}3 Ј ઐ৙-ౠ njo )਑৙-ౠ * ৘৙-1 njo )ਗ৙-1 * І ঱)৘* s.t ʇ৘ > ৙ ઐ৙-1 njo )਑৙-1 * І ঱҄)ʇ҄ઐ* ѿ ܕ৙- ઐܖ Primal Primal-Limit Dual Dual-Limit
  23. From Primal to Dual … … and Back 2 3

    }}৙ ѿ ʇ৘}}3 , ౠ঱)৘* Ј ৘৙-ౠ njo )ਗ৙-ౠ * ঱҄)ʇ҄ઐ* ѿ ܕ৙- ઐܖ , ౠ 3 }}ઐ}}3 Ј ઐ৙-ౠ njo )਑৙-ౠ * ৘৙-1 njo )ਗ৙-1 * І ঱)৘* s.t ʇ৘ > ৙ ઐ৙-1 njo )਑৙-1 * І ঱҄)ʇ҄ઐ* ѿ ܕ৙- ઐܖ Primal Primal-Limit Dual Dual-Limit
  24. From Primal to Dual … … and Back 2 3

    }}৙ ѿ ʇ৘}}3 , ౠ঱)৘* Ј ৘৙-ౠ njo )ਗ৙-ౠ * ঱҄)ʇ҄ઐ* ѿ ܕ৙- ઐܖ , ౠ 3 }}ઐ}}3 Ј ઐ৙-ౠ njo )਑৙-ౠ * ৘৙-1 njo )ਗ৙-1 * І ঱)৘* s.t ʇ৘ > ৙ ઐ৙-1 njo )਑৙-1 * І ঱҄)ʇ҄ઐ* ѿ ܕ৙- ઐܖ Primal Primal-Limit Dual Dual-Limit Dualization
  25. From Primal to Dual … … and Back 2 3

    }}৙ ѿ ʇ৘}}3 , ౠ঱)৘* Ј ৘৙-ౠ njo )ਗ৙-ౠ * ঱҄)ʇ҄ઐ* ѿ ܕ৙- ઐܖ , ౠ 3 }}ઐ}}3 Ј ઐ৙-ౠ njo )਑৙-ౠ * ৘৙-1 njo )ਗ৙-1 * І ঱)৘* s.t ʇ৘ > ৙ ઐ৙-1 njo )਑৙-1 * І ঱҄)ʇ҄ઐ* ѿ ܕ৙- ઐܖ Primal Primal-Limit Dual Dual-Limit
  26. Dual Certificates Proposition • There exists a dual certificate ܧ

    ৘ is a solution of )ਗ৙-1 * • ʇઐ ѵ ౯঱)৘* ܧ ઐ solution of )਑৙-1 * Definition A dual certificate is a vector ઐ such that ʇ҄ઐ ѵ ౯঱)৘*
  27. An Intermediate Result* Theorem Assume ʇ҄ઐ ѵ sj ౯঱)৘1 *

    and Lfs ʇ Җ ঻৘1 > |1~/ Choosing ౠ > ৄ}}ৗ}}3, ৄ ? 1, for any minimizer ৘Գ of )ਗ৙-ౠ * }}৘Գ ѿ ৘1 }}3 ӑ প)ৄ- ઐ*}}ৗ}}3 / No model selection ! What is missing ? *Huge research topic in the German/Austrian school (Grassmair, Hatmaier, Scherzer, etc.)
  28. How to Build a Certificate ? 2 3 }}৙ ѿ

    ʇ৘}}3 , ౠ঱)৘* Ј ৘৙-ౠ njo )ਗ৙-ౠ * ʇ҄ઐ* ѿ ܕ৙- ઐܖ , ౠ 3 }}ઐ}}3 Ј ઐ৙-ౠ njo )਑৙-ౠ * ৘৙-1 njo )ਗ৙-1 * І ঱)৘* s.t ʇ৘ > ৙ ઐ৙-1 njo )਑৙-1 * І ঱҄)ʇ҄ઐ* ѿ ܕ৙- ઐܖ one solution several solutions
  29. How to Build a Certificate ? 2 3 }}৙ ѿ

    ʇ৘}}3 , ౠ঱)৘* Ј ৘৙-ౠ njo )ਗ৙-ౠ * ʇ҄ઐ* ѿ ܕ৙- ઐܖ , ౠ 3 }}ઐ}}3 Ј ઐ৙-ౠ njo )਑৙-ౠ * ৘৙-1 njo )ਗ৙-1 * І ঱)৘* s.t ʇ৘ > ৙ ઐ৙-1 njo )਑৙-1 * І ঱҄)ʇ҄ઐ* ѿ ܕ৙- ઐܖ one solution several solutions Convergence ?
  30. Minimal-norm Certificate 2 3 }}৙ ѿ ʇ৘}}3 , ౠ঱)৘* Ј

    ৘৙-ౠ njo )ਗ৙-ౠ * ʇ҄ઐ* ѿ ܕ৙- ઐܖ , ౠ 3 }}ઐ}}3 Ј ઐ৙-ౠ njo )਑৙-ౠ * ৘৙-1 njo )ਗ৙-1 * І ঱)৘* s.t ʇ৘ > ৙ ઐ৙-1 njo )਑৙-1 * І ঱҄)ʇ҄ઐ* ѿ ܕ৙- ઐܖ Proposition with
  31. Main Theorem Theorem Assume ঱ is partly smooth at ৘1

    relative to Ϥ. Suppose ʇ҄ઐভ ѵ sj ౯঱)৘1 * and Lfs ʇ Җ ঻৘1 > |1~/ There exists প ? 1 such that if nby)ౠ- }}ৗ}}0ౠ* ӑ প- the unique solution ৘Գ of (ਗ৙-ౠ) satisfies ৘Գ ѵ Ϥ and }}৘Գ ѿ ৘1 }} > শ)}}ৗ}}*/
  32. How Tight is This Result ? Theorem Missing part: relative

    boundary of the subdifferential Wait for an example
  33. Compressed Sensing with Nuclear Norm 1000 1500 2000 2500 0

    0.5 1 1.5 2 2.5 3 3.5 4 5 10 15 20 0 0.5 1 1.5 Dashed line : Candes’ threshold Fixed rank Variable measurements Variable rank Fixed measurements
  34. Take-away Message For a large class of recovery problems, under

    some assumptions, one observes that both “energy” recovery and “localization” recovery hold. • Convex Analysis • Geometrical Structure Solutions build around the dual of the constrained problem Partial smoothness is the key
  35. Thanks for your attention Want more ? Review book chapter:

    V., G. Peyré, J. Fadili, Low Complexity Regularizations, LNCS, 2015 Preprint on model selection/consistency: V., G. Peyré, J. Fadili, Manifold Consistency with Partly Smooth Regularizers Special case for analysis sparsity: V., C. Dossal, G. Peyré, J. Fadili, Robust Sparse Analysis Regularization, TIT, 2013