Low Complexity Regularizations: a Localization Result

4807c637e2e5e8a5c5e68b287e8492a9?s=47 Samuel Vaiter
November 13, 2014

Low Complexity Regularizations: a Localization Result

SMATI, Télécom ParisTech, Paris, November 2014.

4807c637e2e5e8a5c5e68b287e8492a9?s=128

Samuel Vaiter

November 13, 2014
Tweet

Transcript

  1. Régularisations de faibles complexités : Un résultat de "localisation" Samuel

    Vaiter 13 November 2014
  2. Ground Truth Observations Recovery

  3. Perfect

  4. Useless

  5. Interesting

  6. What is Robustness ? “Energy” Robustness “Localization” Robustness

  7. Part I Inverse Problems Regularizations Models

  8. Main Theorem: First Version For a large class of recovery

    problems, under some assumptions to be precised, one observes that both “energy” recovery and “localization” recovery hold. Theorem
  9. Linear Inverse Problem (in Finite Dimension) ৙ > ʇ৘1 ,

    ৗ denoising inpainting deblurring
  10. Variational Regularization ৘Գ ѵ bshnjo ৘ѵϓৎ ভ)৘- ৙* , ౠ঱)৘*

    Data fidelity Prior vs
  11. Some Remarks Loss considered: square prediction-error

  12. Some Remarks Loss considered: square prediction-error No noise: use of

    the constrained form
  13. Some Remarks Loss considered: square prediction-error No noise: use of

    the constrained form
  14. Some Remarks Loss considered: square prediction-error No noise: use of

    the constrained form
  15. Choice of a Regularization Wavelet basis sparsity Total Variation Fused

    Lasso Nuclear norm OSCAR Spread representation Group Sparsity Trace Lasso Sobolev TGV Weighted sparsity Elastic net Ridge Analysis sparsity
  16. Models Models (Convex) Functions vector of fixed support matrix of

    fixed rank vector with same saturation pattern vector with same jump set
  17. Sparsity with a Two Pixels Image Ϥ৘

  18. Sparsity with a Two Pixels Image Ϥ৘

  19. Main Theorem: 2nd Version Theorem For a large class of

    variational regularizations, under some assumptions to be precised, one observes that both : Ј ৘Գ ѵ Ϥ৘1 Ј }}৘Գ ѿ ৘1 }} is small
  20. A Little Bit of (Recent) History

  21. Technical Interlude

  22. Subdifferential ে)৘* ৘ Ȣ ৘ ে) Ȣ ৘* ে)৘* Ӓ

    ে) Ȣ ৘* , ܕѴে) Ȣ ৘*- ৘ ѿ Ȣ ৘ܖ
  23. Subdifferential ে)৘* ৘ Ȣ ৘ ে) Ȣ ৘* ে)৘* Ӓ

    ে) Ȣ ৘* , ܕৈ- ৘ ѿ Ȣ ৘ܖ
  24. Subdifferential ে)৘* ৘ Ȣ ৘ ে) Ȣ ৘* |ৈ ң

    ে)৘* Ӓ ে) Ȣ ৘* , ܕৈ- ৘ ѿ Ȣ ৘ܖ~ ౯ে) Ȣ ৘* >
  25. First Order Conditions Ȣ ৘ ѵ bshnjo ে)৘* ܧ Ѵে)

    Ȣ ৘* > 1 Euler equation for convex + smooth function
  26. First Order Conditions Ȣ ৘ ѵ bshnjo ে)৘* ܧ Ѵে)

    Ȣ ৘* > 1 Euler equation for convex + smooth function Ȣ ৘ ѵ bshnjo ে)৘* ܧ ౯ে) Ȣ ৘* Ѹ 1 Euler equation for convex + non-smooth function
  27. Smooth Manifold

  28. Smooth Manifold Tangent space

  29. Smooth Manifold Tangent space Ok, maybe listen Alain instead

  30. Part II Partial Smoothness

  31. What’s Important in Sparsity ? Ϥ৘ |৘ ң ঱)৘* >

    ঱) Ȣ ৘*~ Ȣ ৘
  32. Ȣ ৘ , e৘ What’s Important in Sparsity ? Ϥ৘

    ঱) Ȣ ৘ , e৘*
  33. Ȣ ৘ , e৘ What’s Important in Sparsity ? Ϥ৘

    ঱) Ȣ ৘ , e৘*
  34. What’s Important in Sparsity ? Ϥ৘ Ȣ ৘ 1: }}

    Բ }}2 restricted to Ϥ৘ is locally smooth
  35. What’s Important in Sparsity ? Ϥ৘ ঱) Ȣ ৘ ,

    e৘*
  36. What’s Important in Sparsity ? Ϥ৘ ঱) Ȣ ৘ ,

    e৘*
  37. What’s Important in Sparsity ? Ϥ৘ ঱) Ȣ ৘ ,

    e৘*
  38. What’s Important in Sparsity ? Ϥ৘ Ȣ ৘ 1: }}

    Բ }}2 restricted to Ϥ৘ is locally smooth 2: }} Բ }}2 is sharp (non-differentiable) in the direction of ϤԒ ৘
  39. What’s Important in Sparsity ? Ϥ৘ Ȣ ৘ 1: }}

    Բ }}2 restricted to Ϥ৘ is locally smooth 2: }} Բ }}2 is sharp (non-differentiable) in the direction of ϤԒ ৘ 3: ౯}} Բ }}2 restricted to Ϥ৘ is locally continuous (constant here)
  40. Partial Smoothness Definition ঱ is partly smooth at ৘ relative

    to a C3 -manifold Ϥ if • Smoothness. ঱ restricted to Ϥ is C3 around ৘ • Sharpness. ѭυ ѵ )ਚϤ ৘*Ԓ , ৔ М ঱)৘ , ৔υ* is non-smooth at ৔ > 1. • Continuity. ౯঱ on Ϥ is continuous around ৘. Ϥ nby)1- }}৘}} ѿ 2*
  41. Main Theorem: 3rd Version Theorem Ј ৘Գ ѵ Ϥ৘1 Ј

    }}৘Գ ѿ ৘1 }} is small When ঱ is partly smooth at ৘1 relatively to Ϥ৘1 , under some assumptions to be precised, one observes that:
  42. Examples and Algebraic Stability ৘ М }}৘}}2 - }}৘}}3 -

    }}৘}}ҋ - }}৘}}2-3 - nby ৉ )৘৉ *, - any smooth function Ϳ
  43. Examples and Algebraic Stability ৘ М }}৘}}2 - }}৘}}3 -

    }}৘}}ҋ - }}৘}}2-3 - nby ৉ )৘৉ *, - any smooth function Ϳ ঱ and ম partly smooth ܦ ঱ , ম partly smooth }} Բ }}2 , ౰}} Բ }}3 3 partly smooth
  44. Examples and Algebraic Stability ৘ М }}৘}}2 - }}৘}}3 -

    }}৘}}ҋ - }}৘}}2-3 - nby ৉ )৘৉ *, - any smooth function Ϳ ঱ and ম partly smooth ܦ ঱ , ম partly smooth }} Բ }}2 , ౰}} Բ }}3 3 partly smooth ঱ partly smooth and ঳ linear operator ܦ ঱ ҅ ঳ partly smooth }}Ѵ Բ }}2-3 (isotropic TV) partly smooth
  45. Examples and Algebraic Stability ৘ М }}৘}}2 - }}৘}}3 -

    }}৘}}ҋ - }}৘}}2-3 - nby ৉ )৘৉ *, - any smooth function Ϳ ঱ and ম partly smooth ܦ ঱ , ম partly smooth }} Բ }}2 , ౰}} Բ }}3 3 partly smooth ঱ partly smooth and ঳ linear operator ܦ ঱ ҅ ঳ partly smooth }}Ѵ Բ }}2-3 (isotropic TV) partly smooth ঱ partly smooth ܦ ঱ ҅ ౨ partly smooth (spectral lift) }} Բ }}҄ (nuclear/trace norm) partly smooth
  46. Part III Certificates Restricted Injectivity and Main Result

  47. Assumptions ?

  48. Main Ideas: 1) The Lagrangian problem is nothing more than

    the constrained problem up to "something" that we are going to control (by duality)
  49. Main Ideas: 2 ) To r e c o v

    e r a g o o d estimation of the ground truth with noise, it should be identifiable without noise
  50. Restricted Injectivity

  51. Restricted Injectivity

  52. Restricted Injectivity

  53. Restricted Injectivity

  54. Restricted Injectivity

  55. From Primal to Dual … … and Back 2 3

    }}৙ ѿ ʇ৘}}3 , ౠ঱)৘* Ј ৘৙-ౠ njo )ਗ৙-ౠ * ঱҄)ʇ҄ઐ* ѿ ܕ৙- ઐܖ , ౠ 3 }}ઐ}}3 Ј ઐ৙-ౠ njo )਑৙-ౠ * ৘৙-1 njo )ਗ৙-1 * І ঱)৘* s.t ʇ৘ > ৙ ઐ৙-1 njo )਑৙-1 * І ঱҄)ʇ҄ઐ* ѿ ܕ৙- ઐܖ
  56. From Primal to Dual … … and Back 2 3

    }}৙ ѿ ʇ৘}}3 , ౠ঱)৘* Ј ৘৙-ౠ njo )ਗ৙-ౠ * ঱҄)ʇ҄ઐ* ѿ ܕ৙- ઐܖ , ౠ 3 }}ઐ}}3 Ј ઐ৙-ౠ njo )਑৙-ౠ * ৘৙-1 njo )ਗ৙-1 * І ঱)৘* s.t ʇ৘ > ৙ ઐ৙-1 njo )਑৙-1 * І ঱҄)ʇ҄ઐ* ѿ ܕ৙- ઐܖ Primal Primal-Limit Dual Dual-Limit
  57. From Primal to Dual … … and Back 2 3

    }}৙ ѿ ʇ৘}}3 , ౠ঱)৘* Ј ৘৙-ౠ njo )ਗ৙-ౠ * ঱҄)ʇ҄ઐ* ѿ ܕ৙- ઐܖ , ౠ 3 }}ઐ}}3 Ј ઐ৙-ౠ njo )਑৙-ౠ * ৘৙-1 njo )ਗ৙-1 * І ঱)৘* s.t ʇ৘ > ৙ ઐ৙-1 njo )਑৙-1 * І ঱҄)ʇ҄ઐ* ѿ ܕ৙- ઐܖ Primal Primal-Limit Dual Dual-Limit
  58. From Primal to Dual … … and Back 2 3

    }}৙ ѿ ʇ৘}}3 , ౠ঱)৘* Ј ৘৙-ౠ njo )ਗ৙-ౠ * ঱҄)ʇ҄ઐ* ѿ ܕ৙- ઐܖ , ౠ 3 }}ઐ}}3 Ј ઐ৙-ౠ njo )਑৙-ౠ * ৘৙-1 njo )ਗ৙-1 * І ঱)৘* s.t ʇ৘ > ৙ ઐ৙-1 njo )਑৙-1 * І ঱҄)ʇ҄ઐ* ѿ ܕ৙- ઐܖ Primal Primal-Limit Dual Dual-Limit Dualization
  59. From Primal to Dual … … and Back 2 3

    }}৙ ѿ ʇ৘}}3 , ౠ঱)৘* Ј ৘৙-ౠ njo )ਗ৙-ౠ * ঱҄)ʇ҄ઐ* ѿ ܕ৙- ઐܖ , ౠ 3 }}ઐ}}3 Ј ઐ৙-ౠ njo )਑৙-ౠ * ৘৙-1 njo )ਗ৙-1 * І ঱)৘* s.t ʇ৘ > ৙ ઐ৙-1 njo )਑৙-1 * І ঱҄)ʇ҄ઐ* ѿ ܕ৙- ઐܖ Primal Primal-Limit Dual Dual-Limit
  60. Dual Certificates Proposition • There exists a dual certificate ܧ

    ৘ is a solution of )ਗ৙-1 * • ʇઐ ѵ ౯঱)৘* ܧ ઐ solution of )਑৙-1 * Definition A dual certificate is a vector ઐ such that ʇ҄ઐ ѵ ౯঱)৘*
  61. Tight Dual Certificates (Meet the Relative Interior) Definition relative boundary

    relative interior
  62. An Intermediate Result* Theorem Assume ʇ҄ઐ ѵ sj ౯঱)৘1 *

    and Lfs ʇ Җ ঻৘1 > |1~/ Choosing ౠ > ৄ}}ৗ}}3, ৄ ? 1, for any minimizer ৘Գ of )ਗ৙-ౠ * }}৘Գ ѿ ৘1 }}3 ӑ প)ৄ- ઐ*}}ৗ}}3 / No model selection ! What is missing ? *Huge research topic in the German/Austrian school (Grassmair, Hatmaier, Scherzer, etc.)
  63. How to Build a Certificate ? 2 3 }}৙ ѿ

    ʇ৘}}3 , ౠ঱)৘* Ј ৘৙-ౠ njo )ਗ৙-ౠ * ʇ҄ઐ* ѿ ܕ৙- ઐܖ , ౠ 3 }}ઐ}}3 Ј ઐ৙-ౠ njo )਑৙-ౠ * ৘৙-1 njo )ਗ৙-1 * І ঱)৘* s.t ʇ৘ > ৙ ઐ৙-1 njo )਑৙-1 * І ঱҄)ʇ҄ઐ* ѿ ܕ৙- ઐܖ one solution several solutions
  64. How to Build a Certificate ? 2 3 }}৙ ѿ

    ʇ৘}}3 , ౠ঱)৘* Ј ৘৙-ౠ njo )ਗ৙-ౠ * ʇ҄ઐ* ѿ ܕ৙- ઐܖ , ౠ 3 }}ઐ}}3 Ј ઐ৙-ౠ njo )਑৙-ౠ * ৘৙-1 njo )ਗ৙-1 * І ঱)৘* s.t ʇ৘ > ৙ ઐ৙-1 njo )਑৙-1 * І ঱҄)ʇ҄ઐ* ѿ ܕ৙- ઐܖ one solution several solutions Convergence ?
  65. Minimal-norm Certificate 2 3 }}৙ ѿ ʇ৘}}3 , ౠ঱)৘* Ј

    ৘৙-ౠ njo )ਗ৙-ౠ * ʇ҄ઐ* ѿ ܕ৙- ઐܖ , ౠ 3 }}ઐ}}3 Ј ઐ৙-ౠ njo )਑৙-ౠ * ৘৙-1 njo )ਗ৙-1 * І ঱)৘* s.t ʇ৘ > ৙ ઐ৙-1 njo )਑৙-1 * І ঱҄)ʇ҄ઐ* ѿ ܕ৙- ઐܖ Proposition with
  66. Linearized Precertificate Definition Proposition

  67. Main Theorem Theorem Assume ঱ is partly smooth at ৘1

    relative to Ϥ. Suppose ʇ҄ઐভ ѵ sj ౯঱)৘1 * and Lfs ʇ Җ ঻৘1 > |1~/ There exists প ? 1 such that if nby)ౠ- }}ৗ}}0ౠ* ӑ প- the unique solution ৘Գ of (ਗ৙-ౠ) satisfies ৘Գ ѵ Ϥ and }}৘Գ ѿ ৘1 }} > শ)}}ৗ}}*/
  68. How Tight is This Result ? Theorem Missing part: relative

    boundary of the subdifferential Wait for an example
  69. Implications for First-Order Methods Forward-Backward splitting

  70. Implications for First-Order Methods Forward-Backward splitting Theorem

  71. Variations • Non-deterministic setting • General loss function • Convergence

    rates
  72. Sparse Spike Deconvolution (a.k.a 1D Stars Recovery) “Natural” prior :

    sparsity
  73. Sparse Spike Deconvolution (a.k.a 1D Stars Recovery) “Natural” prior :

    sparsity
  74. Sparse Spike Deconvolution (a.k.a 1D Stars Recovery) “Natural” prior :

    sparsity
  75. 1D Total Variation Denoising (Staircasing is everywhere)

  76. 1D Total Variation Denoising (Staircasing is everywhere)

  77. 1D Total Variation Denoising (Staircasing is everywhere) stable jump unstable

    jump w/ staircasing
  78. Compressed Sensing with Nuclear Norm 1000 1500 2000 2500 0

    0.5 1 1.5 2 2.5 3 3.5 4 5 10 15 20 0 0.5 1 1.5 Dashed line : Candes’ threshold Fixed rank Variable measurements Variable rank Fixed measurements
  79. Perspectives & Conclusion

  80. Mandatory Lena Time

  81. Mandatory Lena Time Linearized precertificate: not a certificate (in general)

  82. Mandatory Lena Time Linearized precertificate: not a certificate (in general)

  83. Take-away Message For a large class of recovery problems, under

    some assumptions, one observes that both “energy” recovery and “localization” recovery hold. • Convex Analysis • Geometrical Structure Solutions build around the dual of the constrained problem Partial smoothness is the key
  84. Thanks for your attention Want more ? Review book chapter:

    V., G. Peyré, J. Fadili, Low Complexity Regularizations, LNCS, 2014 Preprint on model selection/consistency: V., G. Peyré, J. Fadili, Manifold Consistency with Partly Smooth Regularizers Special case for analysis sparsity: V., C. Dossal, G. Peyré, J. Fadili, Robust Sparse Analysis Regularization, TIT, 2013