Low Complexity Regularizations: A Localization Result

4807c637e2e5e8a5c5e68b287e8492a9?s=47 Samuel Vaiter
January 16, 2015

Low Complexity Regularizations: A Localization Result

GT Signal, IRISA, Rennes, January 2015.

4807c637e2e5e8a5c5e68b287e8492a9?s=128

Samuel Vaiter

January 16, 2015
Tweet

Transcript

  1. Régularisations de faibles complexités : Un résultat de "localisation" Samuel

    Vaiter 16 Janvier 2015
  2. Ground Truth Observations Recovery

  3. Perfect

  4. Useless

  5. Interesting

  6. What is Robustness ? “Energy” Robustness “Localization” Robustness

  7. Part I Inverse Problems Regularizations Models

  8. Main Theorem: First Version For a large class of recovery

    problems, under some assumptions to be precised, one observes that both “energy” recovery and “localization” recovery hold. Theorem
  9. Linear Inverse Problem (in Finite Dimension) ৙ > ʇ৘1 ,

    ৗ denoising inpainting deblurring
  10. Variational Regularization ৘Գ ѵ bshnjo ৘ѵϓৎ ভ)৘- ৙* , ౠ঱)৘*

    Data fidelity Prior vs
  11. Some Remarks Loss considered: square prediction-error

  12. Some Remarks Loss considered: square prediction-error No noise: use of

    the constrained form
  13. Some Remarks Loss considered: square prediction-error No noise: use of

    the constrained form
  14. Some Remarks Loss considered: square prediction-error No noise: use of

    the constrained form
  15. Choice of a Regularization Wavelet basis sparsity Total Variation Fused

    Lasso Nuclear norm OSCAR Spread representation Group Sparsity Trace Lasso Sobolev TGV Weighted sparsity Elastic net Ridge Analysis sparsity
  16. Models Models (Convex) Functions vector of fixed support matrix of

    fixed rank vector with same saturation pattern vector with same jump set
  17. Sparsity with a Two Pixels Image Ϥ৘

  18. Sparsity with a Two Pixels Image Ϥ৘

  19. Main Theorem: 2nd Version Theorem For a large class of

    variational regularizations, under some assumptions to be precised, one observes that both : Ј ৘Գ ѵ Ϥ৘1 Ј }}৘Գ ѿ ৘1 }} is small
  20. A Little Bit of (Recent) History

  21. Technical Interlude

  22. Subdifferential ে)৘* ৘ Ȣ ৘ ে) Ȣ ৘* ে)৘* Ӓ

    ে) Ȣ ৘* , ܕѴে) Ȣ ৘*- ৘ ѿ Ȣ ৘ܖ
  23. Subdifferential ে)৘* ৘ Ȣ ৘ ে) Ȣ ৘* ে)৘* Ӓ

    ে) Ȣ ৘* , ܕৈ- ৘ ѿ Ȣ ৘ܖ
  24. Subdifferential ে)৘* ৘ Ȣ ৘ ে) Ȣ ৘* |ৈ ң

    ে)৘* Ӓ ে) Ȣ ৘* , ܕৈ- ৘ ѿ Ȣ ৘ܖ~ ౯ে) Ȣ ৘* >
  25. First Order Conditions Ȣ ৘ ѵ bshnjo ে)৘* ܧ Ѵে)

    Ȣ ৘* > 1 Euler equation for convex + smooth function
  26. First Order Conditions Ȣ ৘ ѵ bshnjo ে)৘* ܧ Ѵে)

    Ȣ ৘* > 1 Euler equation for convex + smooth function Ȣ ৘ ѵ bshnjo ে)৘* ܧ ౯ে) Ȣ ৘* Ѹ 1 Euler equation for convex + non-smooth function
  27. Smooth Manifold

  28. Smooth Manifold Tangent space

  29. Part II Partial Smoothness

  30. What’s Important in Sparsity ? Ϥ৘ |৘ ң ঱)৘* >

    ঱) Ȣ ৘*~ Ȣ ৘
  31. Ȣ ৘ , e৘ What’s Important in Sparsity ? Ϥ৘

    ঱) Ȣ ৘ , e৘*
  32. Ȣ ৘ , e৘ What’s Important in Sparsity ? Ϥ৘

    ঱) Ȣ ৘ , e৘*
  33. What’s Important in Sparsity ? Ϥ৘ Ȣ ৘ 1: }}

    Բ }}2 restricted to Ϥ৘ is locally smooth
  34. What’s Important in Sparsity ? Ϥ৘ ঱) Ȣ ৘ ,

    e৘*
  35. What’s Important in Sparsity ? Ϥ৘ ঱) Ȣ ৘ ,

    e৘*
  36. What’s Important in Sparsity ? Ϥ৘ ঱) Ȣ ৘ ,

    e৘*
  37. What’s Important in Sparsity ? Ϥ৘ Ȣ ৘ 1: }}

    Բ }}2 restricted to Ϥ৘ is locally smooth 2: }} Բ }}2 is sharp (non-differentiable) in the direction of ϤԒ ৘
  38. What’s Important in Sparsity ? Ϥ৘ Ȣ ৘ 1: }}

    Բ }}2 restricted to Ϥ৘ is locally smooth 2: }} Բ }}2 is sharp (non-differentiable) in the direction of ϤԒ ৘ 3: ౯}} Բ }}2 restricted to Ϥ৘ is locally continuous (constant here)
  39. Partial Smoothness Definition ঱ is partly smooth at ৘ relative

    to a C3 -manifold Ϥ if • Smoothness. ঱ restricted to Ϥ is C3 around ৘ • Sharpness. ѭυ ѵ )ਚϤ ৘*Ԓ , ৔ М ঱)৘ , ৔υ* is non-smooth at ৔ > 1. • Continuity. ౯঱ on Ϥ is continuous around ৘. Ϥ nby)1- }}৘}} ѿ 2*
  40. Main Theorem: 3rd Version Theorem Ј ৘Գ ѵ Ϥ৘1 Ј

    }}৘Գ ѿ ৘1 }} is small When ঱ is partly smooth at ৘1 relatively to Ϥ৘1 , under some assumptions to be precised, one observes that:
  41. Examples and Algebraic Stability ৘ М }}৘}}2 - }}৘}}3 -

    }}৘}}ҋ - }}৘}}2-3 - nby ৉ )৘৉ *, - any smooth function Ϳ
  42. Examples and Algebraic Stability ৘ М }}৘}}2 - }}৘}}3 -

    }}৘}}ҋ - }}৘}}2-3 - nby ৉ )৘৉ *, - any smooth function Ϳ ঱ and ম partly smooth ܦ ঱ , ম partly smooth }} Բ }}2 , ౰}} Բ }}3 3 partly smooth
  43. Examples and Algebraic Stability ৘ М }}৘}}2 - }}৘}}3 -

    }}৘}}ҋ - }}৘}}2-3 - nby ৉ )৘৉ *, - any smooth function Ϳ ঱ and ম partly smooth ܦ ঱ , ম partly smooth }} Բ }}2 , ౰}} Բ }}3 3 partly smooth ঱ partly smooth and ঳ linear operator ܦ ঱ ҅ ঳ partly smooth }}Ѵ Բ }}2-3 (isotropic TV) partly smooth
  44. Examples and Algebraic Stability ৘ М }}৘}}2 - }}৘}}3 -

    }}৘}}ҋ - }}৘}}2-3 - nby ৉ )৘৉ *, - any smooth function Ϳ ঱ and ম partly smooth ܦ ঱ , ম partly smooth }} Բ }}2 , ౰}} Բ }}3 3 partly smooth ঱ partly smooth and ঳ linear operator ܦ ঱ ҅ ঳ partly smooth }}Ѵ Բ }}2-3 (isotropic TV) partly smooth ঱ partly smooth ܦ ঱ ҅ ౨ partly smooth (spectral lift) }} Բ }}҄ (nuclear/trace norm) partly smooth
  45. Part III Certificates Restricted Injectivity and Main Result

  46. Assumptions ?

  47. Main Ideas: 1) The Lagrangian problem is nothing more than

    the constrained problem up to "something" that we are going to control (by duality)
  48. Main Ideas: 2 ) To r e c o v

    e r a g o o d estimation of the ground truth with noise, it should be identifiable without noise
  49. Restricted Injectivity

  50. Restricted Injectivity

  51. Restricted Injectivity

  52. Restricted Injectivity

  53. Restricted Injectivity

  54. From Primal to Dual … … and Back 2 3

    }}৙ ѿ ʇ৘}}3 , ౠ঱)৘* Ј ৘৙-ౠ njo )ਗ৙-ౠ * ঱҄)ʇ҄ઐ* ѿ ܕ৙- ઐܖ , ౠ 3 }}ઐ}}3 Ј ઐ৙-ౠ njo )਑৙-ౠ * ৘৙-1 njo )ਗ৙-1 * І ঱)৘* s.t ʇ৘ > ৙ ઐ৙-1 njo )਑৙-1 * І ঱҄)ʇ҄ઐ* ѿ ܕ৙- ઐܖ
  55. From Primal to Dual … … and Back 2 3

    }}৙ ѿ ʇ৘}}3 , ౠ঱)৘* Ј ৘৙-ౠ njo )ਗ৙-ౠ * ঱҄)ʇ҄ઐ* ѿ ܕ৙- ઐܖ , ౠ 3 }}ઐ}}3 Ј ઐ৙-ౠ njo )਑৙-ౠ * ৘৙-1 njo )ਗ৙-1 * І ঱)৘* s.t ʇ৘ > ৙ ઐ৙-1 njo )਑৙-1 * І ঱҄)ʇ҄ઐ* ѿ ܕ৙- ઐܖ Primal Primal-Limit Dual Dual-Limit
  56. From Primal to Dual … … and Back 2 3

    }}৙ ѿ ʇ৘}}3 , ౠ঱)৘* Ј ৘৙-ౠ njo )ਗ৙-ౠ * ঱҄)ʇ҄ઐ* ѿ ܕ৙- ઐܖ , ౠ 3 }}ઐ}}3 Ј ઐ৙-ౠ njo )਑৙-ౠ * ৘৙-1 njo )ਗ৙-1 * І ঱)৘* s.t ʇ৘ > ৙ ઐ৙-1 njo )਑৙-1 * І ঱҄)ʇ҄ઐ* ѿ ܕ৙- ઐܖ Primal Primal-Limit Dual Dual-Limit
  57. From Primal to Dual … … and Back 2 3

    }}৙ ѿ ʇ৘}}3 , ౠ঱)৘* Ј ৘৙-ౠ njo )ਗ৙-ౠ * ঱҄)ʇ҄ઐ* ѿ ܕ৙- ઐܖ , ౠ 3 }}ઐ}}3 Ј ઐ৙-ౠ njo )਑৙-ౠ * ৘৙-1 njo )ਗ৙-1 * І ঱)৘* s.t ʇ৘ > ৙ ઐ৙-1 njo )਑৙-1 * І ঱҄)ʇ҄ઐ* ѿ ܕ৙- ઐܖ Primal Primal-Limit Dual Dual-Limit Dualization
  58. From Primal to Dual … … and Back 2 3

    }}৙ ѿ ʇ৘}}3 , ౠ঱)৘* Ј ৘৙-ౠ njo )ਗ৙-ౠ * ঱҄)ʇ҄ઐ* ѿ ܕ৙- ઐܖ , ౠ 3 }}ઐ}}3 Ј ઐ৙-ౠ njo )਑৙-ౠ * ৘৙-1 njo )ਗ৙-1 * І ঱)৘* s.t ʇ৘ > ৙ ઐ৙-1 njo )਑৙-1 * І ঱҄)ʇ҄ઐ* ѿ ܕ৙- ઐܖ Primal Primal-Limit Dual Dual-Limit
  59. Dual Certificates Proposition • There exists a dual certificate ܧ

    ৘ is a solution of )ਗ৙-1 * • ʇઐ ѵ ౯঱)৘* ܧ ઐ solution of )਑৙-1 * Definition A dual certificate is a vector ઐ such that ʇ҄ઐ ѵ ౯঱)৘*
  60. Tight Dual Certificates (Meet the Relative Interior) Definition relative boundary

    relative interior
  61. An Intermediate Result* Theorem Assume ʇ҄ઐ ѵ sj ౯঱)৘1 *

    and Lfs ʇ Җ ঻৘1 > |1~/ Choosing ౠ > ৄ}}ৗ}}3, ৄ ? 1, for any minimizer ৘Գ of )ਗ৙-ౠ * }}৘Գ ѿ ৘1 }}3 ӑ প)ৄ- ઐ*}}ৗ}}3 / No model selection ! What is missing ? *Huge research topic in the German/Austrian school (Grassmair, Hatmaier, Scherzer, etc.)
  62. How to Build a Certificate ? 2 3 }}৙ ѿ

    ʇ৘}}3 , ౠ঱)৘* Ј ৘৙-ౠ njo )ਗ৙-ౠ * ʇ҄ઐ* ѿ ܕ৙- ઐܖ , ౠ 3 }}ઐ}}3 Ј ઐ৙-ౠ njo )਑৙-ౠ * ৘৙-1 njo )ਗ৙-1 * І ঱)৘* s.t ʇ৘ > ৙ ઐ৙-1 njo )਑৙-1 * І ঱҄)ʇ҄ઐ* ѿ ܕ৙- ઐܖ one solution several solutions
  63. How to Build a Certificate ? 2 3 }}৙ ѿ

    ʇ৘}}3 , ౠ঱)৘* Ј ৘৙-ౠ njo )ਗ৙-ౠ * ʇ҄ઐ* ѿ ܕ৙- ઐܖ , ౠ 3 }}ઐ}}3 Ј ઐ৙-ౠ njo )਑৙-ౠ * ৘৙-1 njo )ਗ৙-1 * І ঱)৘* s.t ʇ৘ > ৙ ઐ৙-1 njo )਑৙-1 * І ঱҄)ʇ҄ઐ* ѿ ܕ৙- ઐܖ one solution several solutions Convergence ?
  64. Minimal-norm Certificate 2 3 }}৙ ѿ ʇ৘}}3 , ౠ঱)৘* Ј

    ৘৙-ౠ njo )ਗ৙-ౠ * ʇ҄ઐ* ѿ ܕ৙- ઐܖ , ౠ 3 }}ઐ}}3 Ј ઐ৙-ౠ njo )਑৙-ౠ * ৘৙-1 njo )ਗ৙-1 * І ঱)৘* s.t ʇ৘ > ৙ ઐ৙-1 njo )਑৙-1 * І ঱҄)ʇ҄ઐ* ѿ ܕ৙- ઐܖ Proposition with
  65. Linearized Precertificate Definition Proposition

  66. Main Theorem Theorem Assume ঱ is partly smooth at ৘1

    relative to Ϥ. Suppose ʇ҄ઐভ ѵ sj ౯঱)৘1 * and Lfs ʇ Җ ঻৘1 > |1~/ There exists প ? 1 such that if nby)ౠ- }}ৗ}}0ౠ* ӑ প- the unique solution ৘Գ of (ਗ৙-ౠ) satisfies ৘Գ ѵ Ϥ and }}৘Գ ѿ ৘1 }} > শ)}}ৗ}}*/
  67. How Tight is This Result ? Theorem Missing part: relative

    boundary of the subdifferential Wait for an example
  68. Implications for First-Order Methods Forward-Backward splitting

  69. Implications for First-Order Methods Forward-Backward splitting Theorem

  70. Variations • Non-deterministic setting • General loss function • Convergence

    rates
  71. Sparse Spike Deconvolution (a.k.a 1D Stars Recovery) “Natural” prior :

    sparsity
  72. Sparse Spike Deconvolution (a.k.a 1D Stars Recovery) “Natural” prior :

    sparsity
  73. Sparse Spike Deconvolution (a.k.a 1D Stars Recovery) “Natural” prior :

    sparsity
  74. 1D Total Variation Denoising (Staircasing is everywhere)

  75. 1D Total Variation Denoising (Staircasing is everywhere)

  76. 1D Total Variation Denoising (Staircasing is everywhere) stable jump unstable

    jump w/ staircasing
  77. Compressed Sensing with Nuclear Norm 1000 1500 2000 2500 0

    0.5 1 1.5 2 2.5 3 3.5 4 5 10 15 20 0 0.5 1 1.5 Dashed line : Candes’ threshold Fixed rank Variable measurements Variable rank Fixed measurements
  78. Perspectives & Conclusion

  79. Mandatory Lena Time

  80. Mandatory Lena Time Linearized precertificate: not a certificate (in general)

  81. Mandatory Lena Time Linearized precertificate: not a certificate (in general)

  82. Take-away Message For a large class of recovery problems, under

    some assumptions, one observes that both “energy” recovery and “localization” recovery hold. • Convex Analysis • Geometrical Structure Solutions build around the dual of the constrained problem Partial smoothness is the key
  83. Thanks for your attention Want more ? Review book chapter:

    V., G. Peyré, J. Fadili, Low Complexity Regularizations, LNCS, 2015 Preprint on model selection/consistency: V., G. Peyré, J. Fadili, Manifold Consistency with Partly Smooth Regularizers Special case for analysis sparsity: V., C. Dossal, G. Peyré, J. Fadili, Robust Sparse Analysis Regularization, TIT, 2013