Low Complexity Regularizations: a Localization Result

SMATI, Télécom ParisTech, Paris, November 2014.

Transcript

1. Régularisations de faibles complexités : Un résultat de "localisation" Samuel Vaiter 13 November 2014
2. Ground Truth Observations Recovery
3. Perfect
4. Useless
5. Interesting
6. What is Robustness ? “Energy” Robustness “Localization” Robustness
7. Part I Inverse Problems Regularizations Models
8. Main Theorem: First Version For a large class of recovery problems, under some assumptions to be precised, one observes that both “energy” recovery and “localization” recovery hold. Theorem
9. Linear Inverse Problem (in Finite Dimension) ৙ > ʇ৘1 , ৗ denoising inpainting deblurring
10. Variational Regularization ৘Գ ѵ bshnjo ৘ѵϓৎ ভ)৘- ৙* , ౠ঱)৘* Data fidelity Prior vs
11. Some Remarks Loss considered: square prediction-error
12. Some Remarks Loss considered: square prediction-error No noise: use of the constrained form
13. Some Remarks Loss considered: square prediction-error No noise: use of the constrained form
14. Some Remarks Loss considered: square prediction-error No noise: use of the constrained form
15. Choice of a Regularization Wavelet basis sparsity Total Variation Fused Lasso Nuclear norm OSCAR Spread representation Group Sparsity Trace Lasso Sobolev TGV Weighted sparsity Elastic net Ridge Analysis sparsity
16. Models Models (Convex) Functions vector of fixed support matrix of fixed rank vector with same saturation pattern vector with same jump set
17. Sparsity with a Two Pixels Image Ϥ৘
18. Sparsity with a Two Pixels Image Ϥ৘
19. Main Theorem: 2nd Version Theorem For a large class of variational regularizations, under some assumptions to be precised, one observes that both : Ј ৘Գ ѵ Ϥ৘1 Ј }}৘Գ ѿ ৘1 }} is small
20. A Little Bit of (Recent) History
21. Technical Interlude
22. Subdifferential ে)৘* ৘ Ȣ ৘ ে) Ȣ ৘* ে)৘* Ӓ ে) Ȣ ৘* , ܕѴে) Ȣ ৘*- ৘ ѿ Ȣ ৘ܖ
23. Subdifferential ে)৘* ৘ Ȣ ৘ ে) Ȣ ৘* ে)৘* Ӓ ে) Ȣ ৘* , ܕৈ- ৘ ѿ Ȣ ৘ܖ
24. Subdifferential ে)৘* ৘ Ȣ ৘ ে) Ȣ ৘* |ৈ ң ে)৘* Ӓ ে) Ȣ ৘* , ܕৈ- ৘ ѿ Ȣ ৘ܖ~ ౯ে) Ȣ ৘* >
25. First Order Conditions Ȣ ৘ ѵ bshnjo ে)৘* ܧ Ѵে) Ȣ ৘* > 1 Euler equation for convex + smooth function
26. First Order Conditions Ȣ ৘ ѵ bshnjo ে)৘* ܧ Ѵে) Ȣ ৘* > 1 Euler equation for convex + smooth function Ȣ ৘ ѵ bshnjo ে)৘* ܧ ౯ে) Ȣ ৘* Ѹ 1 Euler equation for convex + non-smooth function
27. Smooth Manifold
28. Smooth Manifold Tangent space
29. Smooth Manifold Tangent space Ok, maybe listen Alain instead
30. Part II Partial Smoothness
31. What’s Important in Sparsity ? Ϥ৘ |৘ ң ঱)৘* > ঱) Ȣ ৘*~ Ȣ ৘
32. Ȣ ৘ , e৘ What’s Important in Sparsity ? Ϥ৘ ঱) Ȣ ৘ , e৘*
33. Ȣ ৘ , e৘ What’s Important in Sparsity ? Ϥ৘ ঱) Ȣ ৘ , e৘*
34. What’s Important in Sparsity ? Ϥ৘ Ȣ ৘ 1: }} Բ }}2 restricted to Ϥ৘ is locally smooth
35. What’s Important in Sparsity ? Ϥ৘ ঱) Ȣ ৘ , e৘*
36. What’s Important in Sparsity ? Ϥ৘ ঱) Ȣ ৘ , e৘*
37. What’s Important in Sparsity ? Ϥ৘ ঱) Ȣ ৘ , e৘*
38. What’s Important in Sparsity ? Ϥ৘ Ȣ ৘ 1: }} Բ }}2 restricted to Ϥ৘ is locally smooth 2: }} Բ }}2 is sharp (non-differentiable) in the direction of ϤԒ ৘
39. What’s Important in Sparsity ? Ϥ৘ Ȣ ৘ 1: }} Բ }}2 restricted to Ϥ৘ is locally smooth 2: }} Բ }}2 is sharp (non-differentiable) in the direction of ϤԒ ৘ 3: ౯}} Բ }}2 restricted to Ϥ৘ is locally continuous (constant here)
40. Partial Smoothness Definition ঱ is partly smooth at ৘ relative to a C3 -manifold Ϥ if • Smoothness. ঱ restricted to Ϥ is C3 around ৘ • Sharpness. ѭυ ѵ )ਚϤ ৘*Ԓ , ৔ М ঱)৘ , ৔υ* is non-smooth at ৔ > 1. • Continuity. ౯঱ on Ϥ is continuous around ৘. Ϥ nby)1- }}৘}} ѿ 2*
41. Main Theorem: 3rd Version Theorem Ј ৘Գ ѵ Ϥ৘1 Ј }}৘Գ ѿ ৘1 }} is small When ঱ is partly smooth at ৘1 relatively to Ϥ৘1 , under some assumptions to be precised, one observes that:
42. Examples and Algebraic Stability ৘ М }}৘}}2 - }}৘}}3 - }}৘}}ҋ - }}৘}}2-3 - nby ৉ )৘৉ *, - any smooth function Ϳ
43. Examples and Algebraic Stability ৘ М }}৘}}2 - }}৘}}3 - }}৘}}ҋ - }}৘}}2-3 - nby ৉ )৘৉ *, - any smooth function Ϳ ঱ and ম partly smooth ܦ ঱ , ম partly smooth }} Բ }}2 , ౰}} Բ }}3 3 partly smooth
44. Examples and Algebraic Stability ৘ М }}৘}}2 - }}৘}}3 - }}৘}}ҋ - }}৘}}2-3 - nby ৉ )৘৉ *, - any smooth function Ϳ ঱ and ম partly smooth ܦ ঱ , ম partly smooth }} Բ }}2 , ౰}} Բ }}3 3 partly smooth ঱ partly smooth and ঳ linear operator ܦ ঱ ҅ ঳ partly smooth }}Ѵ Բ }}2-3 (isotropic TV) partly smooth
45. Examples and Algebraic Stability ৘ М }}৘}}2 - }}৘}}3 - }}৘}}ҋ - }}৘}}2-3 - nby ৉ )৘৉ *, - any smooth function Ϳ ঱ and ম partly smooth ܦ ঱ , ম partly smooth }} Բ }}2 , ౰}} Բ }}3 3 partly smooth ঱ partly smooth and ঳ linear operator ܦ ঱ ҅ ঳ partly smooth }}Ѵ Բ }}2-3 (isotropic TV) partly smooth ঱ partly smooth ܦ ঱ ҅ ౨ partly smooth (spectral lift) }} Բ }}҄ (nuclear/trace norm) partly smooth
46. Part III Certificates Restricted Injectivity and Main Result
47. Assumptions ?
48. Main Ideas: 1) The Lagrangian problem is nothing more than the constrained problem up to "something" that we are going to control (by duality)
49. Main Ideas: 2 ) To r e c o v e r a g o o d estimation of the ground truth with noise, it should be identifiable without noise
50. Restricted Injectivity
51. Restricted Injectivity
52. Restricted Injectivity
53. Restricted Injectivity
54. Restricted Injectivity
55. From Primal to Dual … … and Back 2 3 }}৙ ѿ ʇ৘}}3 , ౠ঱)৘* Ј ৘৙-ౠ njo )ਗ৙-ౠ * ঱҄)ʇ҄ઐ* ѿ ܕ৙- ઐܖ , ౠ 3 }}ઐ}}3 Ј ઐ৙-ౠ njo )਑৙-ౠ * ৘৙-1 njo )ਗ৙-1 * І ঱)৘* s.t ʇ৘ > ৙ ઐ৙-1 njo )਑৙-1 * І ঱҄)ʇ҄ઐ* ѿ ܕ৙- ઐܖ
56. From Primal to Dual … … and Back 2 3 }}৙ ѿ ʇ৘}}3 , ౠ঱)৘* Ј ৘৙-ౠ njo )ਗ৙-ౠ * ঱҄)ʇ҄ઐ* ѿ ܕ৙- ઐܖ , ౠ 3 }}ઐ}}3 Ј ઐ৙-ౠ njo )਑৙-ౠ * ৘৙-1 njo )ਗ৙-1 * І ঱)৘* s.t ʇ৘ > ৙ ઐ৙-1 njo )਑৙-1 * І ঱҄)ʇ҄ઐ* ѿ ܕ৙- ઐܖ Primal Primal-Limit Dual Dual-Limit
57. From Primal to Dual … … and Back 2 3 }}৙ ѿ ʇ৘}}3 , ౠ঱)৘* Ј ৘৙-ౠ njo )ਗ৙-ౠ * ঱҄)ʇ҄ઐ* ѿ ܕ৙- ઐܖ , ౠ 3 }}ઐ}}3 Ј ઐ৙-ౠ njo )਑৙-ౠ * ৘৙-1 njo )ਗ৙-1 * І ঱)৘* s.t ʇ৘ > ৙ ઐ৙-1 njo )਑৙-1 * І ঱҄)ʇ҄ઐ* ѿ ܕ৙- ઐܖ Primal Primal-Limit Dual Dual-Limit
58. From Primal to Dual … … and Back 2 3 }}৙ ѿ ʇ৘}}3 , ౠ঱)৘* Ј ৘৙-ౠ njo )ਗ৙-ౠ * ঱҄)ʇ҄ઐ* ѿ ܕ৙- ઐܖ , ౠ 3 }}ઐ}}3 Ј ઐ৙-ౠ njo )਑৙-ౠ * ৘৙-1 njo )ਗ৙-1 * І ঱)৘* s.t ʇ৘ > ৙ ઐ৙-1 njo )਑৙-1 * І ঱҄)ʇ҄ઐ* ѿ ܕ৙- ઐܖ Primal Primal-Limit Dual Dual-Limit Dualization
59. From Primal to Dual … … and Back 2 3 }}৙ ѿ ʇ৘}}3 , ౠ঱)৘* Ј ৘৙-ౠ njo )ਗ৙-ౠ * ঱҄)ʇ҄ઐ* ѿ ܕ৙- ઐܖ , ౠ 3 }}ઐ}}3 Ј ઐ৙-ౠ njo )਑৙-ౠ * ৘৙-1 njo )ਗ৙-1 * І ঱)৘* s.t ʇ৘ > ৙ ઐ৙-1 njo )਑৙-1 * І ঱҄)ʇ҄ઐ* ѿ ܕ৙- ઐܖ Primal Primal-Limit Dual Dual-Limit
60. Dual Certificates Proposition • There exists a dual certificate ܧ ৘ is a solution of )ਗ৙-1 * • ʇઐ ѵ ౯঱)৘* ܧ ઐ solution of )਑৙-1 * Definition A dual certificate is a vector ઐ such that ʇ҄ઐ ѵ ౯঱)৘*
61. Tight Dual Certificates (Meet the Relative Interior) Definition relative boundary relative interior
62. An Intermediate Result* Theorem Assume ʇ҄ઐ ѵ sj ౯঱)৘1 * and Lfs ʇ Җ ঻৘1 > |1~/ Choosing ౠ > ৄ}}ৗ}}3, ৄ ? 1, for any minimizer ৘Գ of )ਗ৙-ౠ * }}৘Գ ѿ ৘1 }}3 ӑ প)ৄ- ઐ*}}ৗ}}3 / No model selection ! What is missing ? *Huge research topic in the German/Austrian school (Grassmair, Hatmaier, Scherzer, etc.)
63. How to Build a Certificate ? 2 3 }}৙ ѿ ʇ৘}}3 , ౠ঱)৘* Ј ৘৙-ౠ njo )ਗ৙-ౠ * ʇ҄ઐ* ѿ ܕ৙- ઐܖ , ౠ 3 }}ઐ}}3 Ј ઐ৙-ౠ njo )਑৙-ౠ * ৘৙-1 njo )ਗ৙-1 * І ঱)৘* s.t ʇ৘ > ৙ ઐ৙-1 njo )਑৙-1 * І ঱҄)ʇ҄ઐ* ѿ ܕ৙- ઐܖ one solution several solutions
64. How to Build a Certificate ? 2 3 }}৙ ѿ ʇ৘}}3 , ౠ঱)৘* Ј ৘৙-ౠ njo )ਗ৙-ౠ * ʇ҄ઐ* ѿ ܕ৙- ઐܖ , ౠ 3 }}ઐ}}3 Ј ઐ৙-ౠ njo )਑৙-ౠ * ৘৙-1 njo )ਗ৙-1 * І ঱)৘* s.t ʇ৘ > ৙ ઐ৙-1 njo )਑৙-1 * І ঱҄)ʇ҄ઐ* ѿ ܕ৙- ઐܖ one solution several solutions Convergence ?
65. Minimal-norm Certificate 2 3 }}৙ ѿ ʇ৘}}3 , ౠ঱)৘* Ј ৘৙-ౠ njo )ਗ৙-ౠ * ʇ҄ઐ* ѿ ܕ৙- ઐܖ , ౠ 3 }}ઐ}}3 Ј ઐ৙-ౠ njo )਑৙-ౠ * ৘৙-1 njo )ਗ৙-1 * І ঱)৘* s.t ʇ৘ > ৙ ઐ৙-1 njo )਑৙-1 * І ঱҄)ʇ҄ઐ* ѿ ܕ৙- ઐܖ Proposition with
66. Linearized Precertificate Definition Proposition
67. Main Theorem Theorem Assume ঱ is partly smooth at ৘1 relative to Ϥ. Suppose ʇ҄ઐভ ѵ sj ౯঱)৘1 * and Lfs ʇ Җ ঻৘1 > |1~/ There exists প ? 1 such that if nby)ౠ- }}ৗ}}0ౠ* ӑ প- the unique solution ৘Գ of (ਗ৙-ౠ) satisfies ৘Գ ѵ Ϥ and }}৘Գ ѿ ৘1 }} > শ)}}ৗ}}*/
68. How Tight is This Result ? Theorem Missing part: relative boundary of the subdifferential Wait for an example
69. Implications for First-Order Methods Forward-Backward splitting
70. Implications for First-Order Methods Forward-Backward splitting Theorem
71. Variations • Non-deterministic setting • General loss function • Convergence rates
72. Sparse Spike Deconvolution (a.k.a 1D Stars Recovery) “Natural” prior : sparsity
73. Sparse Spike Deconvolution (a.k.a 1D Stars Recovery) “Natural” prior : sparsity
74. Sparse Spike Deconvolution (a.k.a 1D Stars Recovery) “Natural” prior : sparsity
75. 1D Total Variation Denoising (Staircasing is everywhere)
76. 1D Total Variation Denoising (Staircasing is everywhere)
77. 1D Total Variation Denoising (Staircasing is everywhere) stable jump unstable jump w/ staircasing
78. Compressed Sensing with Nuclear Norm 1000 1500 2000 2500 0 0.5 1 1.5 2 2.5 3 3.5 4 5 10 15 20 0 0.5 1 1.5 Dashed line : Candes’ threshold Fixed rank Variable measurements Variable rank Fixed measurements
79. Perspectives & Conclusion
80. Mandatory Lena Time
81. Mandatory Lena Time Linearized precertificate: not a certificate (in general)
82. Mandatory Lena Time Linearized precertificate: not a certificate (in general)
83. Take-away Message For a large class of recovery problems, under some assumptions, one observes that both “energy” recovery and “localization” recovery hold. • Convex Analysis • Geometrical Structure Solutions build around the dual of the constrained problem Partial smoothness is the key
84. Thanks for your attention Want more ? Review book chapter: V., G. Peyré, J. Fadili, Low Complexity Regularizations, LNCS, 2014 Preprint on model selection/consistency: V., G. Peyré, J. Fadili, Manifold Consistency with Partly Smooth Regularizers Special case for analysis sparsity: V., C. Dossal, G. Peyré, J. Fadili, Robust Sparse Analysis Regularization, TIT, 2013