Tweet
Tweet

Slide 1

Slide 1 text

Régularisations de faibles complexités : Un résultat de "localisation" Samuel Vaiter 13 November 2014

Slide 2

Slide 2 text

Ground Truth Observations Recovery

Slide 3

Slide 3 text

Perfect

Slide 4

Slide 4 text

Useless

Slide 5

Slide 5 text

Interesting

Slide 6

Slide 6 text

What is Robustness ? “Energy” Robustness “Localization” Robustness

Slide 7

Slide 7 text

Part I Inverse Problems Regularizations Models

Slide 8

Slide 8 text

Main Theorem: First Version For a large class of recovery problems, under some assumptions to be precised, one observes that both “energy” recovery and “localization” recovery hold. Theorem

Slide 9

Slide 9 text

Linear Inverse Problem (in Finite Dimension) ৙ > ʇ৘1 , ৗ denoising inpainting deblurring

Slide 10

Slide 10 text

Variational Regularization ৘Գ ѵ bshnjo ৘ѵϓৎ ভ)৘- ৙* , ౠ঱)৘* Data fidelity Prior vs

Slide 11

Slide 11 text

Some Remarks Loss considered: square prediction-error

Slide 12

Slide 12 text

Some Remarks Loss considered: square prediction-error No noise: use of the constrained form

Slide 13

Slide 13 text

Some Remarks Loss considered: square prediction-error No noise: use of the constrained form

Slide 14

Slide 14 text

Some Remarks Loss considered: square prediction-error No noise: use of the constrained form

Slide 15

Slide 15 text

Choice of a Regularization Wavelet basis sparsity Total Variation Fused Lasso Nuclear norm OSCAR Spread representation Group Sparsity Trace Lasso Sobolev TGV Weighted sparsity Elastic net Ridge Analysis sparsity

Slide 16

Slide 16 text

Models Models (Convex) Functions vector of fixed support matrix of fixed rank vector with same saturation pattern vector with same jump set

Slide 17

Slide 17 text

Sparsity with a Two Pixels Image Ϥ৘

Slide 18

Slide 18 text

Sparsity with a Two Pixels Image Ϥ৘

Slide 19

Slide 19 text

Main Theorem: 2nd Version Theorem For a large class of variational regularizations, under some assumptions to be precised, one observes that both : Ј ৘Գ ѵ Ϥ৘1 Ј }}৘Գ ѿ ৘1 }} is small

Slide 20

Slide 20 text

A Little Bit of (Recent) History

Slide 21

Slide 21 text

Technical Interlude

Slide 22

Slide 22 text

Subdifferential ে)৘* ৘ Ȣ ৘ ে) Ȣ ৘* ে)৘* Ӓ ে) Ȣ ৘* , ܕѴে) Ȣ ৘*- ৘ ѿ Ȣ ৘ܖ

Slide 23

Slide 23 text

Subdifferential ে)৘* ৘ Ȣ ৘ ে) Ȣ ৘* ে)৘* Ӓ ে) Ȣ ৘* , ܕৈ- ৘ ѿ Ȣ ৘ܖ

Slide 24

Slide 24 text

Subdifferential ে)৘* ৘ Ȣ ৘ ে) Ȣ ৘* |ৈ ң ে)৘* Ӓ ে) Ȣ ৘* , ܕৈ- ৘ ѿ Ȣ ৘ܖ~ ౯ে) Ȣ ৘* >

Slide 25

Slide 25 text

First Order Conditions Ȣ ৘ ѵ bshnjo ে)৘* ܧ Ѵে) Ȣ ৘* > 1 Euler equation for convex + smooth function

Slide 26

Slide 26 text

First Order Conditions Ȣ ৘ ѵ bshnjo ে)৘* ܧ Ѵে) Ȣ ৘* > 1 Euler equation for convex + smooth function Ȣ ৘ ѵ bshnjo ে)৘* ܧ ౯ে) Ȣ ৘* Ѹ 1 Euler equation for convex + non-smooth function

Slide 27

Slide 27 text

Smooth Manifold

Slide 28

Slide 28 text

Smooth Manifold Tangent space

Slide 29

Slide 29 text

Smooth Manifold Tangent space Ok, maybe listen Alain instead

Slide 30

Slide 30 text

Part II Partial Smoothness

Slide 31

Slide 31 text

What’s Important in Sparsity ? Ϥ৘ |৘ ң ঱)৘* > ঱) Ȣ ৘*~ Ȣ ৘

Slide 32

Slide 32 text

Ȣ ৘ , e৘ What’s Important in Sparsity ? Ϥ৘ ঱) Ȣ ৘ , e৘*

Slide 33

Slide 33 text

Ȣ ৘ , e৘ What’s Important in Sparsity ? Ϥ৘ ঱) Ȣ ৘ , e৘*

Slide 34

Slide 34 text

What’s Important in Sparsity ? Ϥ৘ Ȣ ৘ 1: }} Բ }}2 restricted to Ϥ৘ is locally smooth

Slide 35

Slide 35 text

What’s Important in Sparsity ? Ϥ৘ ঱) Ȣ ৘ , e৘*

Slide 36

Slide 36 text

What’s Important in Sparsity ? Ϥ৘ ঱) Ȣ ৘ , e৘*

Slide 37

Slide 37 text

What’s Important in Sparsity ? Ϥ৘ ঱) Ȣ ৘ , e৘*

Slide 38

Slide 38 text

What’s Important in Sparsity ? Ϥ৘ Ȣ ৘ 1: }} Բ }}2 restricted to Ϥ৘ is locally smooth 2: }} Բ }}2 is sharp (non-differentiable) in the direction of ϤԒ ৘

Slide 39

Slide 39 text

What’s Important in Sparsity ? Ϥ৘ Ȣ ৘ 1: }} Բ }}2 restricted to Ϥ৘ is locally smooth 2: }} Բ }}2 is sharp (non-differentiable) in the direction of ϤԒ ৘ 3: ౯}} Բ }}2 restricted to Ϥ৘ is locally continuous (constant here)

Slide 40

Slide 40 text

Partial Smoothness Definition ঱ is partly smooth at ৘ relative to a C3 -manifold Ϥ if • Smoothness. ঱ restricted to Ϥ is C3 around ৘ • Sharpness. ѭυ ѵ )ਚϤ ৘*Ԓ , ৔ М ঱)৘ , ৔υ* is non-smooth at ৔ > 1. • Continuity. ౯঱ on Ϥ is continuous around ৘. Ϥ nby)1- }}৘}} ѿ 2*

Slide 41

Slide 41 text

Main Theorem: 3rd Version Theorem Ј ৘Գ ѵ Ϥ৘1 Ј }}৘Գ ѿ ৘1 }} is small When ঱ is partly smooth at ৘1 relatively to Ϥ৘1 , under some assumptions to be precised, one observes that:

Slide 42

Slide 42 text

Examples and Algebraic Stability ৘ М }}৘}}2 - }}৘}}3 - }}৘}}ҋ - }}৘}}2-3 - nby ৉ )৘৉ *, - any smooth function Ϳ

Slide 43

Slide 43 text

Examples and Algebraic Stability ৘ М }}৘}}2 - }}৘}}3 - }}৘}}ҋ - }}৘}}2-3 - nby ৉ )৘৉ *, - any smooth function Ϳ ঱ and ম partly smooth ܦ ঱ , ম partly smooth }} Բ }}2 , ౰}} Բ }}3 3 partly smooth

Slide 44

Slide 44 text

Examples and Algebraic Stability ৘ М }}৘}}2 - }}৘}}3 - }}৘}}ҋ - }}৘}}2-3 - nby ৉ )৘৉ *, - any smooth function Ϳ ঱ and ম partly smooth ܦ ঱ , ম partly smooth }} Բ }}2 , ౰}} Բ }}3 3 partly smooth ঱ partly smooth and ঳ linear operator ܦ ঱ ҅ ঳ partly smooth }}Ѵ Բ }}2-3 (isotropic TV) partly smooth

Slide 45

Slide 45 text

Examples and Algebraic Stability ৘ М }}৘}}2 - }}৘}}3 - }}৘}}ҋ - }}৘}}2-3 - nby ৉ )৘৉ *, - any smooth function Ϳ ঱ and ম partly smooth ܦ ঱ , ম partly smooth }} Բ }}2 , ౰}} Բ }}3 3 partly smooth ঱ partly smooth and ঳ linear operator ܦ ঱ ҅ ঳ partly smooth }}Ѵ Բ }}2-3 (isotropic TV) partly smooth ঱ partly smooth ܦ ঱ ҅ ౨ partly smooth (spectral lift) }} Բ }}҄ (nuclear/trace norm) partly smooth

Slide 46

Slide 46 text

Part III Certificates Restricted Injectivity and Main Result

Slide 47

Slide 47 text

Assumptions ?

Slide 48

Slide 48 text

Main Ideas: 1) The Lagrangian problem is nothing more than the constrained problem up to "something" that we are going to control (by duality)

Slide 49

Slide 49 text

Main Ideas: 2 ) To r e c o v e r a g o o d estimation of the ground truth with noise, it should be identifiable without noise

Slide 50

Slide 50 text

Restricted Injectivity

Slide 51

Slide 51 text

Restricted Injectivity

Slide 52

Slide 52 text

Restricted Injectivity

Slide 53

Slide 53 text

Restricted Injectivity

Slide 54

Slide 54 text

Restricted Injectivity

Slide 55

Slide 55 text

From Primal to Dual … … and Back 2 3 }}৙ ѿ ʇ৘}}3 , ౠ঱)৘* Ј ৘৙-ౠ njo )ਗ৙-ౠ * ঱҄)ʇ҄ઐ* ѿ ܕ৙- ઐܖ , ౠ 3 }}ઐ}}3 Ј ઐ৙-ౠ njo )਑৙-ౠ * ৘৙-1 njo )ਗ৙-1 * І ঱)৘* s.t ʇ৘ > ৙ ઐ৙-1 njo )਑৙-1 * І ঱҄)ʇ҄ઐ* ѿ ܕ৙- ઐܖ

Slide 56

Slide 56 text

From Primal to Dual … … and Back 2 3 }}৙ ѿ ʇ৘}}3 , ౠ঱)৘* Ј ৘৙-ౠ njo )ਗ৙-ౠ * ঱҄)ʇ҄ઐ* ѿ ܕ৙- ઐܖ , ౠ 3 }}ઐ}}3 Ј ઐ৙-ౠ njo )਑৙-ౠ * ৘৙-1 njo )ਗ৙-1 * І ঱)৘* s.t ʇ৘ > ৙ ઐ৙-1 njo )਑৙-1 * І ঱҄)ʇ҄ઐ* ѿ ܕ৙- ઐܖ Primal Primal-Limit Dual Dual-Limit

Slide 57

Slide 57 text

From Primal to Dual … … and Back 2 3 }}৙ ѿ ʇ৘}}3 , ౠ঱)৘* Ј ৘৙-ౠ njo )ਗ৙-ౠ * ঱҄)ʇ҄ઐ* ѿ ܕ৙- ઐܖ , ౠ 3 }}ઐ}}3 Ј ઐ৙-ౠ njo )਑৙-ౠ * ৘৙-1 njo )ਗ৙-1 * І ঱)৘* s.t ʇ৘ > ৙ ઐ৙-1 njo )਑৙-1 * І ঱҄)ʇ҄ઐ* ѿ ܕ৙- ઐܖ Primal Primal-Limit Dual Dual-Limit

Slide 58

Slide 58 text

From Primal to Dual … … and Back 2 3 }}৙ ѿ ʇ৘}}3 , ౠ঱)৘* Ј ৘৙-ౠ njo )ਗ৙-ౠ * ঱҄)ʇ҄ઐ* ѿ ܕ৙- ઐܖ , ౠ 3 }}ઐ}}3 Ј ઐ৙-ౠ njo )਑৙-ౠ * ৘৙-1 njo )ਗ৙-1 * І ঱)৘* s.t ʇ৘ > ৙ ઐ৙-1 njo )਑৙-1 * І ঱҄)ʇ҄ઐ* ѿ ܕ৙- ઐܖ Primal Primal-Limit Dual Dual-Limit Dualization

Slide 59

Slide 59 text

From Primal to Dual … … and Back 2 3 }}৙ ѿ ʇ৘}}3 , ౠ঱)৘* Ј ৘৙-ౠ njo )ਗ৙-ౠ * ঱҄)ʇ҄ઐ* ѿ ܕ৙- ઐܖ , ౠ 3 }}ઐ}}3 Ј ઐ৙-ౠ njo )਑৙-ౠ * ৘৙-1 njo )ਗ৙-1 * І ঱)৘* s.t ʇ৘ > ৙ ઐ৙-1 njo )਑৙-1 * І ঱҄)ʇ҄ઐ* ѿ ܕ৙- ઐܖ Primal Primal-Limit Dual Dual-Limit

Slide 60

Slide 60 text

Dual Certificates Proposition • There exists a dual certificate ܧ ৘ is a solution of )ਗ৙-1 * • ʇઐ ѵ ౯঱)৘* ܧ ઐ solution of )਑৙-1 * Definition A dual certificate is a vector ઐ such that ʇ҄ઐ ѵ ౯঱)৘*

Slide 61

Slide 61 text

Tight Dual Certificates (Meet the Relative Interior) Definition relative boundary relative interior

Slide 62

Slide 62 text

An Intermediate Result* Theorem Assume ʇ҄ઐ ѵ sj ౯঱)৘1 * and Lfs ʇ Җ ঻৘1 > |1~/ Choosing ౠ > ৄ}}ৗ}}3, ৄ ? 1, for any minimizer ৘Գ of )ਗ৙-ౠ * }}৘Գ ѿ ৘1 }}3 ӑ প)ৄ- ઐ*}}ৗ}}3 / No model selection ! What is missing ? *Huge research topic in the German/Austrian school (Grassmair, Hatmaier, Scherzer, etc.)

Slide 63

Slide 63 text

How to Build a Certificate ? 2 3 }}৙ ѿ ʇ৘}}3 , ౠ঱)৘* Ј ৘৙-ౠ njo )ਗ৙-ౠ * ʇ҄ઐ* ѿ ܕ৙- ઐܖ , ౠ 3 }}ઐ}}3 Ј ઐ৙-ౠ njo )਑৙-ౠ * ৘৙-1 njo )ਗ৙-1 * І ঱)৘* s.t ʇ৘ > ৙ ઐ৙-1 njo )਑৙-1 * І ঱҄)ʇ҄ઐ* ѿ ܕ৙- ઐܖ one solution several solutions

Slide 64

Slide 64 text

How to Build a Certificate ? 2 3 }}৙ ѿ ʇ৘}}3 , ౠ঱)৘* Ј ৘৙-ౠ njo )ਗ৙-ౠ * ʇ҄ઐ* ѿ ܕ৙- ઐܖ , ౠ 3 }}ઐ}}3 Ј ઐ৙-ౠ njo )਑৙-ౠ * ৘৙-1 njo )ਗ৙-1 * І ঱)৘* s.t ʇ৘ > ৙ ઐ৙-1 njo )਑৙-1 * І ঱҄)ʇ҄ઐ* ѿ ܕ৙- ઐܖ one solution several solutions Convergence ?

Slide 65

Slide 65 text

Minimal-norm Certificate 2 3 }}৙ ѿ ʇ৘}}3 , ౠ঱)৘* Ј ৘৙-ౠ njo )ਗ৙-ౠ * ʇ҄ઐ* ѿ ܕ৙- ઐܖ , ౠ 3 }}ઐ}}3 Ј ઐ৙-ౠ njo )਑৙-ౠ * ৘৙-1 njo )ਗ৙-1 * І ঱)৘* s.t ʇ৘ > ৙ ઐ৙-1 njo )਑৙-1 * І ঱҄)ʇ҄ઐ* ѿ ܕ৙- ઐܖ Proposition with

Slide 66

Slide 66 text

Linearized Precertificate Definition Proposition

Slide 67

Slide 67 text

Main Theorem Theorem Assume ঱ is partly smooth at ৘1 relative to Ϥ. Suppose ʇ҄ઐভ ѵ sj ౯঱)৘1 * and Lfs ʇ Җ ঻৘1 > |1~/ There exists প ? 1 such that if nby)ౠ- }}ৗ}}0ౠ* ӑ প- the unique solution ৘Գ of (ਗ৙-ౠ) satisfies ৘Գ ѵ Ϥ and }}৘Գ ѿ ৘1 }} > শ)}}ৗ}}*/

Slide 68

Slide 68 text

How Tight is This Result ? Theorem Missing part: relative boundary of the subdifferential Wait for an example

Slide 69

Slide 69 text

Implications for First-Order Methods Forward-Backward splitting

Slide 70

Slide 70 text

Implications for First-Order Methods Forward-Backward splitting Theorem

Slide 71

Slide 71 text

Variations • Non-deterministic setting • General loss function • Convergence rates

Slide 72

Slide 72 text

Sparse Spike Deconvolution (a.k.a 1D Stars Recovery) “Natural” prior : sparsity

Slide 73

Slide 73 text

Sparse Spike Deconvolution (a.k.a 1D Stars Recovery) “Natural” prior : sparsity

Slide 74

Slide 74 text

Sparse Spike Deconvolution (a.k.a 1D Stars Recovery) “Natural” prior : sparsity

Slide 75

Slide 75 text

1D Total Variation Denoising (Staircasing is everywhere)

Slide 76

Slide 76 text

1D Total Variation Denoising (Staircasing is everywhere)

Slide 77

Slide 77 text

1D Total Variation Denoising (Staircasing is everywhere) stable jump unstable jump w/ staircasing

Slide 78

Slide 78 text

Compressed Sensing with Nuclear Norm 1000 1500 2000 2500 0 0.5 1 1.5 2 2.5 3 3.5 4 5 10 15 20 0 0.5 1 1.5 Dashed line : Candes’ threshold Fixed rank Variable measurements Variable rank Fixed measurements

Slide 79

Slide 79 text

Perspectives & Conclusion

Slide 80

Slide 80 text

Mandatory Lena Time

Slide 81

Slide 81 text

Mandatory Lena Time Linearized precertificate: not a certificate (in general)

Slide 82

Slide 82 text

Mandatory Lena Time Linearized precertificate: not a certificate (in general)

Slide 83

Slide 83 text

Take-away Message For a large class of recovery problems, under some assumptions, one observes that both “energy” recovery and “localization” recovery hold. • Convex Analysis • Geometrical Structure Solutions build around the dual of the constrained problem Partial smoothness is the key

Slide 84

Slide 84 text

Thanks for your attention Want more ? Review book chapter: V., G. Peyré, J. Fadili, Low Complexity Regularizations, LNCS, 2014 Preprint on model selection/consistency: V., G. Peyré, J. Fadili, Manifold Consistency with Partly Smooth Regularizers Special case for analysis sparsity: V., C. Dossal, G. Peyré, J. Fadili, Robust Sparse Analysis Regularization, TIT, 2013