Sparsity and Compressed Sensing

E34ded36efe4b7abb12510d4e525fee8?s=47 Gabriel Peyré
September 13, 2012

Sparsity and Compressed Sensing

Talk at Cachan

E34ded36efe4b7abb12510d4e525fee8?s=128

Gabriel Peyré

September 13, 2012
Tweet

Transcript

  1. Sparsity and Compressed Sensing Gabriel Peyré www.numerical-tours.com

  2. Signals, Images and More

  3. Signals, Images and More

  4. Signals, Images and More

  5. Signals, Images and More

  6. Signals, Images and More

  7. Overview • Approximation in an Ortho-Basis • Compression and Denoising

    • Compressed Sensing
  8. Orthogonal basis { m }m of L2([0, 1]d) Continuous signal/image

    f L2([0, 1]d). Orthogonal Decompositions
  9. Orthogonal basis { m }m of L2([0, 1]d) f =

    m f, m m ||f|| = |f(x)|2dx = m | f, m ⇥|2 Continuous signal/image f L2([0, 1]d). Orthogonal Decompositions
  10. Orthogonal basis { m }m of L2([0, 1]d) f =

    m f, m m ||f|| = |f(x)|2dx = m | f, m ⇥|2 Continuous signal/image f L2([0, 1]d). Orthogonal Decompositions m
  11. 1-D Wavelet Basis Wavelets: j,n (x) = 1 2j/2 x

    2jn 2j Position n, scale 2j, m = (n, j).
  12. 1-D Wavelet Basis Wavelets: j,n (x) = 1 2j/2 x

    2jn 2j Position n, scale 2j, m = (n, j).
  13. m1,m2 Basis { m1,m2 (x1, x2 )}m1,m2 of L2([0, 1]2)

    m1,m2 (x1, x2 ) = m1 (x1 ) m2 (x2 ) tensor product 2-D Fourier Basis Basis { m (x)}m of L2([0, 1]) m1 m2
  14. m1,m2 Basis { m1,m2 (x1, x2 )}m1,m2 of L2([0, 1]2)

    m1,m2 (x1, x2 ) = m1 (x1 ) m2 (x2 ) tensor product f(x) f, m1,m2 Fourier transform 2-D Fourier Basis Basis { m (x)}m of L2([0, 1]) m1 m2 x m
  15. 3 elementary wavelets { H, V , D}. Orthogonal basis

    of L2([0, 1]2): k j,n (x) = 2 j (2 jx n) k=H,V,D j<0,2j n [0,1]2 2-D Wavelet Basis V (x) H(x) D(x)
  16. 3 elementary wavelets { H, V , D}. Orthogonal basis

    of L2([0, 1]2): k j,n (x) = 2 j (2 jx n) k=H,V,D j<0,2j n [0,1]2 2-D Wavelet Basis V (x) H(x) D(x)
  17. wavelet f, k j,n Example of Wavelet Decomposition f(x) transform

    x (j, n, k)
  18. Discrete Computations Discrete orthogonal basis { m } of CN

    . f = m f, m m
  19. Fast Fourier Transform (FFT), O(N log(N)) operations. Discrete Computations Discrete

    orthogonal basis { m } of CN . m [n] = 1 N e2i N nm f = m f, m m
  20. Fast Fourier Transform (FFT), O(N log(N)) operations. Fast Wavelet Transform,

    O(N) operations. Discrete Wavelet basis: no closed-form expression. Discrete Computations Discrete orthogonal basis { m } of CN . m [n] = 1 N e2i N nm f = m f, m m
  21. Sparse Approximation in a Basis

  22. Sparse Approximation in a Basis

  23. Sparse Approximation in a Basis

  24. Best basis Fastest error decay ||f fM ||2 log(||f fM

    ||) log(M) Efficiency of Transforms Fourier DCT Local DCT Wavelets
  25. Overview • Approximation in an Ortho-Basis • Compression and Denoising

    • Compressed Sensing
  26. JPEG-2000 vs. JPEG, 0.2bit/pixel

  27. Compression by Transform-coding Image f Zoom on f f forward

    a[m] = ⇥f, m ⇤ R transform
  28. Compression by Transform-coding Image f Zoom on f f forward

    a[m] = ⇥f, m ⇤ R transform Quantization: q[m] = sign(a[m]) |a[m]| T ⇥ Z ˜ a[m] T T 2T 2T a[m] Quantized q[m] bin T q[m] Z
  29. Compression by Transform-coding Image f Zoom on f f forward

    a[m] = ⇥f, m ⇤ R coding transform Entropic coding: use statistical redundancy (many 0’s). Quantization: q[m] = sign(a[m]) |a[m]| T ⇥ Z ˜ a[m] T T 2T 2T a[m] Quantized q[m] bin T q[m] Z
  30. Compression by Transform-coding Image f Zoom on f f forward

    a[m] = ⇥f, m ⇤ R coding decoding q[m] Z transform Entropic coding: use statistical redundancy (many 0’s). Quantization: q[m] = sign(a[m]) |a[m]| T ⇥ Z ˜ a[m] T T 2T 2T a[m] Quantized q[m] bin T q[m] Z
  31. Compression by Transform-coding Image f Zoom on f f forward

    Dequantization: ˜ a[m] = sign(q[m]) |q[m] + 1 2 ⇥ T a[m] = ⇥f, m ⇤ R coding decoding q[m] Z ˜ a[m] dequantization transform Entropic coding: use statistical redundancy (many 0’s). Quantization: q[m] = sign(a[m]) |a[m]| T ⇥ Z ˜ a[m] T T 2T 2T a[m] Quantized q[m] bin T q[m] Z
  32. Compression by Transform-coding Image f Zoom on f f ,

    R =0.2 bit/pixel f forward Dequantization: ˜ a[m] = sign(q[m]) |q[m] + 1 2 ⇥ T a[m] = ⇥f, m ⇤ R coding decoding q[m] Z ˜ a[m] dequantization transform backward fR = m IT ˜ a[m] m transform Entropic coding: use statistical redundancy (many 0’s). Quantization: q[m] = sign(a[m]) |a[m]| T ⇥ Z ˜ a[m] T T 2T 2T a[m] Quantized q[m] bin T q[m] Z
  33. Compression by Transform-coding Image f Zoom on f f ,

    R =0.2 bit/pixel f forward Dequantization: ˜ a[m] = sign(q[m]) |q[m] + 1 2 ⇥ T a[m] = ⇥f, m ⇤ R coding decoding q[m] Z ˜ a[m] dequantization transform backward fR = m IT ˜ a[m] m transform Entropic coding: use statistical redundancy (many 0’s). Quantization: q[m] = sign(a[m]) |a[m]| T ⇥ Z ˜ a[m] T T 2T 2T a[m] Quantized q[m] bin T q[m] Z “Theorem:” ||f fM ||2 = O(M ) =⇥ ||f fR ||2 = O(log (R)R )
  34. Noise in Images

  35. Denoising

  36. Denoising thresh. f = N 1 m=0 f, m ⇥

    m ˜ f = | f, m ⇥|>T f, m ⇥ m
  37. Denoising thresh. f = N 1 m=0 f, m ⇥

    m ˜ f = | f, m ⇥|>T f, m ⇥ m In practice: T 3 for T = 2 log(N) Theorem: if ||f0 f0,M ||2 = O(M ), E(|| ˜ f f0 ||2) = O( 2 +1 )
  38. Overview • Approximation in an Ortho-Basis • Compression and Denoising

    • Compressed Sensing
  39. f[n] f0 (n/N) Sampling: ˜ f L2([0, 1]d) f RN

    Idealization: acquisition device Discretization
  40. Data aquisition: Sensors Pointwise Sampling and Smoothness ˜ f L2

    f RN f[i] = ˜ f(i/N)
  41. Data aquisition: Sensors ˜ f(t) = i f[i]h(Nt i) Shannon

    interpolation: if Supp( ˆ ˜ f) [ N , N ] h(t) = sin( t) t Pointwise Sampling and Smoothness ˜ f L2 f RN f[i] = ˜ f(i/N)
  42. Data aquisition: Sensors ˜ f(t) = i f[i]h(Nt i) Natural

    images are not smooth. Shannon interpolation: if Supp( ˆ ˜ f) [ N , N ] h(t) = sin( t) t Pointwise Sampling and Smoothness ˜ f L2 f RN f[i] = ˜ f(i/N)
  43. Data aquisition: Sensors ˜ f(t) = i f[i]h(Nt i) Natural

    images are not smooth. But can be compressed e ciently. Shannon interpolation: if Supp( ˆ ˜ f) [ N , N ] 0,1,0,. . . h(t) = sin( t) t Sample and compress simultaneously? Pointwise Sampling and Smoothness ˜ f L2 f RN f[i] = ˜ f(i/N) JPEG-2k
  44. Single Pixel Camera (Rice) y[i] = f0, i ⇥

  45. Single Pixel Camera (Rice) y[i] = f0, i ⇥ f0,

    N = 2562 f , P/N = 0.16 f , P/N = 0.02
  46. Physical hardware resolution limit: target resolution f RN . ˜

    f L2 f RN y RP micro mirrors array resolution CS hardware K CS Hardware Model CS is about designing hardware: input signals ˜ f L2(R2).
  47. Physical hardware resolution limit: target resolution f RN . ˜

    f L2 f RN y RP micro mirrors array resolution CS hardware , ... K CS Hardware Model CS is about designing hardware: input signals ˜ f L2(R2). , ,
  48. Physical hardware resolution limit: target resolution f RN . ˜

    f L2 f RN y RP micro mirrors array resolution CS hardware , ... f Operator K K CS Hardware Model CS is about designing hardware: input signals ˜ f L2(R2). , ,
  49. Need to solve y = Kf. More unknown than equations.

    dim(ker(K)) = N P is huge. Inversion and Sparsity f Operator K
  50. Need to solve y = Kf. More unknown than equations.

    dim(ker(K)) = N P is huge. Prior information: f is sparse in a basis { m }m . J (f) = Card {m \ | f, m | > } is small. Inversion and Sparsity f Operator K f, m f
  51. 0 reconstruction: Minimize subject to Kf = y y =

    f f, 1 f, 2 CS Reconstruction J0 (f) = Card {m \ f, m = 0}
  52. 0 reconstruction: Minimize subject to Kf = y NP-hard to

    solve. y = f f, 1 f, 2 CS Reconstruction J0 (f) = Card {m \ f, m = 0}
  53. 0 reconstruction: Minimize subject to Kf = y 1 reconstruction:

    m | f, m | Polynomial-time algorithms. NP-hard to solve. y = f f, 1 f, 2 CS Reconstruction J0 (f) = Card {m \ f, m = 0} Minimize subject to Kf = y
  54. Theorem: [Candes, Romberg, Tao, Donoho, 2004] If f is k-sparse,

    i.e. J0 (f) k If P C log(N/k)k then 1-CS reconstruction is exact. Theoretical Performance Guaranties
  55. Theorem: [Candes, Romberg, Tao, Donoho, 2004] If f is k-sparse,

    i.e. J0 (f) k If P C log(N/k)k then 1-CS reconstruction is exact. Extensions to: noisy observation y = Kf + approximate sparsity f = fk sparse + Theoretical Performance Guaranties
  56. Theorem: [Candes, Romberg, Tao, Donoho, 2004] If f is k-sparse,

    i.e. J0 (f) k If P C log(N/k)k then 1-CS reconstruction is exact. Extensions to: noisy observation y = Kf + approximate sparsity f = fk sparse + Research problem: optimal value of C ? for N/k = 4, C log(N/k) 5. “CS is 5 less e cient than JPEG-2k” Theoretical Performance Guaranties
  57. Conclusion

  58. Conclusion

  59. random acquisition. optimization for reconstruction. #measures sparsity Conclusion • Compressed

    sensing.