630

# Signal Processing Course: Non-linear Approximation and Coding

January 01, 2012

## Transcript

2. ### Overview • Approximation and Compression • Decay of Approximation Error

• Fourier for Smooth Functions • Wavelet for Piecewise Smooth Functions • Curvelets and Finite Elements for Cartoons

8. ### (usually polynomial) Approximation Speed Approximation error decay: log 10 (||f

fM ||) Log/Log plot: approx. a ne curve log 10 (M/N)
9. ### Efficiency of Transforms Fourier DCT Local DCT Wavelets log 10

(||f fM ||) log 10 (M/N)
10. ### Overview • Approximation and Compression • Decay of Approximation Error

• Fourier for Smooth Functions • Wavelet for Piecewise Smooth Functions • Curvelets and Finite Elements for Cartoons
11. ### f forward Compression by Transform-coding a[m] = ⇥f, m ⇤

R Image f Zoom on f transform
12. ### f forward Compression by Transform-coding a[m] = ⇥f, m ⇤

R Quantization: q[m] = sign(a[m]) |a[m]| T ⇥ Z Image f Zoom on f transform ˜ a[m] T T 2T 2T a[m Quantized q[m] bin T q[m] Z
13. ### f forward coding Compression by Transform-coding a[m] = ⇥f, m

⇤ R Quantization: q[m] = sign(a[m]) |a[m]| T ⇥ Z Image f Zoom on f transform Entropic coding: use statistical redundancy (many 0’s). ˜ a[m] T T 2T 2T a[m Quantized q[m] bin T q[m] Z
14. ### f forward coding Compression by Transform-coding a[m] = ⇥f, m

⇤ R Quantization: q[m] = sign(a[m]) |a[m]| T ⇥ Z Image f Zoom on f decoding q[m] Z ˜ a[m] dequantization transform Entropic coding: use statistical redundancy (many 0’s). ˜ a[m] T T 2T 2T a[m Quantized q[m] bin T q[m] Z Dequantization:
15. ### f forward coding Compression by Transform-coding a[m] = ⇥f, m

⇤ R Quantization: q[m] = sign(a[m]) |a[m]| T ⇥ Z Image f Zoom on f fR , R =0.2 bit/pixel decoding q[m] Z ˜ a[m] dequantization transform backward fR = m IT ˜ a[m] m transform Entropic coding: use statistical redundancy (many 0’s). ˜ a[m] T T 2T 2T a[m Quantized q[m] bin T q[m] Z Dequantization:

18. ### Non-linear Approximation and Compression ˜ a[m] T T 2T 2T

a[m] Quantization: q[m] = sign(a[m]) |a[m]| T ⇥ Z =⇥ |a[m] ˜ a[m]| T 2 Dequantization: ˜ a[m] = sign(q[m]) |q[m]| + 1 2 ⇥ T
19. ### Non-linear Approximation and Compression ||f fM ||2 # = M

Theorem: ||f fR ||2 ||f fM ||2 + MT2/4 where M = # {m \ ˜ a[m] = 0}. ||f fR ||2 = ⇤ m (a[m] ˜ a[m])2 ⇤ |a[m]|<T |a[m]|2 + ⇤ |a[m]| T T 2 ⇥2 ˜ a[m] T T 2T 2T a[m] Quantization: q[m] = sign(a[m]) |a[m]| T ⇥ Z =⇥ |a[m] ˜ a[m]| T 2 Dequantization: ˜ a[m] = sign(q[m]) |q[m]| + 1 2 ⇥ T
20. ### A Naive Support Coding Approach Coding: relate # bits R

to # coe cients M. (H 1 ) Ordered coe cients | f, m ⇥| decays like m +1 2 . =⇤ ||f fM ||2 ⇥ M .
21. ### A Naive Support Coding Approach Coding: relate # bits R

to # coe cients M. (H 1 ) Ordered coe cients | f, m ⇥| decays like m +1 2 . f RN sampled from f0 , error: (H 2 ) To ensure ||f f0 ||2 ⇥ ||f fM ||2: M ⇥ N⇥/ . ||f f0 ||2 ⇥ N =⇤ ||f fM ||2 ⇥ M .
22. ### A Naive Support Coding Approach Simple coding strategy: R =

Rval + Rind = Rind log 2 N M ⇥ = O(M log 2 (N/M)) = O(M log 2 (M)). Coding: relate # bits R to # coe cients M. (H 1 ) Ordered coe cients | f, m ⇥| decays like m +1 2 . f RN sampled from f0 , error: (H 2 ) To ensure ||f f0 ||2 ⇥ ||f fM ||2: M ⇥ N⇥/ . ||f f0 ||2 ⇥ N =⇤ ||f fM ||2 ⇥ M .
23. ### A Naive Support Coding Approach Simple coding strategy: R =

Rval + Rind = Rind log 2 N M ⇥ = O(M log 2 (N/M)) = O(M log 2 (M)). = Rval = O(M| log 2 (T)|) = O(M log 2 (M)) Coding: relate # bits R to # coe cients M. (H 1 ) Ordered coe cients | f, m ⇥| decays like m +1 2 . f RN sampled from f0 , error: (H 2 ) To ensure ||f f0 ||2 ⇥ ||f fM ||2: M ⇥ N⇥/ . ||f f0 ||2 ⇥ N =⇤ ||f fM ||2 ⇥ M .
24. ### A Naive Support Coding Approach Simple coding strategy: R =

Rval + Rind = Rind log 2 N M ⇥ = O(M log 2 (N/M)) = O(M log 2 (M)). Theorem: Under hypotheses (H 1 ) and (H 2 ), ||f fR ||2 = O(R log (R)). = Rval = O(M| log 2 (T)|) = O(M log 2 (M)) Coding: relate # bits R to # coe cients M. (H 1 ) Ordered coe cients | f, m ⇥| decays like m +1 2 . f RN sampled from f0 , error: (H 2 ) To ensure ||f f0 ||2 ⇥ ||f fM ||2: M ⇥ N⇥/ . ||f f0 ||2 ⇥ N =⇤ ||f fM ||2 ⇥ M .

35. ### Overview • Approximation and Compression • Decay of Approximation Error

• Fourier for Smooth Functions • Wavelet for Piecewise Smooth Functions • Curvelets and Finite Elements for Cartoons

39. ### Singularities and Fourier 0 0.2 0.4 0.6 0.8 1 0

0.2 0.4 0.6 0.8 1 0 0.2 0.4 0.6 0.8 1 0 0.2 0.4 0.6 0.8 1

42. ### Overview • Approximation and Compression • Decay of Approximation Error

• Fourier for Smooth Functions • Wavelet for Piecewise Smooth Functions • Curvelets and Finite Elements for Cartoons
43. ### Vanishing moments: p = 3 p = 4 Magnitude of

Wavelet Coefficients f(x) −1 0 1 2 −2 −1 0 1 2 −2 −1 0 1 2 −1 0 1 2 −2 0 2 4 −1 −0.5 0 0.5 1 1.5 k < p, (x)xkdx = 0 p = 2
44. ### Vanishing moments: p = 3 p = 4 Magnitude of

Wavelet Coefficients f(x) −1 0 1 2 −2 −1 0 1 2 −2 −1 0 1 2 −1 0 1 2 −2 0 2 4 −1 −0.5 0 0.5 1 1.5 k < p, (x)xkdx = 0 p = 2 | f, j,n ⇥| Cf || ||1 2j( +d/2) t = x 2jn 2j ⇥f, j,n ⇤ = 1 2j d 2 ⇤ f(x) x 2jn 2j ⇥ dx = 2j d 2 ⇤ R(2jt) (t)dt If f is C on supp(⇥j,n ), p : f(x) = P(x 2jn) + R(x 2jn) = P(2jt) + R(2jt)
45. ### Vanishing moments: p = 3 p = 4 Magnitude of

Wavelet Coefficients f(x) −1 0 1 2 −2 −1 0 1 2 −2 −1 0 1 2 −1 0 1 2 −2 0 2 4 −1 −0.5 0 0.5 1 1.5 k < p, (x)xkdx = 0 | f, j,n ⇥| ||f|| || ||1 2j d 2 p = 2 | f, j,n ⇥| Cf || ||1 2j( +d/2) t = x 2jn 2j ⇥f, j,n ⇤ = 1 2j d 2 ⇤ f(x) x 2jn 2j ⇥ dx = 2j d 2 ⇤ R(2jt) (t)dt If f is C on supp(⇥j,n ), p : f(x) = P(x 2jn) + R(x 2jn) = P(2jt) + R(2jt)
46. ### 1D Wavelet Coefficient Behavior −0.2 −0.1 0 0.1 0.2 −0.2

−0.1 0 0.1 0.2 −0.5 0 0.5 −0.5 0 0.5 0 0.2 0.4 0.6 0.8 1
47. ### 1D Wavelet Coefficient Behavior If f is C in supp(

j,n ), then | f, j,n ⇥| 2j( +1/2)||f||C || ||1 −0.2 −0.1 0 0.1 0.2 −0.2 −0.1 0 0.1 0.2 −0.5 0 0.5 −0.5 0 0.5 0 0.2 0.4 0.6 0.8 1
48. ### 1D Wavelet Coefficient Behavior If f is C in supp(

j,n ), then | f, j,n ⇥| 2j( +1/2)||f||C || ||1 | f, j,n ⇥| 2j/2||f|| || ||1 If f is bounded (e.g. around a singularity), then −0.2 −0.1 0 0.1 0.2 −0.2 −0.1 0 0.1 0.2 −0.5 0 0.5 −0.5 0 0.5 0 0.2 0.4 0.6 0.8 1
49. ### For Fourier, linear non-linear, sub-optimal. For wavelets, linear non-linear, optimal.

Piecewise Regular Functions in 1D Theorem: If f is C outside a ﬁnite set of discontinuities: n[M] = ||f fn M ||2 = O(M 1) (Fourier), O(M 2 ) (wavelets).
50. ### Examples of 1D Approximations 0 0.2 0.4 0.6 0.8 1

0 0.2 0.4 0.6 0.8 1 0 0.2 0.4 0.6 0.8 1 0 0.2 0.4 0.6 0.8 1 0 0.2 0.4 0.6 0.8 1 0 0.2 0.4 0.6 0.8 1 0 0.2 0.4 0.6 0.8 1 −2 −1.8 −1.6 −1.4 −1.2 −1 −0.8 −6 −5.5 −5 −4.5 −4 −3.5 −3 −2.5 −2 −1.5 −1
51. ### Large coe cient | f, jn ⇥| < T x1

x2 S = {x1, x2 } Localizing the Singular Support f(x) j1 j2 Small coe cient Singular support: |Cj | K|S| = constant
52. ### Large coe cient | f, jn ⇥| < T x1

x2 S = {x1, x2 } Localizing the Singular Support f(x) j1 j2 Small coe cient Singular support: Coe cient behavior: Regular, n Cc j : |⇥f, j,n ⇤| C2j( +1/2) Singular, n Cj : |⇥f, j,n ⇤| C2j/2 |Cj | K|S| = constant
53. ### Large coe cient | f, jn ⇥| < T x1

x2 S = {x1, x2 } Localizing the Singular Support f(x) j1 j2 Small coe cient Singular support: Coe cient behavior: Regular, n Cc j : |⇥f, j,n ⇤| C2j( +1/2) Singular, n Cj : |⇥f, j,n ⇤| C2j/2 Cut-o scales (depends on T): Singular: 2j1 = (T/C) 1 +1/2 Regular: 2j2 = (T/C)2 |Cj | K|S| = constant
54. ### Large coe cient | f, jn ⇥| < T Hand-made

approximate: ˜ fM = j j2 n Cj f, j,n ⇥ j,n + j j1 n Cc j f, j,n ⇥ j,n x1 x2 S = {x1, x2 } Localizing the Singular Support f(x) j1 j2 Small coe cient Singular support: Coe cient behavior: Regular, n Cc j : |⇥f, j,n ⇤| C2j( +1/2) Singular, n Cj : |⇥f, j,n ⇤| C2j/2 Cut-o scales (depends on T): Singular: 2j1 = (T/C) 1 +1/2 Regular: 2j2 = (T/C)2 |Cj | K|S| = constant
55. ### ||f fM ||2 ||f ˜ fM ||2 j<j2,n Cj |⇥f,

j,n ⇤|2 + j<j1,n Cc j |⇥f, j,n ⇤|2 Computing Error and #Coefficients f(x) j1 j2 n Cc j : |⇥f, j,n ⇤| C2j( +1/2) n Cj : |⇥f, j,n ⇤| C2j/2 2j1 = (T/C) 1 +1/2 2j2 = (T/C)2 j<j2 (K|S|) C22j + j<j1 2 j C22j(2 +1) = O(2j2 + 22 j1 ) = O(T2 + T 2 +1/2 ) = O(T 2 +1/2 ) |Cj | K|S| = constant Singulatities regular part
56. ### ||f fM ||2 ||f ˜ fM ||2 j<j2,n Cj |⇥f,

j,n ⇤|2 + j<j1,n Cc j |⇥f, j,n ⇤|2 Computing Error and #Coefficients f(x) j1 j2 n Cc j : |⇥f, j,n ⇤| C2j( +1/2) n Cj : |⇥f, j,n ⇤| C2j/2 2j1 = (T/C) 1 +1/2 2j2 = (T/C)2 j<j2 (K|S|) C22j + j<j1 2 j C22j(2 +1) M j j2 |Cj | + j j1 |Cc j | j j2 K|S| + j j1 2 j = O(| log(T)| + T 1 +1/2 ) = O(T 1 +1/2 ) = O(2j2 + 22 j1 ) = O(T2 + T 2 +1/2 ) = O(T 2 +1/2 ) |Cj | K|S| = constant Singulatities regular part
57. ### ||f fM ||2 ||f ˜ fM ||2 j<j2,n Cj |⇥f,

j,n ⇤|2 + j<j1,n Cc j |⇥f, j,n ⇤|2 Computing Error and #Coefficients f(x) j1 j2 n Cc j : |⇥f, j,n ⇤| C2j( +1/2) n Cj : |⇥f, j,n ⇤| C2j/2 2j1 = (T/C) 1 +1/2 2j2 = (T/C)2 j<j2 (K|S|) C22j + j<j1 2 j C22j(2 +1) M j j2 |Cj | + j j1 |Cc j | j j2 K|S| + j j1 2 j = O(| log(T)| + T 1 +1/2 ) = O(T 1 +1/2 ) ||f fM ||2 = O(M 2 ) = O(2j2 + 22 j1 ) = O(T2 + T 2 +1/2 ) = O(T 2 +1/2 ) |Cj | K|S| = constant Singulatities regular part
58. ### 2D Wavelet Approximation If f is C in supp( j,n

), then | f, j,n ⇥| 2j( +1)||f||C || ||1 | f, j,n ⇥| 2j||f|| || ||1 If f is bounded (e.g. around a singularity), then
59. ### Fourier Wavelet, both sub-optimal. Wavelets: same result for BV functions

(optimal). Piecewise Regular Functions in 2D n[M] = ||f fn M ||2 = O(M 1/2) (Fourier), O(M 1) (wavelets). Theorem: If f is C outside a set of ﬁnite length edge curves,

61. ### Length(S) = L Localizing the Singular Support in 2D j1

j2 f(x, y) |Cj | LK2 j = constant
62. ### Length(S) = L Localizing the Singular Support in 2D j1

j2 f(x, y) Coe cient behavior: Regular, n Cc j : |⇥f, j,n ⇤| C2j( +1) Singular, n Cj : |⇥f, j,n ⇤| C2j |Cj | LK2 j = constant
63. ### Length(S) = L Localizing the Singular Support in 2D j1

j2 f(x, y) Coe cient behavior: Regular, n Cc j : |⇥f, j,n ⇤| C2j( +1) Singular, n Cj : |⇥f, j,n ⇤| C2j Cut-o scales (depends on T): Singular: 2j1 = (T/C) 1 +1 Regular: 2j2 = T/C |Cj | LK2 j = constant
64. ### Hand-made approximate: ˜ fM = j j2 n Cj f,

j,n ⇥ j,n + j j1 n Cc j f, j,n ⇥ j,n Length(S) = L Localizing the Singular Support in 2D j1 j2 f(x, y) Coe cient behavior: Regular, n Cc j : |⇥f, j,n ⇤| C2j( +1) Singular, n Cj : |⇥f, j,n ⇤| C2j Cut-o scales (depends on T): Singular: 2j1 = (T/C) 1 +1 Regular: 2j2 = T/C |Cj | LK2 j = constant
65. ### ||f fM ||2 ||f ˜ fM ||2 j<j2,n Cj |⇥f,

j,n ⇤|2 + j<j1,n Cc j |⇥f, j,n ⇤|2 Singulatities regular part Computing Error and #Coefficients j1 j2 f(x, y) n Cc j : |⇥f, j,n ⇤| C2j( +1) n Cj : |⇥f, j,n ⇤| C2j 2j2 = T/C 2j1 = (T/C) 1 +1 j<j2 LK2 j C222j + j<j1 2 2j C222j( +1) |Cj | LK2 j = constant = O(2j2 + 22 j1 ) = O(T + T 2 +1 ) = O(T)
66. ### ||f fM ||2 ||f ˜ fM ||2 j<j2,n Cj |⇥f,

j,n ⇤|2 + j<j1,n Cc j |⇥f, j,n ⇤|2 Singulatities regular part Computing Error and #Coefficients j1 j2 f(x, y) n Cc j : |⇥f, j,n ⇤| C2j( +1) n Cj : |⇥f, j,n ⇤| C2j 2j2 = T/C 2j1 = (T/C) 1 +1 j<j2 LK2 j C222j + j<j1 2 2j C222j( +1) |Cj | LK2 j = constant = O(2j2 + 22 j1 ) = O(T + T 2 +1 ) = O(T) M j j2 |Cj | + j j1 |Cc j | j j2 LK2 j + j j1 2 2j = O(T 1 + T 1 +1 ) = O(T 1)
67. ### ||f fM ||2 = O(M 1) ||f fM ||2 ||f

˜ fM ||2 j<j2,n Cj |⇥f, j,n ⇤|2 + j<j1,n Cc j |⇥f, j,n ⇤|2 Singulatities regular part Computing Error and #Coefficients j1 j2 f(x, y) n Cc j : |⇥f, j,n ⇤| C2j( +1) n Cj : |⇥f, j,n ⇤| C2j 2j2 = T/C 2j1 = (T/C) 1 +1 j<j2 LK2 j C222j + j<j1 2 2j C222j( +1) |Cj | LK2 j = constant = O(2j2 + 22 j1 ) = O(T + T 2 +1 ) = O(T) M j j2 |Cj | + j j1 |Cc j | j j2 LK2 j + j j1 2 2j = O(T 1 + T 1 +1 ) = O(T 1)
68. ### Overview • Approximation and Compression • Decay of Approximation Error

• Fourier for Smooth Functions • Wavelet for Piecewise Smooth Functions • Curvelets and Finite Elements for Cartoons
69. ### Geometric image model: f is C outside a set of

C edge curves. BV image: level sets have ﬁnite lengths. Geometric image: level sets are regular. Geometry = cartoon image Sharp edges Smoothed edges Geometrically Regular Images | f|
70. ### Approximation of f, C2 outside C2 edges. Piecewise linear approximation

on M triangles: ˜ fM . Geometic Construction : Finite Elements
71. ### Approximation of f, C2 outside C2 edges. Piecewise linear approximation

on M triangles: ˜ fM . Geometic Construction : Finite Elements Regular areas: M/2 equilateral triangles. M 1/2 M 1/2
72. ### Approximation of f, C2 outside C2 edges. Piecewise linear approximation

on M triangles: ˜ fM . Geometic Construction : Finite Elements Regular areas: M/2 equilateral triangles. M 1/2 M 1/2 M/2 anisotropic triangles. Singular areas:
73. ### Approximation of f, C2 outside C2 edges. Piecewise linear approximation

on M triangles: ˜ fM . Di culties to build e cient approximations. No optimal strategies (greedy solutions). Theorem: If f is C2 outside a set of C2 contours, then one has for an adapted triangulation ||f ˜ fM ||2 = O(M 2). Geometic Construction : Finite Elements Regular areas: M/2 equilateral triangles. M 1/2 M 1/2 M/2 anisotropic triangles. Singular areas:

75. ### Curvelet Atoms Parabolic dyadic scaling: Rotation: [Candes, Donoho] [Candes, Demanet,

Ying, Donoho] “width length2”

sampling:
77. ### Discrete curvelets: O(N log(N)) algorithm. Redundancy 5 =⇥ not e

cient for compression. M-term curvelet approximation: Curvelet Approximation Theorem: If f is C2 outside a set of C2 edges, ||f fM ||2 = O(M 2(log M)3).
78. ### Works on elongated edges. Works also on locally parallel textures

! Curvelets Denoising