slides of Anouar Seghir for the Reading Classics Seminar

presentation of the paper Adapting to unknown smoothness via wavelet shrinkage by Donoho and Johnstone, JASA 1995

Transcript

1. Adapting  to  unknown  
   smoothness  via  wavelet  
   shrinkage    
   David  L.  Donoho  and  Iain  M.  Johnstone    
   J.  American  Statistical  Assoc  Vol.  90,  No.432    (Dec  1995).  
    
    
   Presented  by  :  Anouar  Seghir  
   Master  TSI  
   January  13,2014  
   

   View Slide

2. Outline
   Introduction
   I.  Discret  Wavelet  Transform
   II.  Threshold
   III. Adaptive  Estimation
   IV. Examples
   Conclusion
   2
   

   View Slide

3. Introduction
   Ø Noise:
   
   3
   

   View Slide

4. Introduction
   Ø Denoising  Signal:
   
   4
   

   View Slide

5. Introduction
   Ø Problem  formulation:  
            Given  N  noisy  samples  of  a  function  f:
   
   
   
   
   Ø Goal:  
              To  estimate  the  vector        
   5
   y
   i
   = f (t
   i
   )+ z
   i
   !!!!!i =1,..., N
   t
   i
   =
   (i !1)
   N
   !!!!!!!z
   i
   !iid!N(0,! 2 )
   f = ( f (t
   i
   )
   i=1
   N )
   

   View Slide

6. Introduction
   Ø Estimation:  
   Ø Usual  approach:
              We  specify  a  class  of  function            :
    Minimax  Risk
    
   We  seek  an  estimator                aJaining    the  minimax  risk.
   6
   f
   ^
   = arg!min
   f
   ^
   !R( f
   ^
   , f )
   R( f
   ^
   , f ) = N !1.!E || f
   ^
   ! f ||
   2
   2
   R(N,!) = inf
   f
   ^
   !sup
   f
   !R( f
   ^
   , f )
   f
   ^
   !
   

   View Slide

7. Introduction
   Ø Many  theoritical  developpements:
   •  Stone(1982)
   •  Nemirovsky
   •  Polyak  and  Tsybakov(1985)
   Ø In  Practice:
   •  No  knowledge  of  an  a  priori  class  
   •  Unknown  smoothness
   
   
   
   7
   !
   

   View Slide

8. Introduction
   Ø Donoho  and  Johnstone    Sureshrink  approach:
   
   1.  Apply  DWT(discret  wavelet  transform)  to  noisy  data
   2.  Thresholding  of  Noisy  Wavelet  Coefficients
   3.  Stein’s  Unbiased  Estimate  of  Risk  for  Threshold  Choice
   Ø  Sureshrink  is  asymptotically  near-­‐‑minimax  over  large  
   intervals  of  the  besov,  Sobolev  and  Triebel  Scales.
   
   
   
   
   
   8
   

   View Slide

9. I.  Discret  Wavelet  Transform
   1.  Wavelet  recall
   
   2.  Discret  wavelet  transform  (DWT)
   
   
   
   9
   

   View Slide

10. I.  Discret  Wavelet  transform
    
     
    
    10
    1  Wavelet  recall
    

    View Slide

11. I.  Discret  Wavelet  transform
    Ø  Continious  wavelet  transform:
    
    
    
    •  Ψ  is  the  mother  wavelet
    •  a  is  the  scale  parameter
    •  b  is  the  translation  parameter
    •   Љhas  mean  equal  to  zero
    
    
    
    11
    1  Wavelet  recall
    !!
    !"# !! ! !
    !
    !
    !!
    ! ! !
    !
    !!
    !!
    !! ! !"!!!!!!!! ! !!!!!!
    !
    !
    !
    !"# !! ! ! ! !!!!
    !
    !
    !!!!
    ! !
    !
    !
    !
    ! ! !
    !
    !
    !
    !
    !
    

    View Slide

12. I.  Discret  Wavelet  transform
    Ø  Discret  wavelet  transform:
    •  One  restricts  the  parameters  a,b  to  only  discrete  values:  
    
    •  Dyadic  transform:  
    12
    2  Wavelet  recall
    ! ! !
    !
    !!!!!!! ! !!
    !
    !!!!!!!! ! !!
    !
    !
    !"# !!! ! ! !!!!
    ! !
    !
    !
    !
    ! !!!
    !
    !!! !
    !!
    !!
    !!!
    !! ! !"!!
    ! ! ! ! !!!!
    !
    !
    !!!!
    !!!!
    !
    !
    !
    !
    !
    !!!!
    ! ! !
    !
    !!!!! !!
    !!! ! !!!
    !
    !
    !
    !
    !!
    ! !!!!!!!!
    ! !!!
    !
    !!!!
    ! ! !!!!!! !!!! ! ! !
    !
    !
    !
    

    View Slide

13. I.  Discret  Wavelet  transform
    
    
    •  Localized  in  time  an  frequency
    •  Many  choice  of  mother  wavelet:
    §  Continuous  case:  Meyer,  Mexican  hat
    §  Discrete  case:  Haar  ,  daubechies
    
    
    13
    3  Wavelet  recall
    

    View Slide

14. I.  Discret  Wavelet  Transform
    1.  Wavelet  recall
    
    2.  Discret  wavelet  transform  (DWT)
    
    
    
    14
    

    View Slide

15. I.  Discret  Wavelet  transform
    Ø  Orthonormal  Family  of  Wavlet:
    
    
    
    
    15
    2  Discret  Wavelet  transform
    • !"#$%&'()$"#%)*+,*-$.(%(+)/+(0-1'++%(+ !! !!!
    ("12+3+
    o !!! ! !!!!
    ! !!!!
    +
    o !"#!!!!
    !!
    ! ! +
    o !"#!!!!
    !!
    ! !+
    o !!!+("12+#2-#!! !!! ! !! !!!!
    +%(+-*+)#2)*)&4-$+5-(%(+)/++!!
    ++
    o !!! ! !! ! ! !!
    ! !!!! ! ! !!!!
    +
    o !! !! ! ! !!! ! ! !!
    ! !!! !!!!! ! !!
    +
    !
    !
    ! ! !
    ! !!!! ! ! !
    ! ! ! ! ! !!
    ! ! !
    !
    !
    !
    !
    !!!!
    !
    !! ! !
    ! ! ! !!!!!!
    !!!
    ! !!!!
    !
    

    View Slide

16. I.  Discret  Wavelet  transform
    Ø  Mallat’s  Algorithm:
             
    
    •  H  is  a  high  pass  filter
    •  G  is  a  low  pass  filter
    
    
    16
    2  Discret  Wavelet  transform
    

    View Slide

17. I.  Discret  Wavelet  transform
    Ø  DWT  in  the  problem:
    
             
    
    
    
    
    
    17
    2  Discret  Wavelet  transform
    • !"#"$$! ! !!!
    !
    !!!
    !!!!!! ! !!$
    • !!$%&'()('$*+,$
    o !!!"#$%&!!"!!"!#$%&$
    o -$$./0012#$3(&4#5$
    • 612$7$"&'$-$8%)('$9($4(#$"$#2"&.812:"#%1&$:"#2%)$;,$
    o ! ! !"$
    o ! ! !!!$$$
    $
    !!
    ! !!!!
    !!!!
    !
    !!!
    $$$$ $ $$$$
    !! ! !!!!! ! !!!!!! ! !!!!!!!! ! !!!!!!!"!#!"##$$
    $
    

    View Slide

18. II.  Thresholding
    1.  Noisy  Wavelet  Coefficients
    
    2.  Threshold  Selection  by  Sure
    
    
    18
    

    View Slide

19. II.  Thresholding
    Ø  W  transforms  whithe  noise  into  white  noise:
    
    
    Ø  W  transforms  an  estimator  in  one  domain  into  an  estimator  in  the  
    other  domain:
     
    
    
    19
    1  Noisy  Wavelet  Coefficients
    !!!!
    ! !!!!
    ! !!!!
    !!!!!!!!!!!!!!!
    !!!"!!!!!!!!!
    ! ! !!!!
    

    View Slide

20. II.  Thresholding
    Ø   Most  of  the  coefficients  in  a  noiseless  wavelet  transform  are  
    effectively  zero.
    Ø  Recovering  f  is  equivalent  to  recover  the  significantly  nonzero  
    coefficients  of  f.
    
    
    Ø  Thresholding  scheme:
    
    
    20
    1  Noisy  Wavelet  Coefficients
    • !"#$$%&'%()$$'!!!!
    '
    • !*++,%&'$)-.+'!!!!
    '
    

    View Slide

21. II.  Thresholding
    Ø  Hard  Thresholding  :  
    Ø  Plot  du  Hard  Treshold  pr  un  certain  t
    Ø  Soft  Thresholding:    
    Ø  Plot  du  soft  threshold  pr  le  meme  t  que  above
    
    
    21
    1  Noisy  Wavelet  Coefficients
    !!
    !!!!
    !
    !!!!!"! !!!!
    ! !
    !!!!
    !!!!"! !!!!
    ! !
    !
    !!
    !!!!
    ! !"#!!!!!
    !! !!!!
    ! !!!!
    

    View Slide

22. II.  Thresholding
    
    
    
    22
    1  Noisy  Wavelet  Coefficients
    

    View Slide

23. II.  Thresholding
    1.  Noisy  Wavelet  Coefficients
    
    2.  Threshold  Selection  by  Sure
    
    
    23
    

    View Slide

24. II.  Thresholding
    Ø Recall  on  SURE  Estimator:
    •  SURE:  Stein’s  Unbiased  risk  estimator
    •  Unbiased  estimator  of  the  MSE  of  an  arbitrary  estimator
    
    24
    2  Threshold  Selection  by  Sure
    • ! ! !!!!"!!"#!!"#"$%"!!"#"$%&%#!
    • ! ! !!!!!"#$!!"#!!! ! !!"!!!!!!!!!
    • ! ! !!"#!$%&'("&)*!)+!!!!%,-.!&."&!!! ! ! ! ! !!!!!!
    • !!!"#$%&!!"##$%$&'"()*$!!
    !"#$ ! ! !!! ! ! ! ! ! !!!
    !
    !!!
    !!
    !!!
    !
    !!!
    !
    !
    

    View Slide

25. II.  Thresholding
    
    Ø  We  consider  the  soft  threshold  estimator:
    •  By  Stein’s  result:
    •  Threshold  selection:
    
       
    25
    2  Threshold  Selection  by  Sure
    !!
    ! !!
    ! !!
    !!!
    !!
    !"#$ !!! ! ! ! !!! !! !!
    ! ! ! ! !!
    ! !!!
    !
    !!!
    !
    !! ! !"#!"#
    !!!! !!"#!!!!
    !"#$ !!! !
    

    View Slide

26. II.  Thresholding
    
    Ø Donoho  &  Johnoston’s  VisuShrink:
    •  Use  a  global  threshold:  
     
    •  No  computational  complexity  but  less  adaptive  to  unknow  
    smoothness.  
    
    
    
       
    26
    2  Threshold  Selection  by  Sure
    !! ! !"#$!!!!
    

    View Slide

27. II.  Thresholding
    
    Ø Extreme  Sparsity  Situation:
    •  Hybrid  Scheme:
    o  Measure  of  sparsity:
    
    o  Threshold  choice:
    
    
    
       
    27
    2  Threshold  Selection  by  Sure
    !!
    ! ! !!
    !
    !
    !!
    !
    ! ! !!
    !
    !!!
    !"#
    !
    !
    !!!!
    !
    !!"#$!%#&'(
    ! !
    !!!!!!!!!!"!!!
    ! ! !
    !!!!!!!!!!!!!!!"#$%&'($
    !
    

    View Slide

28. II.  Thresholding
    
    Ø Threshold  estimator:
    
    
    
       
    28
    2  Threshold  Selection  by  Sure
    !!!"#$%!!"#$ !!
    !
    !
    !!
    !!!
    !!!!!!!!"!!!
    ! ! !
    !!!!!
    !!
    !!!
    !!!!!!!!!"#$%&'($
    !
    !"#$%'()!!*+($,+")-('%)!!),$&)$.,+("+)*$"/(+#+)0&,+1+#)#(&."-'(%)
    

    View Slide

29. II.  Thresholding
    
    Ø Computational  effort:
    
    •  Threshold  selection    
    
    
    •  To  apply  Threshold
    
       
    29
    2  Threshold  Selection  by  Sure
    !!!"#$ ! !!
    !!!!!
    

    View Slide

30. III.  Adaptive  Estimation
    1.  The  Besov  Scale
    
    2.  Adaptive  Estimation  over  Besov  Bodies
    
    
    30
    

    View Slide

31. III.  Adaptive  Estimation
    Ø   Besov  space:
    •  De  Voor  and  Popov  (1988)
    
    
    
    
    
    
       
    31
    1  The  Besov  Scale
    • !
    !
    ! !!"!#$%&!'())*#*+,*!
    !
    !
    ! ! !
    !
    !
    !! !!!! ! !"!
    !
    !!!
    !
    !
    • !!!!
    !!! !!!#$%&!-.'/0/1!.)!1-..%&+*11!.)!!! ! !!! !! ! !!
    ! ! ! ! !
    !!!!
    !! ! ! !
    !
    ! !
    !!! !!!!!" !
    !
    !
    • 2&*!3*1.4!5*-(+.#-!.)!(+'*6!!!! !! !!!).#!! ! !"!
    !
    ! !!!!
    !
    !
    !!!!
    !! !
    !!
    !
    !"
    !
    !
    !
    !!!!!!!"!! ! !
    !"#!!!!!
    !!!!
    !!! !!
    !!
    !!!!!!!!!!!"!! ! !
    !
    !
    !
    

    View Slide

32. III.  Adaptive  Estimation
    Ø   Besov  space:
    Ø Besov  Ball:
    
    
    
       
    32
    1  The  Besov  Scale
    !!!!
    ! ! !! !!! ! !!! ! !!! !!! !! ! !!!!
    ! ! ! !
    !!!!
    ! !!! ! !! !!! ! !!! ! !!! !!! !! ! !!!!
    ! ! ! !
    

    View Slide

33. III.  Adaptive  Estimation
    Ø   Besov  space  and  others  spaces:
    
    
    
    
       
    33
    1  The  Besov  Scale
    !
    • !!!!
    ! !"#$%"$&'!($)*!)*'!+,!!-#.#/'0!123"'!!!!
    !!
    !
    • 4#5'!6'%'53//7!3//!)*'!!+8!-#.#/'0!123"'1!!!
    !!35'!"#%)'$%'&!$%!!
    !!!!
    ! !!!
    !
    • +')91!! ! !!!:#/&'5!;/311!)*'%!!!!! ! !! !!! !
    !!!!
    !!!
    ! !!!!
    !
    • <*'!1')!#=!.#>%&'&!035$3)$#%!$1!!3!1>.1')!#=!!!!!
    ! !!!
    !
    !" ! !"# ! !!
    ! ! !!!!
    ! ! ! !!
    ! !!
    ! ! ! !!!!
    ! !!
    !! ! ! !
    

    View Slide

34. III.  Adaptive  Estimation
    1.  The  Besov  Scale
    
    2.  Adaptive  Estimation  over  Besov  Bodies
    
    
    34
    

    View Slide

35. III.  Adaptive  Estimation
    Ø   Problem  formulation  to  theory:
    
    
    
    
    
    
       
    35
    2  Adaptive  Estimation  over  Besov  Bodies
    !!!!
    ! !!!!
    ! !!!!
    !!!!!!!!!
    !!!"!!!!! !!!!!!!!! ! !
    !
    !
    !
    • "#$%&!!! ! !!!!
    !
    • '#&()!'(*+#!! !!!!
    ! ! !!! ! !!!!
    !! ! !!!!!
    !
    ! ! !
    o !!,-+!.-(/!!! ! !!!!
    ! !!
    o ! ! ! ! !
    !
    ! !
    !
    !
    o ! !!!!!
    !
    ! !!! !!!!
    !
    !!!!!!
    !!! !
    !!!
    !!!!!
    • 0+1+/,2!-+&3!+&4!
    !
    !
    ! !! !!!!
    ! ! !"#
    !
    !!"!!!!
    !
    ! ! ! !
    !
    !
    !
    o 5/(16!$7#!$7-#&7(8*4!
    !
    !
    ! !! !!!!
    ! ! !"#
    !
    !!"!!!!
    !
    ! !
    !!!"#$!!!"#$!
    ! !
    !
    !
    !
    o !
    !!!"#$!!!"#$!
    ! !
    !!"#$!!!"#$
    !!!!!
    !!!!
    !
    

    View Slide

36. III.  Adaptive  Estimation
    Ø   Fundamental  theorem:
    
    Ø  Using  this  result  we  the  have  the  following  Theorem:
    
    
    
    
       
    36
    2  Adaptive  Estimation  over  Besov  Bodies
    !"#$!! ! !
    !
    ! !
    !
    $$#%"&'$
    !"#!
    !!!!!!
    !
    !!
    !
    !!!"#$!!!"#$!
    ! !
    !
    !
    ! !!
    ! !! !!!!
    ! ! ! ! ! ! !!!!!! ! !$
    • !"#$#%"$&'()*"#"$+,-"."#$,/,.0('($)1**"(21/&$#1$,$+,-"."#$!$%,-'/3$*$/4..$
    515"/#($,/&$*$)1/#'/4"($&"*'-,#'-"(6$! ! !"# !! ! !$
    • !"#$#%"$5'/'5,7$*'(8$9"$&"/1#"&$90$$
    ! !! !!!!
    ! !!! ! !"#
    !
    !!"
    !!!!!!!
    !
    ! ! ! !
    !
    !
    $
    • :%"/$;4*";%*'/8$'($(45.#,/"14($/",*.0$5'/'5,7<$
    $
    !!"
    !
    !!! !
    !
    ! !!"#$!!!"#$! ! ! ! !! !!!!
    ! ! !!!!!!!! ! !$
    $
    =1*$,..$!! ! ! !!! !!6$=1*$,..$! ! !!! !!6$,/&$=1*$,..$!!
    ! ! ! !$
    

    View Slide

37. IV.  Examples
    1.  VisuShrink  versus  Sureshrink
    
    
    
    
    37
    • !"#$%&'''''''''''''''! ! ! !!
    ! ! ! !!
    !'''''''! ! ! !!!!"# ! !
    !
    '
    • !()*&'''''''''! ! ! !!
    ! !!!!!
    !
    !!
    !''''''''''! ! ! !! ! ! !!'
    • +#**",-'''! ! ! !!! ! !! !"# !! !!!
    !!!
    !!!!!! ! !!!"'
    • .,/0&12,'''! ! ! ! !"# !!! ! !"# ! ! !!! ! !"#!!!!" ! !!'
    '
    TBlock=[0.1,0.13,0.15,0.23,0.25,0.40,0.44,0.65,0.76,
    0.78,0.81];
    HBlock=[4,-5,3,-4,5,-4.2,2.1,4.3,-3.1,2.1,-4.2];
    HBump=[4,5,3,4,5,4.2,2.1,4.3,3.1,5.1,4.2];
    WBump=[0.005,0.005,0.006,0.01,0.01,0.03,0.01,0.01,0.
    005,0.008,0.005];
    • ! ! !"#!!"#! ! ! !!"!#!!"#$%
    '
    

    View Slide

38. III.  Adaptive  Estimation
    
    
    
    
    
       
    38
    2  Adaptive  Estimation  over  Besov  Bodies
    

    View Slide

39. Conclusion
    1.  SureShrink  is  nearly  Minimax  over  a  large  range  of  
    Besov  Spaces.
    
    2.     Low  computational  complexity
    
    3.  More  important  is  the  choose  of  the  wavelet  family  and  
    the  number  of  resolution.
    
    
    
    39
    

    View Slide

40. References
    1.  Ten  Lectures  on  Wavelet  -­‐‑Daubechies
    
    2.     Ideal  Spatial  adaptation  by  wavelet  shrinkage-­‐‑
    Donoho&Johnstone
    
    3.  Multiresolution  Analysis  and  fast  algorithms  on  an  interval  –
    Daubechies-­‐‑Cohen-­‐‑Vial-­‐‑Jawerth
    
    4.  OndeleJes  sur  l’intervalle-­‐‑Meyer
    
    
    
    40
    

    View Slide