slides of Anouar Seghir for the Reading Classics Seminar

Dfbaebe5e96e827d993483f842c74fa2?s=47 Xi'an
January 13, 2014

slides of Anouar Seghir for the Reading Classics Seminar

presentation of the paper Adapting to unknown smoothness via wavelet shrinkage by Donoho and Johnstone, JASA 1995

Dfbaebe5e96e827d993483f842c74fa2?s=128

Xi'an

January 13, 2014
Tweet

Transcript

  1. Adapting  to  unknown   smoothness  via  wavelet   shrinkage  

      David  L.  Donoho  and  Iain  M.  Johnstone     J.  American  Statistical  Assoc  Vol.  90,  No.432    (Dec  1995).       Presented  by  :  Anouar  Seghir   Master  TSI   January  13,2014  
  2. Outline Introduction I.  Discret  Wavelet  Transform II.  Threshold III. Adaptive  Estimation

    IV. Examples Conclusion 2
  3. Introduction Ø Noise: 3

  4. Introduction Ø Denoising  Signal: 4

  5. Introduction Ø Problem  formulation:            Given  N

     noisy  samples  of  a  function  f: Ø Goal:              To  estimate  the  vector         5 y i = f (t i )+ z i !!!!!i =1,..., N t i = (i !1) N !!!!!!!z i !iid!N(0,! 2 ) f = ( f (t i ) i=1 N )
  6. Introduction Ø Estimation:   Ø Usual  approach:          

     We  specify  a  class  of  function            :  Minimax  Risk   We  seek  an  estimator                aJaining    the  minimax  risk. 6 f ^ = arg!min f ^ !R( f ^ , f ) R( f ^ , f ) = N !1.!E || f ^ ! f || 2 2 R(N,!) = inf f ^ !sup f !R( f ^ , f ) f ^ !
  7. Introduction Ø Many  theoritical  developpements: •  Stone(1982) •  Nemirovsky •  Polyak

     and  Tsybakov(1985) Ø In  Practice: •  No  knowledge  of  an  a  priori  class   •  Unknown  smoothness 7 !
  8. Introduction Ø Donoho  and  Johnstone    Sureshrink  approach: 1.  Apply  DWT(discret

     wavelet  transform)  to  noisy  data 2.  Thresholding  of  Noisy  Wavelet  Coefficients 3.  Stein’s  Unbiased  Estimate  of  Risk  for  Threshold  Choice Ø  Sureshrink  is  asymptotically  near-­‐‑minimax  over  large   intervals  of  the  besov,  Sobolev  and  Triebel  Scales. 8
  9. I.  Discret  Wavelet  Transform 1.  Wavelet  recall 2.  Discret  wavelet

     transform  (DWT) 9
  10. I.  Discret  Wavelet  transform   10 1  Wavelet  recall

  11. I.  Discret  Wavelet  transform Ø  Continious  wavelet  transform: •  Ψ

     is  the  mother  wavelet •  a  is  the  scale  parameter •  b  is  the  translation  parameter •   Љhas  mean  equal  to  zero 11 1  Wavelet  recall !! !"# !! ! ! ! ! !! ! ! ! ! !! !! !! ! !"!!!!!!!! ! !!!!!! ! ! ! !"# !! ! ! ! !!!! ! ! !!!! ! ! ! ! ! ! ! ! ! ! ! ! !
  12. I.  Discret  Wavelet  transform Ø  Discret  wavelet  transform: •  One

     restricts  the  parameters  a,b  to  only  discrete  values:   •  Dyadic  transform:   12 2  Wavelet  recall ! ! ! ! !!!!!!! ! !! ! !!!!!!!! ! !! ! ! !"# !!! ! ! !!!! ! ! ! ! ! ! !!! ! !!! ! !! !! !!! !! ! !"!! ! ! ! ! !!!! ! ! !!!! !!!! ! ! ! ! ! !!!! ! ! ! ! !!!!! !! !!! ! !!! ! ! ! ! !! ! !!!!!!!! ! !!! ! !!!! ! ! !!!!!! !!!! ! ! ! ! ! !
  13. I.  Discret  Wavelet  transform •  Localized  in  time  an  frequency

    •  Many  choice  of  mother  wavelet: §  Continuous  case:  Meyer,  Mexican  hat §  Discrete  case:  Haar  ,  daubechies 13 3  Wavelet  recall
  14. I.  Discret  Wavelet  Transform 1.  Wavelet  recall 2.  Discret  wavelet

     transform  (DWT) 14
  15. I.  Discret  Wavelet  transform Ø  Orthonormal  Family  of  Wavlet: 15

    2  Discret  Wavelet  transform • !"#$%&'()$"#%)*+,*-$.(%(+)/+(0-1'++%(+ !! !!! ("12+3+ o !!! ! !!!! ! !!!! + o !"#!!!! !! ! ! + o !"#!!!! !! ! !+ o !!!+("12+#2-#!! !!! ! !! !!!! +%(+-*+)#2)*)&4-$+5-(%(+)/++!! ++ o !!! ! !! ! ! !! ! !!!! ! ! !!!! + o !! !! ! ! !!! ! ! !! ! !!! !!!!! ! !! + ! ! ! ! ! ! !!!! ! ! ! ! ! ! ! ! !! ! ! ! ! ! ! ! !!!! ! !! ! ! ! ! ! !!!!!! !!! ! !!!! !
  16. I.  Discret  Wavelet  transform Ø  Mallat’s  Algorithm:      

        •  H  is  a  high  pass  filter •  G  is  a  low  pass  filter 16 2  Discret  Wavelet  transform
  17. I.  Discret  Wavelet  transform Ø  DWT  in  the  problem:  

            17 2  Discret  Wavelet  transform • !"#"$$! ! !!! ! !!! !!!!!! ! !!$ • !!$%&'()('$*+,$ o !!!"#$%&!!"!!"!#$%&$ o -$$./0012#$3(&4#5$ • 612$7$"&'$-$8%)('$9($4(#$"$#2"&.812:"#%1&$:"#2%)$;,$ o ! ! !"$ o ! ! !!!$$$ $ !! ! !!!! !!!! ! !!! $$$$ $ $$$$ !! ! !!!!! ! !!!!!! ! !!!!!!!! ! !!!!!!!"!#!"##$$ $
  18. II.  Thresholding 1.  Noisy  Wavelet  Coefficients 2.  Threshold  Selection  by

     Sure 18
  19. II.  Thresholding Ø  W  transforms  whithe  noise  into  white  noise:

    Ø  W  transforms  an  estimator  in  one  domain  into  an  estimator  in  the   other  domain:   19 1  Noisy  Wavelet  Coefficients !!!! ! !!!! ! !!!! !!!!!!!!!!!!!!! !!!"!!!!!!!!! ! ! !!!!
  20. II.  Thresholding Ø   Most  of  the  coefficients  in  a  noiseless

     wavelet  transform  are   effectively  zero. Ø  Recovering  f  is  equivalent  to  recover  the  significantly  nonzero   coefficients  of  f. Ø  Thresholding  scheme: 20 1  Noisy  Wavelet  Coefficients • !"#$$%&'%()$$'!!!! ' • !*++,%&'$)-.+'!!!! '
  21. II.  Thresholding Ø  Hard  Thresholding  :   Ø  Plot  du

     Hard  Treshold  pr  un  certain  t Ø  Soft  Thresholding:     Ø  Plot  du  soft  threshold  pr  le  meme  t  que  above 21 1  Noisy  Wavelet  Coefficients !! !!!! ! !!!!!"! !!!! ! ! !!!! !!!!"! !!!! ! ! ! !! !!!! ! !"#!!!!! !! !!!! ! !!!!
  22. II.  Thresholding 22 1  Noisy  Wavelet  Coefficients

  23. II.  Thresholding 1.  Noisy  Wavelet  Coefficients 2.  Threshold  Selection  by

     Sure 23
  24. II.  Thresholding Ø Recall  on  SURE  Estimator: •  SURE:  Stein’s  Unbiased

     risk  estimator •  Unbiased  estimator  of  the  MSE  of  an  arbitrary  estimator 24 2  Threshold  Selection  by  Sure • ! ! !!!!"!!"#!!"#"$%"!!"#"$%&%#! • ! ! !!!!!"#$!!"#!!! ! !!"!!!!!!!!! • ! ! !!"#!$%&'("&)*!)+!!!!%,-.!&."&!!! ! ! ! ! !!!!!! • !!!"#$%&!!"##$%$&'"()*$!! !"#$ ! ! !!! ! ! ! ! ! !!! ! !!! !! !!! ! !!! ! !
  25. II.  Thresholding Ø  We  consider  the  soft  threshold  estimator: • 

    By  Stein’s  result: •  Threshold  selection:     25 2  Threshold  Selection  by  Sure !! ! !! ! !! !!! !! !"#$ !!! ! ! ! !!! !! !! ! ! ! ! !! ! !!! ! !!! ! !! ! !"#!"# !!!! !!"#!!!! !"#$ !!! !
  26. II.  Thresholding Ø Donoho  &  Johnoston’s  VisuShrink: •  Use  a  global

     threshold:     •  No  computational  complexity  but  less  adaptive  to  unknow   smoothness.       26 2  Threshold  Selection  by  Sure !! ! !"#$!!!!
  27. II.  Thresholding Ø Extreme  Sparsity  Situation: •  Hybrid  Scheme: o  Measure

     of  sparsity: o  Threshold  choice:     27 2  Threshold  Selection  by  Sure !! ! ! !! ! ! !! ! ! ! !! ! !!! !"# ! ! !!!! ! !!"#$!%#&'( ! ! !!!!!!!!!!"!!! ! ! ! !!!!!!!!!!!!!!!"#$%&'($ !
  28. II.  Thresholding Ø Threshold  estimator:     28 2  Threshold  Selection

     by  Sure !!!"#$%!!"#$ !! ! ! !! !!! !!!!!!!!"!!! ! ! ! !!!!! !! !!! !!!!!!!!!"#$%&'($ ! !"#$%&#'()!!*+($,+")-('%)!!),$&)$.,+("+)*$"/(+#+)0&,+1+#)#(&."-'(%)
  29. II.  Thresholding Ø Computational  effort: •  Threshold  selection     • 

    To  apply  Threshold     29 2  Threshold  Selection  by  Sure !!!"#$ ! !! !!!!!
  30. III.  Adaptive  Estimation 1.  The  Besov  Scale 2.  Adaptive  Estimation

     over  Besov  Bodies 30
  31. III.  Adaptive  Estimation Ø   Besov  space: •  De  Voor  and

     Popov  (1988)     31 1  The  Besov  Scale • ! ! ! !!"!#$%&!'())*#*+,*! ! ! ! ! ! ! ! !! !!!! ! !"! ! !!! ! ! • !!!! !!! !!!#$%&!-.'/0/1!.)!1-..%&+*11!.)!!! ! !!! !! ! !! ! ! ! ! ! !!!! !! ! ! ! ! ! ! !!! !!!!!" ! ! ! • 2&*!3*1.4!5*-(+.#-!.)!(+'*6!!!! !! !!!).#!! ! !"! ! ! !!!! ! ! !!!! !! ! !! ! !" ! ! ! !!!!!!!"!! ! ! !"#!!!!! !!!! !!! !! !! !!!!!!!!!!!"!! ! ! ! ! !
  32. III.  Adaptive  Estimation Ø   Besov  space: Ø Besov  Ball:    

    32 1  The  Besov  Scale !!!! ! ! !! !!! ! !!! ! !!! !!! !! ! !!!! ! ! ! ! !!!! ! !!! ! !! !!! ! !!! ! !!! !!! !! ! !!!! ! ! ! !
  33. III.  Adaptive  Estimation Ø   Besov  space  and  others  spaces:  

      33 1  The  Besov  Scale ! • !!!! ! !"#$%"$&'!($)*!)*'!+,!!-#.#/'0!123"'!!!! !! ! • 4#5'!6'%'53//7!3//!)*'!!+8!-#.#/'0!123"'1!!! !!35'!"#%)'$%'&!$%!! !!!! ! !!! ! • +')91!! ! !!!:#/&'5!;/311!)*'%!!!!! ! !! !!! ! !!!! !!! ! !!!! ! • <*'!1')!#=!.#>%&'&!035$3)$#%!$1!!3!1>.1')!#=!!!!! ! !!! ! !" ! !"# ! !! ! ! !!!! ! ! ! !! ! !! ! ! ! !!!! ! !! !! ! ! !
  34. III.  Adaptive  Estimation 1.  The  Besov  Scale 2.  Adaptive  Estimation

     over  Besov  Bodies 34
  35. III.  Adaptive  Estimation Ø   Problem  formulation  to  theory:    

    35 2  Adaptive  Estimation  over  Besov  Bodies !!!! ! !!!! ! !!!! !!!!!!!!! !!!"!!!!! !!!!!!!!! ! ! ! ! ! • "#$%&!!! ! !!!! ! • '#&()!'(*+#!! !!!! ! ! !!! ! !!!! !! ! !!!!! ! ! ! ! o !!,-+&#!.-(/!!! ! !!!! ! !! o ! ! ! ! ! ! ! ! ! ! o ! !!!!! ! ! !!! !!!! ! !!!!!! !!! ! !!! !!!!! • 0+1+/,2!-+&3!+&4! ! ! ! !! !!!! ! ! !"# ! !!"!!!! ! ! ! ! ! ! ! ! o 5/(16!$7#!$7-#&7(8*4! ! ! ! !! !!!! ! ! !"# ! !!"!!!! ! ! ! !!!"#$!!!"#$! ! ! ! ! ! o ! !!!"#$!!!"#$! ! ! !!"#$!!!"#$ !!!!! !!!! !
  36. III.  Adaptive  Estimation Ø   Fundamental  theorem: Ø  Using  this  result

     we  the  have  the  following  Theorem:     36 2  Adaptive  Estimation  over  Besov  Bodies !"#$!! ! ! ! ! ! ! $$#%"&'$ !"#! !!!!!! ! !! ! !!!"#$!!!"#$! ! ! ! ! ! !! ! !! !!!! ! ! ! ! ! ! !!!!!! ! !$ • !"#$#%"$&'()*"#"$+,-"."#$,/,.0('($)1**"(21/&$#1$,$+,-"."#$!$%,-'/3$*$/4..$ 515"/#($,/&$*$)1/#'/4"($&"*'-,#'-"(6$! ! !"# !! ! !$ • !"#$#%"$5'/'5,7$*'(8$9"$&"/1#"&$90$$ ! !! !!!! ! !!! ! !"# ! !!" !!!!!!! ! ! ! ! ! ! ! $ • :%"/$;4*";%*'/8$'($(45.#,/"14($/",*.0$5'/'5,7<$ $ !!" ! !!! ! ! ! !!"#$!!!"#$! ! ! ! !! !!!! ! ! !!!!!!!! ! !$ $ =1*$,..$!! ! ! !!! !!6$=1*$,..$! ! !!! !!6$,/&$=1*$,..$!! ! ! ! !$
  37. IV.  Examples 1.  VisuShrink  versus  Sureshrink 37 • !"#$%&'''''''''''''''! !

    ! !! ! ! ! !! !'''''''! ! ! !!!!"# ! ! ! ' • !()*&'''''''''! ! ! !! ! !!!!! ! !! !''''''''''! ! ! !! ! ! !!' • +#**",-'''! ! ! !!! ! !! !"# !! !!! !!! !!!!!! ! !!!"' • .,/0&12,'''! ! ! ! !"# !!! ! !"# ! ! !!! ! !"#!!!!" ! !!' ' TBlock=[0.1,0.13,0.15,0.23,0.25,0.40,0.44,0.65,0.76, 0.78,0.81]; HBlock=[4,-5,3,-4,5,-4.2,2.1,4.3,-3.1,2.1,-4.2]; HBump=[4,5,3,4,5,4.2,2.1,4.3,3.1,5.1,4.2]; WBump=[0.005,0.005,0.006,0.01,0.01,0.03,0.01,0.01,0. 005,0.008,0.005]; • ! ! !"#!!"#! ! ! !!"!#!!"#$% '
  38. III.  Adaptive  Estimation     38 2  Adaptive  Estimation  over

     Besov  Bodies
  39. Conclusion 1.  SureShrink  is  nearly  Minimax  over  a  large  range

     of   Besov  Spaces. 2.     Low  computational  complexity 3.  More  important  is  the  choose  of  the  wavelet  family  and   the  number  of  resolution. 39
  40. References 1.  Ten  Lectures  on  Wavelet  -­‐‑Daubechies 2.     Ideal

     Spatial  adaptation  by  wavelet  shrinkage-­‐‑ Donoho&Johnstone 3.  Multiresolution  Analysis  and  fast  algorithms  on  an  interval  – Daubechies-­‐‑Cohen-­‐‑Vial-­‐‑Jawerth 4.  OndeleJes  sur  l’intervalle-­‐‑Meyer 40