PRML Chapter5 Hessian Matrix

PRML復々習レーン #7
http://atnd.org/events/33833
での発表に用いた資料

Transcript

1. 13.-෮ʑशϨʔϯ
   jrj
   ϔοηߦྻ
   )T0
   ڵຯɿ// ).. ڧԽֶशͱ͔
   12೥12݄16೔೔༵೔
   

   View Slide

2. ϔοηߦྻ
   )FTTJBO.BUSJY
   
   w ϔοηߦྻͷ໾ׂ
   w ϔοηߦྻΛٻΊΔ͜ͱʹΑΓɺޡࠩؔ਺ͷॏΈʹؔ͢
   Δ̎֊ඍ෼ΛධՁ͢Δ͜ͱ͕Մೳ
   w ϔοηߦྻͷݻ༗஋͕શͯਖ਼ʢਖ਼ఆ஋ʣͳΒۃখ఺ɺݻ
   ༗஋͕શͯෛͳΒۃେ఺ɺਖ਼ͱෛͷ྆ํͳΒҌ఺Λࣔ͢
   z = x2 + y2
   z = xe−x2 −y2
   λ
   1
   = 1
   2
   (3 + 33) λ
   2
   = 1
   2
   (3 - 33) λ
   1
   = 1.7 λ
   2
   = 0.86
   λ
   1
   = −1.7 λ
   2
   = −0.86
   12೥12݄16೔೔༵೔
   

   View Slide

3. χϡʔϥϧίϯϐϡʔςΟϯά
   Ͱͷ໾ׂ
   w ඇઢܗ࠷దԽΞϧΰϦζϜʹ͓͚Δޡࠩۂ໘ͷධՁ
   w ֶशσʔλͷมಈ࣌ʹ͓͚Δ࠶ֶशͷߴ଎Խ
   w ʮמΓࠐΈʯΞϧΰϦζϜʹ͓͚Δ৑௕ͳॏΈͷಛఆ
   w ϕΠζωοτϫʔΫͷϥϓϥεۙࣅʹ͓͚Δ༧ଌ෼
   ෍ٴͼ௒ύϥϝʔλͷܾఆɺϞσϧΤϏσϯεධՁ
   ʢઅʣ
   12೥12݄16೔೔༵೔
   

   View Slide

4. ͦͷଞ
   wը૾ॲཧ
   w ϔοηߦྻ͸ฏ໘ͷۂ཰Λද͢ͷͰಛ௃఺நग़ʹར༻
   w YͷϐΫηϧ΢Οϯυ΢Ͱɺࠩ෼σʔλʹجͮ͘ϔοηߦྻΛ
   ܭࢉ͠ɺͦͷݻ༗஋͕ඈͼൈ͚ͯେ͖͘ͳΔࠩ෼த৺఺͸ɺίʔ
   φʔͷՄೳੑେͳͲ
   w਺ཧܭը๏
   wߏ଄෺σβΠϯ
   12೥12݄16೔೔༵೔
   

   View Slide

5. ϔοηߦྻ
   )FTTJBO.BUSJY
   
   ∂2 E
   ∂w
   kl
   ∂w
   mn
   →
   ∂2 E
   ∂w
   ij
   2
   = H
   ij
   i, j ∈{1…W}
   8ύϥϝʔλͷ૯਺
   //ʹ͓͚Δϔοη
   ߦྻͷ֤ཁૉ
   H =
   ∂2 E
   ∂w
   1
   ∂w
   1
   …
   ∂2 E
   ∂w
   1
   ∂w
   W
     
   ∂2 E
   ∂w
   W
   ∂w
   1
   
   ∂2 E
   ∂w
   W
   ∂w
   W
   ⎛
   ⎝
   ⎜
   ⎜
   ⎜
   ⎜
   ⎜
   ⎜
   ⎞
   ⎠
   ⎟
   ⎟
   ⎟
   ⎟
   ⎟
   ⎟
   ॏΈ΍όΠΞεͱ͍ͬͨ
   ύϥϝʔλʹؔͯ͠Ұ༷ʹฒ΂௚͢
   12೥12݄16೔೔༵೔
   

   View Slide

6. jrjͷ֓ཁ
   w ϔοηߦྻ0 8?
   ͷܭࢉྔ͕ඞཁͩ͠ɺೋ֊ඍ෼΋େม
   w ର֯ۙࣅ
   w ඇର֯ཁૉ͕ແࢹͰ͖ΔͷͰ͋Ε͹0 8
   ͰܭࢉՄೳ
   w ֎ੵʹΑΔۙࣅ
   w े෼ʹֶशࡁΈͷঢ়ଶͰ͸ɺ0 8?
   ͰܭࢉՄೳ
   w ٯߦྻͷޮ཰ͷྑ͍ஞ࣍ܭࢉ͕Մೳʢˠ४χϡʔτϯ๏ʣ
   w ༗ݶ෯ͷࠩ෼ʹΑΔۙࣅ
   w ٯ఻ൖ๏Λ༻͍ͯܭࢉ͞ΕͨϔοηߦྻͷνΣοΫʹ༗ޮ
   w Ұ෦ٯ఻ൖެࣜΛऔΓೖΕΔ͜ͱʹΑΓ0 8?
   ͰܭࢉՄೳ
   w ϔοηߦྻͷݫີͳධՁ
   w ϔοηߦྻͷੵͷߴ଎ͳܭࢉ
   12೥12݄16೔೔༵೔
   

   View Slide

7. ࠷దԽΞϧΰϦζϜ
   w(τ +1) = w(τ ) −η∇E
   n
   w(τ +1) = w(τ ) −η 
   H
   n
   (τ )
   ( )−1 ∇E
   n
   w(τ +1) = w(τ ) −ηH
   n
   −1∇E
   n
   ࠷ٸ߱Լ๏
   χϡʔτϯ๏
   ४χϡʔτϯ๏
   ର֯χϡʔτϯ๏
   ϔοηߦྻͷஞ࣍ۙࣅʹ͍ͭͯ͸ɺ%'1 #'(4 -#'(4
   w ڭՊॻͷྲྀΕͰ͸ͳ͍Ͱ͕͢ɺΠϝʔδ͔ͭΉͨΊʹɾɾɾ
   w(τ +1) = w(τ ) −η diagH
   n
   ( )−1 ∇E
   n
   ΑΓߴ଎ͳऩଋ
   ܭࢉྔ࡟ݮʢͨͩ͠ෆ҆ఆԽʣ
   ΋͏গ͠Ϛγͳۙࣅ
   
   H
   n
   (τ ) → H
   n
   ઴ۙ
   12೥12݄16೔೔༵೔
   

   View Slide

8. ର֯ۙࣅ
   w ϔοηߦྻ͕ର֯ߦྻͩͱ͢Ε͹ɺٯߦྻ΍ݻ༗஋ΛٻΊΔͷ
   ΋༰қ͘ɺඇର߲֯ͷӨڹ͕গͳ͍໰୊Ͱ͸༗ҙٛɻ࣮ࡍܭࢉ
   ྔ΋0 8
   ͰࡁΉ
   a
   j
   = w
   ji
   z
   i
   i
   ∑
   ∂2 E
   n
   ∂a
   j
   2
   = h'(a
   j
   )2 w
   kj
   w
   k' j
   ∂2 E
   n
   ∂a
   k
   ∂a
   k'
   k'
   ∑
   k
   ∑ + h''(a
   j
   ) w
   kj
   ∂E
   n
   ∂a
   k
   k
   ∑
   ∂2 E
   n
   ∂w
   ij
   2
   =
   ∂
   ∂w
   ij
   ∂E
   n
   ∂a
   j
   z
   i
   ⎛
   ⎝
   ⎜
   ⎞
   ⎠
   ⎟ =
   ∂2 E
   n
   ∂a
   j
   2
   z
   i
   2
   ∂2 E
   n
   ∂a
   j
   2
   = h'(a
   j
   )2 w
   kj
   ∂2 E
   n
   ∂a
   k
   2
   k
   ∑ + h''(a
   j
   ) w
   kj
   ∂E
   n
   ∂a
   k
   k
   ∑
   ඇର߲͕֯ແࢹग़དྷΔͱ͢Ε͹ର߲֯ͷΈܭࢉ͢Ε͹͍͍ͷͰɹɹɹͷ
   ܭࢉྔͰࡁΈɺ֤੒෼ͷܭࢉ΋ҎԼͷΑ͏ʹ୯७Խ͞ΕΔ
   ∂E
   n
   ∂a
   j
   = h'(a
   j
   ) w
   kj
   ∂E
   n
   ∂a
   k
   k
   ∑
   ΑΓɺϔοηߦྻͷ֤੒෼͸
   ٯ఻ൖͷެࣜ ʹର͠ɺੵͷඍ෼Λߦ͑͹
   O(W )
   ຊདྷ͸ɺ͜Εʹؔͯ͠ϔοηߦ
   ྻશମΛͷܭࢉͰධՁ
   O(W 2 )
   ͨͩ͠ɺϔοηߦྻΛඞཁͱ͢Δओཁ
   ͳ໰୊Ͱ͸ۃ୺ʹඇର֯Ͱ͋Δ͜ͱ͕
   ଟ͘ɺద༻ʹ͸஫ҙ͕ඞཁɾɾɾ
   12೥12݄16೔೔༵೔
   

   View Slide

9. ֎ੵʹΑΔۙࣅ
   ʢΨ΢εɾχϡʔτϯ๏ͱ΋
   
   w ޡࠩؔ਺͕ฏํ࿨ͱͳΔΑ͏ͳ໰୊ʹ͓͍ͯɺे෼ʹ܇࿅͞Εͨঢ়ଶͰ
   ͸ɺϔοηߦྻͷҰ෦ͷ߲͸ແࢹग़དྷΔ͘Β͍খ͘͞ͳΓɺ0 8?
   ͷ
   ୯७ͳੵԋࢉͰධՁ͕Մೳ
   Η = ∇∇E
   = ∇y
   n
   (∇y
   n
   )T
   n=1
   N
   ∑ + (y
   n
   − t
   n
   )∇∇y
   n
   n=1
   N
   ∑
   े෼ʹ܇࿅͞ΕͨωοτϫʔΫͰ͸ɺZ͕Uʹे෼ۙͮ͘͜ͱʹΑΓɺୈ߲͕̎θϩʹۙ
   ͖ͮɺແࢹ͢Δ͜ͱ͕ग़དྷΔͷͰҎԼͷΑ͏ʹۙࣅՄೳ
   Η  ∇y
   n
   (∇y
   n
   )T
   n=1
   N
   ∑ ≡ b
   n
   b
   n
   T
   n=1
   N
   ∑ -FWFOWFSH.BSRVBSEUۙࣅ
   ʢ֎ੵʹΑΔۙࣅʣ
   E = 1
   2
   (y
   n
   − t
   n
   )2
   n=1
   N
   ∑ ͷΑ͏ͳճؼ໰୊ʹର͠ɺɹɹɹɹɹɹɹɹɹɹΑΓ
   ∇E = (y
   n
   − t
   n
   )∇y
   n
   n=1
   N
   ∑
   12೥12݄16೔೔༵೔
   

   View Slide

10. ϔοηٯߦྻͷஞ࣍ܭࢉ
    w ֎ੵʹΑΔۙࣅΛϕʔεʹͯ͠ɺ-ݸͷσʔλ఺ʹର͠ஞ࣍
    ܭࢉ͠ɺ)Ћ*ͷٯߦྻΛঃʑʹݟ͚ͭΔ͜ͱ͕Մೳ
    Η
    L+1
    = Η
    L
    + b
    L+1
    b
    L+1
    T
    8PPECVSZ߃౳ࣜ
    Η
    N
    = b
    n
    b
    n
    T
    n=1
    N
    ∑
    Η
    L+1
    −1 = Η
    L
    + b
    L+1
    b
    L+1
    T
    ( )−1 = Η
    L
    −1 +
    Η
    L
    −1b
    L+1
    b
    L+1
    T Η
    L
    −1
    1+ b
    L+1
    T Η
    L
    −1b
    L+1
    Η
    0
    = b
    1
    b
    1
    T = αI (α 1)
    Η +αI
    L +1= N ʹͳΔ·Ͱ܁Γฦ͢͜ͱͰ ͷٯߦྻ͕ݟ͔ͭΔ
    ͱͯ͠
    12೥12݄16೔೔༵೔
    

    View Slide

11. ༗ݶ෯ͷࠩ෼ʹΑΔۙࣅ
    w ਺஋ਫ਼౓্ͷݶք͕͋Δ΋ͷͷɺٯ఻ൖ๏ͰٻΊΒΕͨϔοηߦ
    ྻͷݕূʹ༗ޮɻී௨ʹܭࢉ͢Δͱ0 8?
    ͷͱ͜Ζɺ̍֊ඍ෼
    ͸ٯ఻ൖͷެࣜΛద༻͢Δ͜ͱͰ0 8?
    Ͱܭࢉ͢Δ͜ͱ΋Մೳ
    ∂2 E
    ∂w
    ji
    ∂w
    lk
    = 1
    4ε2
    E(w
    ji
    + ε,w
    lk
    + ε)− E(w
    ji
    + ε,w
    lk
    − ε)− E(w
    ji
    − ε,w
    lk
    + ε)+ E(w
    ji
    − ε,w
    lk
    − ε)
    { }+O(ε2 )
    ∂2 E
    ∂w
    ji
    ∂w
    lk
    =
    ∂
    ∂w
    lk
    ∂E
    ∂w
    ji
    = 1
    2ε
    ∂E
    ∂w
    ji
    (w
    lk
    + ε)−
    ∂E
    ∂w
    ji
    (w
    lk
    − ε)
    ⎧
    ⎨
    ⎪
    ⎩
    ⎪
    ⎫
    ⎬
    ⎪
    ⎭
    ⎪
    +O(ε2 )
    ∂E
    ∂w
    ji
    ͸ٯ఻ൖ๏Ͱޮ཰తʹٻ
    ΊΔ͜ͱ͕ग़དྷΔͷͰ
    ઁಈЏʹԠͨ͡࢒ཹޡࠩ
    ϔοηߦྻͷ֤ཁૉ͸
    12೥12݄16೔೔༵೔
    

    View Slide

12. ϔοηߦྻͷݫີͳධՁ
    w ٯ఻ൖͷςΫχοΫͷ֦ுΛར༻ͨ͠ɺ೚ҙͷϑΟʔ
    υϑΥϫʔυߏ଄Λ΋ͭωοτϫʔΫͷݫີͳϔοη
    ߦྻΛ0 8?
    ͰٻΊΔख๏͕ଘࡏ͢Δ Β͍͠ʣ
    #JTIPQ #JTIPQ
    
    w ̎૚ωοτϫʔΫͷ৔߹͸ɺͦΕͧΕͷॏΈͷ૊Έ߹
    ΘͤͰϒϩοΫԽ͢Δ͜ͱͰ༰қʹࣜΛಋग़͢Δ͜ͱ
    ͕Ͱ͖ΔʢΒ͍͠ʣ
    12೥12݄16೔೔༵೔
    

    View Slide

13. ϔοηߦྻͷੵͷߴ଎ͳܭࢉ
    w ී௨ʹϔοηߦྻΛධՁͯ͠ɺੵΛٻΊΔͱ
    0 8?
    ɻͨͩ͠ɺϔοηߦྻࣗମͰ͸ͳ͘ɺ͋Δϕ
    Ϋτϧͱͷੵ͕ඞཁͳ͚ͩͰ͋Ε͹ɺޡࠩٯ఻ൖͷख
    ๏΋༻͍Δ͜ͱʹΑΓ0 8
    ʹͰ͖Δ
    vTH = vT∇ ∇E
    ( )
    Ͱ͋Δ͜ͱ͔Βɺ㲆&ΛٻΊΔͷʹඞཁͳޡࠩٯ
    ఻ൖʹؔΘΔશͯͷํఔࣜͷܭࢉ෦෼ʹඍ෼ԋ
    ࢉࢠɹɹɹΛ࡞༻ͤ͞Δɻ
    vT∇
    12೥12݄16೔೔༵೔
    

    View Slide