PRML Chapter5 Hessian Matrix

PRML復々習レーン #7
http://atnd.org/events/33833
での発表に用いた資料

Transcript

1. 13.-෮ʑशϨʔϯ
    j
   r
   j ϔοηߦྻ )T0 ڵຯɿ//
   )..
   ڧԽֶशͱ͔ 12೥12݄16೔೔༵೔
2. ϔοηߦྻ
   )FTTJBO
   .BUSJY w ϔοηߦྻͷ໾ׂ w ϔοηߦྻΛٻΊΔ͜ͱʹΑΓɺޡࠩؔ਺ͷॏΈʹؔ͢ Δ̎֊ඍ෼ΛධՁ͢Δ͜ͱ͕Մೳ w ϔοηߦྻͷݻ༗஋͕શͯਖ਼ʢਖ਼ఆ஋ʣͳΒۃখ఺ɺݻ ༗஋͕શͯෛͳΒۃେ఺ɺਖ਼ͱෛͷ྆ํͳΒҌ఺Λࣔ͢ z = x2 + y2 z = xe−x2 −y2 λ 1 = 1 2 (3 + 33) λ 2 = 1 2 (3 - 33) λ 1 = 1.7 λ 2 = 0.86 λ 1 = −1.7 λ 2 = −0.86 12೥12݄16೔೔༵೔
3. χϡʔϥϧίϯϐϡʔςΟϯά Ͱͷ໾ׂ w ඇઢܗ࠷దԽΞϧΰϦζϜʹ͓͚Δޡࠩۂ໘ͷධՁ w ֶशσʔλͷมಈ࣌ʹ͓͚Δ࠶ֶशͷߴ଎Խ w ʮמΓࠐΈʯΞϧΰϦζϜʹ͓͚Δ৑௕ͳॏΈͷಛఆ w ϕΠζωοτϫʔΫͷϥϓϥεۙࣅʹ͓͚Δ༧ଌ෼ ෍ٴͼ௒ύϥϝʔλͷܾఆɺϞσϧΤϏσϯεධՁ ʢઅʣ 12೥12݄16೔೔༵೔
4. ͦͷଞ wը૾ॲཧ w ϔοηߦྻ͸ฏ໘ͷۂ཰Λද͢ͷͰಛ௃఺நग़ʹར༻ w YͷϐΫηϧ΢Οϯυ΢Ͱɺࠩ෼σʔλʹجͮ͘ϔοηߦྻΛ ܭࢉ͠ɺͦͷݻ༗஋͕ඈͼൈ͚ͯେ͖͘ͳΔࠩ෼த৺఺͸ɺίʔ φʔͷՄೳੑେͳͲ w਺ཧܭը๏ wߏ଄෺σβΠϯ 12೥12݄16೔೔༵೔
5. ϔοηߦྻ
   )FTTJBO
   .BUSJY ∂2 E ∂w kl ∂w mn → ∂2 E ∂w ij 2 = H ij i, j ∈{1…W} 8ύϥϝʔλͷ૯਺ //ʹ͓͚Δϔοη ߦྻͷ֤ཁૉ H = ∂2 E ∂w 1 ∂w 1 … ∂2 E ∂w 1 ∂w W    ∂2 E ∂w W ∂w 1  ∂2 E ∂w W ∂w W ⎛ ⎝ ⎜ ⎜ ⎜ ⎜ ⎜ ⎜ ⎞ ⎠ ⎟ ⎟ ⎟ ⎟ ⎟ ⎟ ॏΈ΍όΠΞεͱ͍ͬͨ ύϥϝʔλʹؔͯ͠Ұ༷ʹฒ΂௚͢ 12೥12݄16೔೔༵೔
6. j
   r
   jͷ֓ཁ w ϔοηߦྻ
   
   0 8? ͷܭࢉྔ͕ඞཁͩ͠ɺೋ֊ඍ෼΋େม w ର֯ۙࣅ w ඇର֯ཁૉ͕ແࢹͰ͖ΔͷͰ͋Ε͹0 8 ͰܭࢉՄೳ w ֎ੵʹΑΔۙࣅ w े෼ʹֶशࡁΈͷঢ়ଶͰ͸ɺ0 8? ͰܭࢉՄೳ w ٯߦྻͷޮ཰ͷྑ͍ஞ࣍ܭࢉ͕Մೳ
   ʢˠ
   ४χϡʔτϯ๏ʣ w ༗ݶ෯ͷࠩ෼ʹΑΔۙࣅ w ٯ఻ൖ๏Λ༻͍ͯܭࢉ͞ΕͨϔοηߦྻͷνΣοΫʹ༗ޮ w Ұ෦ٯ఻ൖެࣜΛऔΓೖΕΔ͜ͱʹΑΓ0 8? ͰܭࢉՄೳ w ϔοηߦྻͷݫີͳධՁ w ϔοηߦྻͷੵͷߴ଎ͳܭࢉ 12೥12݄16೔೔༵೔
7. ࠷దԽΞϧΰϦζϜ w(τ +1) = w(τ ) −η∇E n w(τ +1) = w(τ ) −η  H n (τ ) ( )−1 ∇E n w(τ +1) = w(τ ) −ηH n −1∇E n ࠷ٸ߱Լ๏ χϡʔτϯ๏ ४χϡʔτϯ๏ ର֯χϡʔτϯ๏ ϔοηߦྻͷஞ࣍ۙࣅʹ͍ͭͯ͸ɺ%'1
   #'(4
   -#'(4 w ڭՊॻͷྲྀΕͰ͸ͳ͍Ͱ͕͢ɺΠϝʔδ͔ͭΉͨΊʹɾɾɾ w(τ +1) = w(τ ) −η diagH n ( )−1 ∇E n ΑΓߴ଎ͳऩଋ ܭࢉྔ࡟ݮʢͨͩ͠ෆ҆ఆԽʣ ΋͏গ͠Ϛγͳۙࣅ  H n (τ ) → H n ઴ۙ 12೥12݄16೔೔༵೔
8. ର֯ۙࣅ w ϔοηߦྻ͕ର֯ߦྻͩͱ͢Ε͹ɺٯߦྻ΍ݻ༗஋ΛٻΊΔͷ ΋༰қ͘ɺඇର߲֯ͷӨڹ͕গͳ͍໰୊Ͱ͸༗ҙٛɻ࣮ࡍܭࢉ ྔ΋0 8 ͰࡁΉ a j = w ji z i i ∑ ∂2 E n ∂a j 2 = h'(a j )2 w kj w k' j ∂2 E n ∂a k ∂a k' k' ∑ k ∑ + h''(a j ) w kj ∂E n ∂a k k ∑ ∂2 E n ∂w ij 2 = ∂ ∂w ij ∂E n ∂a j z i ⎛ ⎝ ⎜ ⎞ ⎠ ⎟ = ∂2 E n ∂a j 2 z i 2 ∂2 E n ∂a j 2 = h'(a j )2 w kj ∂2 E n ∂a k 2 k ∑ + h''(a j ) w kj ∂E n ∂a k k ∑ ඇର߲͕֯ແࢹग़དྷΔͱ͢Ε͹ର߲֯ͷΈܭࢉ͢Ε͹͍͍ͷͰɹɹɹͷ ܭࢉྔͰࡁΈɺ֤੒෼ͷܭࢉ΋ҎԼͷΑ͏ʹ୯७Խ͞ΕΔ ∂E n ∂a j = h'(a j ) w kj ∂E n ∂a k k ∑ ΑΓɺϔοηߦྻͷ֤੒෼͸ ٯ఻ൖͷެࣜ ʹର͠ɺੵͷඍ෼Λߦ͑͹ O(W ) ຊདྷ͸ɺ͜Εʹؔͯ͠ϔοηߦ ྻશମΛ
   
   
   
   
   
   
   
   
   
   
   ͷܭࢉͰධՁ O(W 2 ) ͨͩ͠ɺϔοηߦྻΛඞཁͱ͢Δओཁ ͳ໰୊Ͱ͸ۃ୺ʹඇର֯Ͱ͋Δ͜ͱ͕ ଟ͘ɺద༻ʹ͸஫ҙ͕ඞཁɾɾɾ 12೥12݄16೔೔༵೔
9. ֎ੵʹΑΔۙࣅ ʢΨ΢εɾχϡʔτϯ๏ͱ΋ w ޡࠩؔ਺͕ฏํ࿨ͱͳΔΑ͏ͳ໰୊ʹ͓͍ͯɺे෼ʹ܇࿅͞Εͨঢ়ଶͰ ͸ɺϔοηߦྻͷҰ෦ͷ߲͸ແࢹग़དྷΔ͘Β͍খ͘͞ͳΓɺ0 8? ͷ ୯७ͳੵԋࢉͰධՁ͕Մೳ Η = ∇∇E = ∇y n (∇y n )T n=1 N ∑ + (y n − t n )∇∇y n n=1 N ∑ े෼ʹ܇࿅͞ΕͨωοτϫʔΫͰ͸ɺZ͕Uʹे෼ۙͮ͘͜ͱʹΑΓɺୈ߲͕̎θϩʹۙ ͖ͮɺແࢹ͢Δ͜ͱ͕ग़དྷΔͷͰҎԼͷΑ͏ʹۙࣅՄೳ Η  ∇y n (∇y n )T n=1 N ∑ ≡ b n b n T n=1 N ∑ -FWFOWFSH.BSRVBSEUۙࣅ ʢ֎ੵʹΑΔۙࣅʣ E = 1 2 (y n − t n )2 n=1 N ∑ ͷΑ͏ͳճؼ໰୊ʹର͠ɺɹɹɹɹɹɹɹɹɹɹΑΓ ∇E = (y n − t n )∇y n n=1 N ∑ 12೥12݄16೔೔༵೔
10. ϔοηٯߦྻͷஞ࣍ܭࢉ w ֎ੵʹΑΔۙࣅΛϕʔεʹͯ͠ɺ-ݸͷσʔλ఺ʹର͠ஞ࣍ ܭࢉ͠ɺ) Ћ*ͷٯߦྻΛঃʑʹݟ͚ͭΔ͜ͱ͕Մೳ Η L+1 = Η L + b L+1 b L+1 T 8PPECVSZ߃౳ࣜ Η N = b n b n T n=1 N ∑ Η L+1 −1 = Η L + b L+1 b L+1 T ( )−1 = Η L −1 + Η L −1b L+1 b L+1 T Η L −1 1+ b L+1 T Η L −1b L+1 Η 0 = b 1 b 1 T = αI (α 1) Η +αI L +1= N ʹͳΔ·Ͱ܁Γฦ͢͜ͱͰ ͷٯߦྻ͕ݟ͔ͭΔ ͱͯ͠ 12೥12݄16೔೔༵೔
11. ༗ݶ෯ͷࠩ෼ʹΑΔۙࣅ w ਺஋ਫ਼౓্ͷݶք͕͋Δ΋ͷͷɺٯ఻ൖ๏ͰٻΊΒΕͨϔοηߦ ྻͷݕূʹ༗ޮɻී௨ʹܭࢉ͢Δͱ0 8? ͷͱ͜Ζɺ̍֊ඍ෼ ͸ٯ఻ൖͷެࣜΛద༻͢Δ͜ͱͰ0 8? Ͱܭࢉ͢Δ͜ͱ΋Մೳ ∂2 E ∂w ji ∂w lk = 1 4ε2 E(w ji + ε,w lk + ε)− E(w ji + ε,w lk − ε)− E(w ji − ε,w lk + ε)+ E(w ji − ε,w lk − ε) { }+O(ε2 ) ∂2 E ∂w ji ∂w lk = ∂ ∂w lk ∂E ∂w ji = 1 2ε ∂E ∂w ji (w lk + ε)− ∂E ∂w ji (w lk − ε) ⎧ ⎨ ⎪ ⎩ ⎪ ⎫ ⎬ ⎪ ⎭ ⎪ +O(ε2 ) ∂E ∂w ji ͸ٯ఻ൖ๏Ͱޮ཰తʹٻ ΊΔ͜ͱ͕ग़དྷΔͷͰ ઁಈЏʹԠͨ͡࢒ཹޡࠩ ϔοηߦྻͷ֤ཁૉ͸ 12೥12݄16೔೔༵೔
12. ϔοηߦྻͷݫີͳධՁ w ٯ఻ൖͷςΫχοΫͷ֦ுΛར༻ͨ͠ɺ೚ҙͷϑΟʔ υϑΥϫʔυߏ଄Λ΋ͭωοτϫʔΫͷݫີͳϔοη ߦྻΛ0 8? ͰٻΊΔख๏͕ଘࡏ͢Δ Β͍͠ʣ #JTIPQ
    
    #JTIPQ
     w ̎૚ωοτϫʔΫͷ৔߹͸ɺͦΕͧΕͷॏΈͷ૊Έ߹ ΘͤͰϒϩοΫԽ͢Δ͜ͱͰ༰қʹࣜΛಋग़͢Δ͜ͱ ͕Ͱ͖ΔʢΒ͍͠ʣ 12೥12݄16೔೔༵೔
13. ϔοηߦྻͷੵͷߴ଎ͳܭࢉ w ී௨ʹϔοηߦྻΛධՁͯ͠ɺੵΛٻΊΔͱ 0 8? ɻͨͩ͠ɺϔοηߦྻࣗମͰ͸ͳ͘ɺ͋Δϕ Ϋτϧͱͷੵ͕ඞཁͳ͚ͩͰ͋Ε͹ɺޡࠩٯ఻ൖͷख ๏΋༻͍Δ͜ͱʹΑΓ0 8 ʹͰ͖Δ vTH = vT∇ ∇E ( ) Ͱ͋Δ͜ͱ͔Βɺ㲆&ΛٻΊΔͷʹඞཁͳޡࠩٯ ఻ൖʹؔΘΔશͯͷํఔࣜͷܭࢉ෦෼ʹඍ෼ԋ ࢉࢠɹɹɹΛ࡞༻ͤ͞Δɻ vT∇ 12೥12݄16೔೔༵೔