Upgrade to Pro — share decks privately, control downloads, hide ads and more …

PRML Chapter5 Hessian Matrix

hassaku
December 16, 2012

PRML Chapter5 Hessian Matrix

PRML復々習レーン #7
http://atnd.org/events/33833
での発表に用いた資料

hassaku

December 16, 2012
Tweet

More Decks by hassaku

Other Decks in Research

Transcript

  1. 13.-෮ʑशϨʔϯ
    jrj
    ϔοηߦྻ
    )T0
    ڵຯɿ// ).. ڧԽֶशͱ͔
    12೥12݄16೔೔༵೔

    View Slide

  2. ϔοηߦྻ
    )FTTJBO.BUSJY

    w ϔοηߦྻͷ໾ׂ
    w ϔοηߦྻΛٻΊΔ͜ͱʹΑΓɺޡࠩؔ਺ͷॏΈʹؔ͢
    Δ̎֊ඍ෼ΛධՁ͢Δ͜ͱ͕Մೳ
    w ϔοηߦྻͷݻ༗஋͕શͯਖ਼ʢਖ਼ఆ஋ʣͳΒۃখ఺ɺݻ
    ༗஋͕શͯෛͳΒۃେ఺ɺਖ਼ͱෛͷ྆ํͳΒҌ఺Λࣔ͢
    z = x2 + y2
    z = xe−x2 −y2
    λ
    1
    = 1
    2
    (3 + 33) λ
    2
    = 1
    2
    (3 - 33) λ
    1
    = 1.7 λ
    2
    = 0.86
    λ
    1
    = −1.7 λ
    2
    = −0.86
    12೥12݄16೔೔༵೔

    View Slide

  3. χϡʔϥϧίϯϐϡʔςΟϯά
    Ͱͷ໾ׂ
    w ඇઢܗ࠷దԽΞϧΰϦζϜʹ͓͚Δޡࠩۂ໘ͷධՁ
    w ֶशσʔλͷมಈ࣌ʹ͓͚Δ࠶ֶशͷߴ଎Խ
    w ʮמΓࠐΈʯΞϧΰϦζϜʹ͓͚Δ৑௕ͳॏΈͷಛఆ
    w ϕΠζωοτϫʔΫͷϥϓϥεۙࣅʹ͓͚Δ༧ଌ෼
    ෍ٴͼ௒ύϥϝʔλͷܾఆɺϞσϧΤϏσϯεධՁ
    ʢઅʣ
    12೥12݄16೔೔༵೔

    View Slide

  4. ͦͷଞ
    wը૾ॲཧ
    w ϔοηߦྻ͸ฏ໘ͷۂ཰Λද͢ͷͰಛ௃఺நग़ʹར༻
    w YͷϐΫηϧ΢Οϯυ΢Ͱɺࠩ෼σʔλʹجͮ͘ϔοηߦྻΛ
    ܭࢉ͠ɺͦͷݻ༗஋͕ඈͼൈ͚ͯେ͖͘ͳΔࠩ෼த৺఺͸ɺίʔ
    φʔͷՄೳੑେͳͲ
    w਺ཧܭը๏
    wߏ଄෺σβΠϯ
    12೥12݄16೔೔༵೔

    View Slide

  5. ϔοηߦྻ
    )FTTJBO.BUSJY

    ∂2 E
    ∂w
    kl
    ∂w
    mn

    ∂2 E
    ∂w
    ij
    2
    = H
    ij
    i, j ∈{1…W}
    8ύϥϝʔλͷ૯਺
    //ʹ͓͚Δϔοη
    ߦྻͷ֤ཁૉ
    H =
    ∂2 E
    ∂w
    1
    ∂w
    1

    ∂2 E
    ∂w
    1
    ∂w
    W
      
    ∂2 E
    ∂w
    W
    ∂w
    1

    ∂2 E
    ∂w
    W
    ∂w
    W
















    ॏΈ΍όΠΞεͱ͍ͬͨ
    ύϥϝʔλʹؔͯ͠Ұ༷ʹฒ΂௚͢
    12೥12݄16೔೔༵೔

    View Slide

  6. jrjͷ֓ཁ
    w ϔοηߦྻ0 8?
    ͷܭࢉྔ͕ඞཁͩ͠ɺೋ֊ඍ෼΋େม
    w ର֯ۙࣅ
    w ඇର֯ཁૉ͕ແࢹͰ͖ΔͷͰ͋Ε͹0 8
    ͰܭࢉՄೳ
    w ֎ੵʹΑΔۙࣅ
    w े෼ʹֶशࡁΈͷঢ়ଶͰ͸ɺ0 8?
    ͰܭࢉՄೳ
    w ٯߦྻͷޮ཰ͷྑ͍ஞ࣍ܭࢉ͕Մೳʢˠ४χϡʔτϯ๏ʣ
    w ༗ݶ෯ͷࠩ෼ʹΑΔۙࣅ
    w ٯ఻ൖ๏Λ༻͍ͯܭࢉ͞ΕͨϔοηߦྻͷνΣοΫʹ༗ޮ
    w Ұ෦ٯ఻ൖެࣜΛऔΓೖΕΔ͜ͱʹΑΓ0 8?
    ͰܭࢉՄೳ
    w ϔοηߦྻͷݫີͳධՁ
    w ϔοηߦྻͷੵͷߴ଎ͳܭࢉ
    12೥12݄16೔೔༵೔

    View Slide

  7. ࠷దԽΞϧΰϦζϜ
    w(τ +1) = w(τ ) −η∇E
    n
    w(τ +1) = w(τ ) −η 
    H
    n
    (τ )
    ( )−1 ∇E
    n
    w(τ +1) = w(τ ) −ηH
    n
    −1∇E
    n
    ࠷ٸ߱Լ๏
    χϡʔτϯ๏
    ४χϡʔτϯ๏
    ର֯χϡʔτϯ๏
    ϔοηߦྻͷஞ࣍ۙࣅʹ͍ͭͯ͸ɺ%'1 #'(4 -#'(4
    w ڭՊॻͷྲྀΕͰ͸ͳ͍Ͱ͕͢ɺΠϝʔδ͔ͭΉͨΊʹɾɾɾ
    w(τ +1) = w(τ ) −η diagH
    n
    ( )−1 ∇E
    n
    ΑΓߴ଎ͳऩଋ
    ܭࢉྔ࡟ݮʢͨͩ͠ෆ҆ఆԽʣ
    ΋͏গ͠Ϛγͳۙࣅ

    H
    n
    (τ ) → H
    n
    ઴ۙ
    12೥12݄16೔೔༵೔

    View Slide

  8. ର֯ۙࣅ
    w ϔοηߦྻ͕ର֯ߦྻͩͱ͢Ε͹ɺٯߦྻ΍ݻ༗஋ΛٻΊΔͷ
    ΋༰қ͘ɺඇର߲֯ͷӨڹ͕গͳ͍໰୊Ͱ͸༗ҙٛɻ࣮ࡍܭࢉ
    ྔ΋0 8
    ͰࡁΉ
    a
    j
    = w
    ji
    z
    i
    i

    ∂2 E
    n
    ∂a
    j
    2
    = h'(a
    j
    )2 w
    kj
    w
    k' j
    ∂2 E
    n
    ∂a
    k
    ∂a
    k'
    k'

    k
    ∑ + h''(a
    j
    ) w
    kj
    ∂E
    n
    ∂a
    k
    k

    ∂2 E
    n
    ∂w
    ij
    2
    =

    ∂w
    ij
    ∂E
    n
    ∂a
    j
    z
    i





    ⎟ =
    ∂2 E
    n
    ∂a
    j
    2
    z
    i
    2
    ∂2 E
    n
    ∂a
    j
    2
    = h'(a
    j
    )2 w
    kj
    ∂2 E
    n
    ∂a
    k
    2
    k
    ∑ + h''(a
    j
    ) w
    kj
    ∂E
    n
    ∂a
    k
    k

    ඇର߲͕֯ແࢹग़དྷΔͱ͢Ε͹ର߲֯ͷΈܭࢉ͢Ε͹͍͍ͷͰɹɹɹͷ
    ܭࢉྔͰࡁΈɺ֤੒෼ͷܭࢉ΋ҎԼͷΑ͏ʹ୯७Խ͞ΕΔ
    ∂E
    n
    ∂a
    j
    = h'(a
    j
    ) w
    kj
    ∂E
    n
    ∂a
    k
    k

    ΑΓɺϔοηߦྻͷ֤੒෼͸
    ٯ఻ൖͷެࣜ ʹର͠ɺੵͷඍ෼Λߦ͑͹
    O(W )
    ຊདྷ͸ɺ͜Εʹؔͯ͠ϔοηߦ
    ྻશମΛͷܭࢉͰධՁ
    O(W 2 )
    ͨͩ͠ɺϔοηߦྻΛඞཁͱ͢Δओཁ
    ͳ໰୊Ͱ͸ۃ୺ʹඇର֯Ͱ͋Δ͜ͱ͕
    ଟ͘ɺద༻ʹ͸஫ҙ͕ඞཁɾɾɾ
    12೥12݄16೔೔༵೔

    View Slide

  9. ֎ੵʹΑΔۙࣅ
    ʢΨ΢εɾχϡʔτϯ๏ͱ΋

    w ޡࠩؔ਺͕ฏํ࿨ͱͳΔΑ͏ͳ໰୊ʹ͓͍ͯɺे෼ʹ܇࿅͞Εͨঢ়ଶͰ
    ͸ɺϔοηߦྻͷҰ෦ͷ߲͸ແࢹग़དྷΔ͘Β͍খ͘͞ͳΓɺ0 8?
    ͷ
    ୯७ͳੵԋࢉͰධՁ͕Մೳ
    Η = ∇∇E
    = ∇y
    n
    (∇y
    n
    )T
    n=1
    N
    ∑ + (y
    n
    − t
    n
    )∇∇y
    n
    n=1
    N

    े෼ʹ܇࿅͞ΕͨωοτϫʔΫͰ͸ɺZ͕Uʹे෼ۙͮ͘͜ͱʹΑΓɺୈ߲͕̎θϩʹۙ
    ͖ͮɺແࢹ͢Δ͜ͱ͕ग़དྷΔͷͰҎԼͷΑ͏ʹۙࣅՄೳ
    Η  ∇y
    n
    (∇y
    n
    )T
    n=1
    N
    ∑ ≡ b
    n
    b
    n
    T
    n=1
    N
    ∑ -FWFOWFSH.BSRVBSEUۙࣅ
    ʢ֎ੵʹΑΔۙࣅʣ
    E = 1
    2
    (y
    n
    − t
    n
    )2
    n=1
    N
    ∑ ͷΑ͏ͳճؼ໰୊ʹର͠ɺɹɹɹɹɹɹɹɹɹɹΑΓ
    ∇E = (y
    n
    − t
    n
    )∇y
    n
    n=1
    N

    12೥12݄16೔೔༵೔

    View Slide

  10. ϔοηٯߦྻͷஞ࣍ܭࢉ
    w ֎ੵʹΑΔۙࣅΛϕʔεʹͯ͠ɺ-ݸͷσʔλ఺ʹର͠ஞ࣍
    ܭࢉ͠ɺ)Ћ*ͷٯߦྻΛঃʑʹݟ͚ͭΔ͜ͱ͕Մೳ
    Η
    L+1
    = Η
    L
    + b
    L+1
    b
    L+1
    T
    8PPECVSZ߃౳ࣜ
    Η
    N
    = b
    n
    b
    n
    T
    n=1
    N

    Η
    L+1
    −1 = Η
    L
    + b
    L+1
    b
    L+1
    T
    ( )−1 = Η
    L
    −1 +
    Η
    L
    −1b
    L+1
    b
    L+1
    T Η
    L
    −1
    1+ b
    L+1
    T Η
    L
    −1b
    L+1
    Η
    0
    = b
    1
    b
    1
    T = αI (α 1)
    Η +αI
    L +1= N ʹͳΔ·Ͱ܁Γฦ͢͜ͱͰ ͷٯߦྻ͕ݟ͔ͭΔ
    ͱͯ͠
    12೥12݄16೔೔༵೔

    View Slide

  11. ༗ݶ෯ͷࠩ෼ʹΑΔۙࣅ
    w ਺஋ਫ਼౓্ͷݶք͕͋Δ΋ͷͷɺٯ఻ൖ๏ͰٻΊΒΕͨϔοηߦ
    ྻͷݕূʹ༗ޮɻී௨ʹܭࢉ͢Δͱ0 8?
    ͷͱ͜Ζɺ̍֊ඍ෼
    ͸ٯ఻ൖͷެࣜΛద༻͢Δ͜ͱͰ0 8?
    Ͱܭࢉ͢Δ͜ͱ΋Մೳ
    ∂2 E
    ∂w
    ji
    ∂w
    lk
    = 1
    4ε2
    E(w
    ji
    + ε,w
    lk
    + ε)− E(w
    ji
    + ε,w
    lk
    − ε)− E(w
    ji
    − ε,w
    lk
    + ε)+ E(w
    ji
    − ε,w
    lk
    − ε)
    { }+O(ε2 )
    ∂2 E
    ∂w
    ji
    ∂w
    lk
    =

    ∂w
    lk
    ∂E
    ∂w
    ji
    = 1

    ∂E
    ∂w
    ji
    (w
    lk
    + ε)−
    ∂E
    ∂w
    ji
    (w
    lk
    − ε)










    +O(ε2 )
    ∂E
    ∂w
    ji
    ͸ٯ఻ൖ๏Ͱޮ཰తʹٻ
    ΊΔ͜ͱ͕ग़དྷΔͷͰ
    ઁಈЏʹԠͨ͡࢒ཹޡࠩ
    ϔοηߦྻͷ֤ཁૉ͸
    12೥12݄16೔೔༵೔

    View Slide

  12. ϔοηߦྻͷݫີͳධՁ
    w ٯ఻ൖͷςΫχοΫͷ֦ுΛར༻ͨ͠ɺ೚ҙͷϑΟʔ
    υϑΥϫʔυߏ଄Λ΋ͭωοτϫʔΫͷݫີͳϔοη
    ߦྻΛ0 8?
    ͰٻΊΔख๏͕ଘࡏ͢Δ Β͍͠ʣ
    #JTIPQ #JTIPQ

    w ̎૚ωοτϫʔΫͷ৔߹͸ɺͦΕͧΕͷॏΈͷ૊Έ߹
    ΘͤͰϒϩοΫԽ͢Δ͜ͱͰ༰қʹࣜΛಋग़͢Δ͜ͱ
    ͕Ͱ͖ΔʢΒ͍͠ʣ
    12೥12݄16೔೔༵೔

    View Slide

  13. ϔοηߦྻͷੵͷߴ଎ͳܭࢉ
    w ී௨ʹϔοηߦྻΛධՁͯ͠ɺੵΛٻΊΔͱ
    0 8?
    ɻͨͩ͠ɺϔοηߦྻࣗମͰ͸ͳ͘ɺ͋Δϕ
    Ϋτϧͱͷੵ͕ඞཁͳ͚ͩͰ͋Ε͹ɺޡࠩٯ఻ൖͷख
    ๏΋༻͍Δ͜ͱʹΑΓ0 8
    ʹͰ͖Δ
    vTH = vT∇ ∇E
    ( )
    Ͱ͋Δ͜ͱ͔Βɺ㲆&ΛٻΊΔͷʹඞཁͳޡࠩٯ
    ఻ൖʹؔΘΔશͯͷํఔࣜͷܭࢉ෦෼ʹඍ෼ԋ
    ࢉࢠɹɹɹΛ࡞༻ͤ͞Δɻ
    vT∇
    12೥12݄16೔೔༵೔

    View Slide