ベイズ深層学習(4.1)

Transcript

1. ϕΠζਂ૚ֶश
   
   αϯϓϦϯάʹجͮ͘ਪ࿦ख๏
   ܡɹঘً
   

   View Slide

2. αϯϓϦϯάʹجͮ͘ਪ࿦ख๏
   ˞ʮύλʔϯೝࣝͱػցֶशɹԼרʯষ΋ࢀߟ
   

   View Slide

3. αϯϓϦϯάʹجͮ͘ਪ࿦ख๏
   ɹ؍ଌσʔλΛ ɼඇ؍ଌͷม਺ͷू߹ʢFHύϥϝʔλɼજࡏม਺
   Λ ͱͨ͠ͱ
   ͖ɼϕΠζਪ࿦ʹΑΔ౷ܭղੳͰ͸ɼ֬཰Ϟσϧ Λઃܭ͢Δඞཁ͕͋Δɽ
   ɹ࣮ࡍʹɼ֬཰ϞσϧΛ༻ֶ͍ͯश΍༧ଌ͸ɼࣄޙ෼෍ Λܭࢉͯ͠ߦΘΕΔɽ
   X Z
   p(X, Z)
   p(Z|X)
   ໰୊఺
   ղܾࡦ
   ɹෳࡶͳϞσϧʢFHχϡʔϥϧωοτʣ͸ɼ ͕ղੳతʹٻΊΒΕͳ͍͜ͱ͕
   ଟ͍ɽ
   p(Z|X)
   ɹ ΛղੳతʹٻΊΔ୅ΘΓʹɼ͜ͷ෼෍͔Βෳ਺ͷαϯϓϧΛಘΔ͜ͱͰɼ෼෍
   ͷಛੑΛௐ΂Δɽ
   ɹͱ͍͏͜ͱ͔ΒɼαϯϓϦϯά͢Δํ๏Λࠓճ͸ษڧ͢ΔΑʂ
   p(Z|X)
   

   View Slide

4. ຊ೔ͷ಺༰
   ୯७ϞϯςΧϧϩ๏
   غ٫αϯϓϦϯά
   ࣗݾਖ਼نԽॏ఺αϯϓϦϯά
   ʢϚϧίϑ࿈࠯ϞϯςΧϧϩ๏ʣ
   ϝτϩϙϦεɾϔΠεςΟϯάε๏ ϝτϩϙϦε๏
   ϋϛϧτχΞϯϞϯςΧϧϩ๏ ϥϯδϡόϯϞϯςΧϧϩ๏
   ΪϒεαϯϓϦϯά
   

   View Slide

5. ຊ೔ͷ಺༰
   ୯७ϞϯςΧϧϩ๏
   غ٫αϯϓϦϯά
   ࣗݾਖ਼نԽॏ৺αϯϓϦϯά
   ʢϚϧίϑ࿈࠯ϞϯςΧϧϩ๏ʣ
   ϝτϩϙϦεɾϔΠεςΟϯάε๏ ϝτϩϙϦε๏
   ϋϛϧτχΞϯϞϯςΧϧϩ๏ ϥϯδϡόϯϞϯςΧϧϩ๏
   ΪϒεαϯϓϦϯά
   .$.$
   

   View Slide

6. ୯७ϞϯςΧϧϩ๏
   ɹ෼෍ ʹର͢Δؔ਺ ͷظ଴஋ɹ ΛٻΊ͍ͨɽɹ
   p(z) f(z) p(z)
   [ f(z)] =
   ∫
   f(z)p(z)dz
   ໨త
   ঢ়گ
   ɹظ଴஋ ͷղੳతͳੵ෼ܭࢉ͕ࠔ೉ɽ
   ɹ෼෍ ͔ΒͷαϯϓϦϯά͸༰қɽ
   ∫
   f(z)p(z)dz
   p(z)
   ख๏
   ɹ Λे෼େ͖ͳ஋ͱͨ͠ͱ͖ɼ
   
   
   T
   z(1), z(2), …, z(T) ∼ p(z)
   ∫
   f(z)p(z)dz ≈
   1
   T
   T
   ∑
   t=1
   f(z(t))
   ͔Β ݸαϯϓϦϯά
   ⟵ p(z) T
   

   View Slide

7. ɹύϥϝʔλ Λ࣋ͭϞσϧ ͷपล໬౓ Λܭࢉ͢Δࡍʹ࢖༻͢
   Δ৔߹ɼ
   
   θ p(X, θ) = p(X|θ)p(θ) p(X)
   p(X) =
   ∫
   p(X|θ)p(θ)dθ =
   ∫
   N
   ∏
   n=1
   p(xn
   |θ)p(θ)dθ
   = p(θ)
   [p(X|θ)]
   ≈
   1
   T
   T
   ∑
   t=1
   N
   ∏
   n=1
   p(xn
   |θ(t)), (θ(1), …, θ(T) ∼ p(θ))
   ɹظ଴஋ ʹ͓͍ͯɼ ͔Βͷαϯϓϧ ͷൣғ͸෯޿͘ͱΔඞཁ͕͋ΓɼҰํ
   Ͱɼ ͸ڱ͍ ͷൣғͰ͔͠େ͖ͳ஋ΛऔΒͳ͍έʔε͕ଟ͍ɽ
   ɹ ൚༻త͚ͩͲɼܭࢉޮ཰͕ѱ͍ɽ
   p(z)
   [ f(z)] p(z) z
   f(z) z
   ⟹
   ୯७ϞϯςΧϧϩ๏
   ໰୊఺
   

   View Slide

8. غ٫αϯϓϦϯά
   ɹີ౓ܭࢉ͕ࠔ೉ͳ֬཰෼෍ ͔ΒαϯϓϧΛಘΔɽɹ
   p(z)
   z(1), z(2), … ∼ p(z)
   ໨త
   ঢ়گ
   ɹਖ਼نԽ͞Ε͍ͯͳ͍ؔ਺ ͸ܭࢉՄೳɽͭ·Γɼ ɽ
   ˜
   p(z)( = Zp
   ⋅ p(z))
   ∫
   ˜
   p(z)dz ≠ 1
   ख๏
   ɹఏҊ෼෍ Λઃఆ͢Δɽ೚ҙͷ ʹରͯ͠ɼ
   ɹɹɹɹɹɹɹɹɹɹɹɹɹɹɹɹɹ
   ͱͳΔΑ͏ʹɼਖ਼ͷఆ਺ ΛఆΊΔɽ
   q(z) z
   kq(z) > ˜
   p(z)
   k
   ఏҊ෼෍
   ɹαϯϓϦϯά͕؆୯ʹߦ͑ΔΑ͏ͳ
   Ծͷ෼෍ɽ
   

   View Slide

9. غ٫αϯϓϦϯά
   ख๏ ͖ͭͮʣ
   ɹఏҊ෼෍ ͔ΒαϯϓϧΛಘΔɽ
   
   ɹҰ༷෼෍ ͔ΒͷαϯϓϧΛಘΔɽ
   
   ɹαϯϓϧ ͷड༰ʢBDDFQUʣغ٫ʢSFKFDUʣબ୒ɽ
   ɹɹɹ
   q(z)
   z(t) ∼ q(z)
   Uni(0,kq(z))
   ˜
   u ∼ Uni (0,kq(z(t)))
   z(t)
   if ˜
   u > ˜
   p(z(t)) then SFKFDU else BDDFQU
   ड༰཰
   ∫
   q(z)
   ˜
   p(z)
   kq(z)
   dz =
   1
   k ∫
   ˜
   p(z)dz
   ໰୊఺
   ߴ࣍ݩͷม਺ͷαϯϓϦϯά͕ඞཁͳ৔߹ɼड༰཰͕ඇৗʹ௿͘ͳΔɽ
   

   View Slide

10. غ٫αϯϓϦϯά
    z
    ˜
    p(z)
    ͷαϯϓϧΛغ٫αϯϓϧϦϯάͰ֫ಘ͢Δɽ
    ͸ະ஌ɽ ͸ط஌ɽ
    p(z)
    p(z) ˜
    p(z)
    p(z)
    

    View Slide

11. غ٫αϯϓϦϯά
    z
    ˜
    p(z)
    αϯϓϧ͕༰қͳఏҊ෼෍ Λઃఆɽ
    p(z)
    q(z)
    

    View Slide

12. غ٫αϯϓϦϯά
    z
    ˜
    p(z)
    kq(z)
    Λ෴͍͔Ϳ͞ΔΑ͏ʹ Λઃఆɽ
    ˜
    p(z) k
    kq(z) > ˜
    p(z)
    q(z)
    × k
    

    View Slide

13. غ٫αϯϓϦϯά
    z
    ˜
    p(z)
    kq(z)
    z(t)
    ఏҊ෼෍ ͔ΒαϯϓϧΛಘΔɽ
    q(z)
    z(t) ∼ q(z)
    q(z)
    

    View Slide

14. غ٫αϯϓϦϯά
    z
    ˜
    p(z)
    kq(z)
    z(t)
    kq(z(t))
    ˜
    u
    Ұ༷෼෍ ͔ΒͷαϯϓϧΛಘΔ
    Uni(0,kq(z))
    ˜
    u ∼ Uni (0,kq(z(t)))
    

    View Slide

15. غ٫αϯϓϦϯά
    z
    ˜
    p(z)
    kq(z)
    z(t)
    kq(z(t))
    ड༰
    غ٫
    ˜
    u
    αϯϓϧ ͷड༰ʢBDDFQUʣغ٫ʢSFKFDUʣબ୒ɽ
    z(t)
    if ˜
    u > ˜
    p(z(t)) then SFKFDU else BDDFQU
    ˜
    p(z(t))
    

    View Slide

16. ࣗݾਖ਼نԽॏ৺αϯϓϦϯά
    ɹ෼෍ ʹର͢Δؔ਺ ͷظ଴஋ɹ Λ୯७ϞϯςΧϧϩ๏Α
    Γ΋ޮ཰తʹٻΊ͍ͨɽɹ
    p(z) f(z) p(z)
    [ f(z)] =
    ∫
    f(z)p(z)dz
    ໨త
    ঢ়گ
    ɹظ଴஋ ͷղੳతͳੵ෼ܭࢉ͕ࠔ೉ɽ
    ɹ෼෍ ͔Β௚઀αϯϓϦϯάΛಘΒΕͳ͍ɽ
    ɹਖ਼نԽ͞Ε͍ͯͳ͍ؔ਺ ͸ܭࢉՄೳɽ
    ∫
    f(z)p(z)dz
    p(z)
    ˜
    p(z)( = Zp
    ⋅ p(z))
    എܠ
    ɹغ٫αϯϓϦϯάΛ༻͍ͯɼ Λ࢖ΘͣʹαϯϓϧΛऔಘ͠ɼظ଴஋ Λٻ
    ΊΔ͜ͱ΋Ͱ͖Δ͕ɼ ͷ஋͕খ͞ͳྖҬʹαϯϓϧ͕ूத͢ΔՄೳੑ͕͋Δɽ
    ୯७ϞϯςΧϧϩ๏ͷܭࢉ΁ͷد༩͕গͳ͍ɽ ͷ஋͕େ͖͘ͳΔΑ͏ͳ
    ྖҬΛॏ఺తʹαϯϓϧͨ͠ํ͕ޮ཰͕͍͍ɽ
    p(z) p(z)
    [ f(z)]
    f(z)
    ⟹ f(z)p(z)
    

    View Slide

17. ࣗݾਖ਼نԽॏ৺αϯϓϦϯά
    ɹ·ͣɼఏҊ෼෍ Λઃఆ͢Δɽਖ਼نԽ͞Ε͍ͯͳ͍ؔ਺ ͸ɼҎԼͷΑ͏ʹද
    ͞ΕΔɽ
    
    ɹظ଴஋ͷܭࢉ͸ɼҎԼͷΑ͏ʹมܗͰ͖Δɽ
    
    q(z) ˜
    p(z), ˜
    q(z)
    p(z) =
    1
    Zp
    ˜
    p(z), q(z) =
    1
    Zq
    ˜
    q(z)
    ∫
    f(z)p(z)dz =
    ∫
    f(z)
    p(z)
    q(z)
    q(z)dz = q(z)
    [ f(z)
    p(z)
    q(z)
    ]
    =
    ∫
    f(z)
    1
    Zp
    ˜
    p(z)
    1
    Zq
    ˜
    q(z)
    q(z)dz =
    Zq
    Zp
    q(z)
    [ f(z)
    ˜
    p(z)
    ˜
    q(z)
    ]
    ≈
    Zq
    Zp
    1
    T
    T
    ∑
    t=1
    f(z(t))
    ˜
    p(z(t))
    ˜
    q(z(t))
    =
    Zq
    Zp
    1
    T
    T
    ∑
    t=1
    f(z(t))w(t),
    w(t) =
    ˜
    p(z(t))
    ˜
    q(z(t))
    ख๏
    

    View Slide

18. ࣗݾਖ਼نԽॏ৺αϯϓϦϯά
    ɹ ͸ະ஌ͷ஋ͳͷͰɼҎԼͷΑ͏ʹਖ਼نԽ߲ͷൺΛۙࣅ͢Δɽ
    
    
    ɹΑͬͯɼؔ਺ ͷظ଴஋͕ۙࣅతʹಘΒΕΔɽ
    Zp
    Zp
    Zq
    =
    ∫
    ˜
    p(z)
    Zq
    dz
    =
    ∫
    ˜
    p(z)
    ˜
    q(z)
    q(z)dz
    = q(z) [
    ˜
    p(z)
    ˜
    q(z) ]
    ≈
    1
    T
    T
    ∑
    t=1
    w(t),
    z(1), …, z(T) ∼ q(z)
    f(z)
    ख๏ ͖ͭͮʣ
    (
    ∵ p(z) =
    1
    Zp
    ˜
    p(z)
    )
    (
    ∵
    ∫
    p(z)dz =
    1
    Zp
    ∫
    ˜
    p(z)dz = 1
    )
    

    View Slide

19. ࣗݾਖ਼نԽॏ৺αϯϓϦϯά
    ɹ ͸ະ஌ͷ஋ͳͷͰɼҎԼͷΑ͏ʹਖ਼نԽ߲ͷൺΛۙࣅ͢Δɽ
    
    
    ɹΑͬͯɼؔ਺ ͷظ଴஋͕ۙࣅతʹಘΒΕΔɽ
    Zp
    Zp
    Zq
    =
    ∫
    ˜
    p(z)
    Zq
    dz
    =
    ∫
    ˜
    p(z)
    ˜
    q(z)
    q(z)dz
    = q(z) [
    ˜
    p(z)
    ˜
    q(z) ]
    ≈
    1
    T
    T
    ∑
    t=1
    w(t),
    z(1), …, z(T) ∼ q(z)
    f(z)
    ख๏ ͖ͭͮʣ
    (
    ∵ p(z) =
    1
    Zp
    ˜
    p(z)
    )
    (
    ∵
    ∫
    p(z)dz =
    1
    Zp
    ∫
    ˜
    p(z)dz = 1
    )
    ͳΜͷͨΊʹ Λ
    ͖࣋ͬͯͨΜͩΖ͏ʜ
    ˜
    q(z)
    

    View Slide

20. ࣗݾਖ਼نԽॏ఺αϯϓϦϯάʢޡهͷՄೳੑʣ
    ʮϕΠζਂ૚ֶशʯQ
    ɹࣗݾਖ਼نԽॏ఺αϯϓϦϯάͷஈམ໨ͷ࠷ॳ
    ޡΓ
    ɹغ٫αϯϓϦϯάͱҟͳΔ఺͸ɼ ͔Βͷαϯϓϧʜʜ
    p(z)
    ɹ୯७ϞϯςΧϧϩ๏ͱҟͳΔ఺͸ɼ ͔Βͷαϯϓϧʜʜ
    p(z)
    

    View Slide

21. Ϛϧίϑ࿈࠯ϞϯςΧϧϩ๏ʢ.$.$ʣ
    ɹغ٫αϯϓϦϯάͷ໰୊఺͸ɼߴ࣍ݩʹͳΔͱड༰཰͕ඇৗʹখ͘͞ͳΔ͜ͱɽ࣮ࡍ
    ʹ͸ɼ࣍ݩఔ౓ͷ؆୯ͳੵ෼ۙࣅʹ͔͠ద༻Ͱ͖ͳ͍ɽ
    ɹͰ͸ɼߴ࣍ݩۭؒͰޮ཰తʹαϯϓϦϯά͢Δʹ͸ʜʜ
    ɹɹɹ Ϛϧίϑ࿈࠯ϞϯςΧϧϩ๏ʢ.$.$ʣ͕ఏҊ͞Ε͍ͯΔɽ
    ⟹
    ࣍Ϛϧίϑ࿈࠯
    ɹ֬཰ม਺ͷܥྻ ʹରͯ͠
    
    ͕੒Γཱͭͱ͖ͷܥྻ ͷ͜ͱɽ
    z(1), z(2), …
    p(z(t) |z(1), z(2), …, z(t−1)) = p(z(t) |z(t−1))
    z(1), z(2), …
    άϥϑΟΧϧϞσϧ
    z(1) z(2) z(t−1) z(t)
    ⋯
    ɹભҠ֬཰ɹΛ ͱ͓͍ͨͱ͖ɼ
    
    ͕੒Γཱͭͱ͖ɼ Λɹఆৗ෼෍ɹͱ͍͏ɽ
    (z(t−1), z(t)) = p(z(t) |z(t−1))
    p*
    (z) =
    ∫
    (z′ , z)p*
    (z′ )dz′
    p*
    (z)
    

    View Slide

22. Ϛϧίϑ࿈࠯ϞϯςΧϧϩ๏ʢ.$.$ʣ
    ɹఆৗ෼෍ ʹऩଋ͢ΔΑ͏ͳભҠ֬཰ Λઃܭ͢Δͱɼ ͔Βαϯϓϧ
    ΛಘΔ͜ͱ͕Ͱ͖Δɽ ఆৗ෼෍ʹີ౓ܭࢉ͕ࠔ೉ͳ֬཰෼෍Λ͓͘ɽ
    p*
    (z) (z(t−1), z(t)) p*
    (z)
    ⟹
    ख๏ͷΩϞ
    ৄࡉ௼Γ߹͍৚݅
    p*
    (z)(z, z′ ) = p*
    (z′ )(z′ , z)
    ʲे෼৚݅ʳৄࡉ௼Γ߹͍৚͕݅੒Γཱͭ ͸ఆৗ෼෍ͱͳΔɽ
    ⟹ p*
    (z)
    p*
    (z)(z, z′ ) = p*
    (z′ )(z′ , z)
    ⇒
    ∫
    p*
    (z)(z, z′ )dz′ =
    ∫
    p*
    (z′ )(z′ , z)dz′
    ⇔ p*
    (z)
    ∫
    p(z′ |z)dz′ =
    ∫
    p*
    (z′ )(z′ , z)dz′
    ⇔ p*
    (z) =
    ∫
    p*
    (z′ )(z′ , z)dz′
    

    View Slide

23. Ϛϧίϑ࿈࠯ϞϯςΧϧϩ๏ʢ.$.$ʣ
    ɹ௼Γ߹͍৚݅ʹՃ͑ͯɼαϯϓϧ਺͕ ͱͨ͠ͱ͖ɼॳظঢ়ଶ ʹ͔͔ΘΒ
    ͣɼ ͕ఆৗ෼෍ ʹऩଋ͢Δඞཁ͕͋Δɽ Τϧΰʔυੑ
    t → ∞ p(z(1))
    p(z(t)) p*
    (z) ⟹
    Τϧΰʔυੑ
    w ط໿ੑɹɿ೚ҙͷঢ়ଶ͔Β೚ҙͷঢ়ଶ΁༗ݶճ਺ͰભҠՄೳɽ
    w ඇपظੑɿ͢΂ͯͷঢ়ଶ͕ݻఆͷपظੑΛ΋ͨͳ͍ɽ
    w ਖ਼࠶ؼੑɿಉ͡ঢ়ଶ͕༗ݶճͰ໭Δ͜ͱ͕Մೳɽ
    

    View Slide

24. ϝτϩϙϦεɾϔΠεςΟϯάε๏ʢ.)๏ʣ
    ໨త
    ɹະ஌ͷ֬཰෼෍ ͔ΒαϯϓϦϯάΛಘΔɽ
    p(z)
    લఏ
    ɹ ͱͳΔਖ਼نԽ͞Ε͍ͯͳ͍ؔ਺ ͸ط஌Ͱ͋Δɽ
    p(z) ∝ ˜
    p(z) ˜
    p(z)
    ख๏
    ɹભҠ֬཰ ͕௚઀ઃܭ͕೉͍͠৔߹͸ɼભҠͷఏҊ෼෍ Λ࢖͑Δɽ
    (z′ , z) q(z|z′ )
    ɽఏҊ෼෍ ͔Β࣍ͷαϯϓϧ఺ͷީิ ΛαϯϓϦϯά͢Δɽ
    ɽ࣍ͷൺ཰ Λܭࢉ͢Δɽ
    
    ɽ Λ֬཰ ʹΑͬͯ ͱͯ͠ड༰͠ɼͦ͏Ͱͳ͍৔߹
    ͸ɼ ͱ͢Δɽ
    q( ⋅ |z(t)) z*
    r
    r =
    ˜
    p(z*
    )q(z(t) |z*
    )
    ˜
    p(z(t))q(z*
    |z(t))
    z*
    min(1,r) z(t+1) ⟵ z*
    z(t+1) ⟵ z(t)
    ΞϧΰϦζϜͷྲྀΕ
    

    View Slide

25. ϝτϩϙϦεɾϔΠεςΟϯάε๏ʢ.)๏ʣ
    ৄࡉ௼Γ߹͍৚݅ͷূ໌
    ɹભҠ֬཰͸ɼҎԼͷΑ͏ʹͳΔɽ
    (z, z′ ) = q(z′ |z) min (1,
    ˜
    p(z′ )q(z|z′ )
    ˜
    p(z)q(z′ |z) )
    p(z)(z, z′ ) = p(z)q(z′ |z) min (1,
    ˜
    p(z′ )q(z|z′ )
    ˜
    p(z)q(z′ |z) )
    = p(z)q(z′ |z) min (1,
    p(z′ )q(z|z′ )
    p(z)q(z′ |z) )
    = min (p(z)q(z′ |z), p(z′ )q(z|z′ ))
    = min (p(z′ )q(z|z′ ), p(z)q(z′ |z))
    = p(z′ )q(z|z′ ) min (1,
    p(z)q(z′ |z)
    p(z′ )q(z|z′ ))
    = p(z′ )q(z|z′ ) min (1,
    ˜
    p(z)q(z′ |z)
    ˜
    p(z′ )q(z|z′ ))
    = p(z′ )(z′ , z)
    ɹ ͷ৔߹͸ɼϝτϩϙϦε๏ͱݺ͹ΕΔɽ
    q(z′ |z) = q(z|z′ )
    

    View Slide

26. ϝτϩϙϦεɾϔΠεςΟϯάε๏ʢ.)๏ʣ
    ʮϕΠζਂ૚ֶशʯQࣜʢʣ
    ޡΓ
    (z, z′ ) = q(z′ |z) min (1,
    ˜
    p(z′ )q(z|z′ )
    ˜
    p(z)q(z′ |z) )
    (z, z′ ) = q(z|z′ ) min (1,
    ˜
    p(z′ )q(z|z′ )
    ˜
    p(z)q(z′ |z) )
    

    View Slide

27. ϝτϩϙϦεɾϔΠεςΟϯάε๏ʢ.)๏ʣ
    ۩ମྫͰֶͿ
    ໨ඪ෼෍ʢະ஌ʣ
    ɹɹɹɹɹɹɹ
    ਖ਼نԽ͞Ε͍ͯͳ͍ؔ਺ʢط஌ʣ
    
    ఏҊ෼෍
    ͭ·Γ
    
    p(z) = (z|μ, Σ) =
    1
    (2π)D |Σ|
    exp {−
    1
    2
    (z − μ)TΣ−1(z − μ)}
    ˜
    p(z) = exp {−
    1
    2
    (z − μ)TΣ−1(z − μ)}
    q(z*
    |z) = (z′ |z, I) z*
    ∼ (z, I)
    https://drive.google.com/open?id=1vcBZWp9HPzfCzBjj2INdkH_CJUDL-JA3
    

    View Slide

28. ɹαϯϓϦϯάͷલʹϋϛϧτχΞϯΛར༻ͨ͠ղੳֶతͳ਺஋γϛϡϨʔγϣϯΛղ
    આɽຎࡲʹΑΔΤωϧΪʔͷݮগ͕ͳ͍ͱԾఆ͢ΔͱɼϋϛϧτχΞϯ͸ҎԼͷΑ͏ʹ
    ද͞ΕΔɽ
    ℋ(z, p) = (z) + (p),
    (p) =
    1
    2m
    pTp,
    z ∈ ℝD: ෺ମͷҐஔϕΫτϧ,
    p ∈ ℝD: ෺ମͷӡಈྔϕΫτϧ,
    m ∈ ℝ: ෺ମͷ࣭ྔ,
    ℋ(z, p): ϋϛϧτχΞϯ,
    (z): ϙςϯγϟϧΤωϧΪʔ,
    (p): ӡಈΤωϧΪʔ .
    ϋϛϧτχΞϯϞϯςΧϧϩ๏ʢ).$๏ʣ
    ).$๏ʢϋΠϒϦοτϞϯςΧϧϩ๏ʣʹ෺ମͷيಓͷझຯϨʔγϣϯ .)๏
    ×
    ɹ).$๏͸ɼϥϯμϜ΢ΥʔΫతͳ.)ͱൺ΂ͯɼޮ཰తʹۭؒΛ୳ࡧՄೳɽ
    ϋϛϧτχΞϯͷγϛϡϨʔγϣϯ
    

    View Slide

29. ϋϛϧτχΞϯϞϯςΧϧϩ๏ʢ).$๏ʣ
    ɹ࣭ྔΛͱ͠ɼ ͱ ͷ࣌ؒ ʹؔ͢Δڍಈ͸ɼϋϛϧτχΞϯͷภඍ෼ʹΑܾͬͯఆɽ
    
    ͜ͷඍ෼ํఔ͕ࣜղੳతʹղ͚ͳ͍΋ͷͱ͠ɼ਺஋γϛϡϨʔγϣϯʹΑͬͯيಓΛܭ
    ࢉ͢Δɽ
    z p τ
    dpi
    dτ
    = −
    dℋ
    dzi
    = −
    d
    dzi
    ,
    dzi
    dτ
    =
    dℋ
    dpi
    =
    d
    dpi
    .
    ΦΠϥʔ๏
    ࣌ࠁ ઌͷڍಈΛۙࣅతʹ༧ଌɽ
    ϵ > 0
    pi
    (τ + ϵ) = pi
    (τ) + ϵ
    dpi
    dτ
    τ
    = pi
    (τ) − ϵ
    d
    dzi
    zi
    (τ)
    ,
    zi
    (τ + ϵ) = zi
    (τ) + ϵ
    dzi
    dτ
    τ
    = zi
    (τ) + ϵpi
    (τ)
    ໰୊఺
    ཭ࢄԽʹΑΔ਺஋ޡ͕ࠩେ͖͍ɽ Ϧʔϓϑϩοά๏
    ⟹
    

    View Slide

30. ϋϛϧτχΞϯϞϯςΧϧϩ๏ʢ).$๏ʣ
    Ϧʔϓϑϩοά๏
    pi (τ +
    ϵ
    2 ) = pi
    (τ) −
    ϵ
    2
    d
    dzi
    zi
    (τ)
    ,
    zi
    (τ + ϵ) = zi
    (τ) + ϵpi (τ +
    ϵ
    2 ),
    pi
    (τ + ϵ) = pi (τ +
    ϵ
    2 ) −
    ϵ
    2
    d
    dzi
    zi
    (τ + ϵ)
    .
    ͜ΕΛ ճ܁Γฦ͢͜ͱͰ࣌ࠁ ઌͷ෺ମͷҐஔ ͱӡಈྔ ΛۙࣅతʹܭࢉͰ͖Δɽ
    L ϵL z*
    p*
    ϋϛϧτχΞϯͷੑ࣭
    ɽ ͸࣌ؒ ʹΑͬͯෆมɽ
    ɽՄٯੑɿ ͔Β ͷભҠ͸ҰରҰɽ
    ɽମੵอଘ
    ℋ τ
    (z, p) (z*
    , p*
    )
    

    View Slide

31. ϋϛϧτχΞϯϞϯςΧϧϩ๏ʢ).$๏ʣ
    αϯϓϦϯάΞϧΰϦζϜ΁ͷద༻
    ໨త
    ɹະ஌ͷ֬཰෼෍ ͔ΒαϯϓϦϯάΛಘΔɽ
    p(z)
    લఏ
    ɹ ͱͳΔਖ਼نԽ͞Ε͍ͯͳ͍ؔ਺ ͸ط஌Ͱ͋Δɽ
    ɹ ͱ֦ு͢Δͱɼ ͸पล෼෍ ͔Βαϯϓϧ͕ಘΒΕΔɽ
    ɹ
    
    p(z) ∝ ˜
    p(z) ˜
    p(z)
    p(z, p) = p(z)p(p) z p(z)
    p(p) = (p|0, I)
    (z) = − log (˜
    p(z))
    (p) =
    1
    2
    pTp
    

    View Slide

32. ϋϛϧτχΞϯϞϯςΧϧϩ๏ʢ).$๏ʣ
    ɹಉ࣌෼෍Λܭࢉ͢ΔͱɼҎԼͷΑ͏ʹͳΔɽ
    
    ϝτϩϙϦε๏Ͱ࢖ΘΕΔൺ཰ ͸ɼҎԼͷΑ͏ʹͳΔɽ
    p(z, p) = p(z)p(p)
    = exp (log p(z) + log p(p))
    ∝ exp (log ˜
    p(z) −
    1
    2
    pTIp
    )
    = exp (−(z) − (p))
    = exp (−ℋ(z, p))
    r
    r =
    p(z*
    , p*
    )
    p(z, p)
    = exp (−ℋ(z*
    , p*
    ) + ℋ(z, p))
    

    View Slide

33. ϋϛϧτχΞϯϞϯςΧϧϩ๏ʢ).$๏ʣ
    ख๏
    ɽӡಈྔΛαϯϓϦϯά
    
    ɽϦʔϓϑϩοά๏Ͱݱࡏͷ఺ ͔Βީิ఺ ΛಘΔɽ
    ɽ࣍ͷൺ཰ Λܭࢉ͢Δɽ
    
    ɽ Λ֬཰ ʹΑͬͯ ͱͯ͠ड༰͠ɼͦ͏Ͱͳ͍৔߹
    ͸ɼ ͱ͢Δɽ
    p ∼ (0, I)
    (z(t), p) (z*
    , p*
    )
    r
    r =
    p(z*
    , p*
    )
    p(z, p)
    z*
    min(1,r) z(t+1) ⟵ z*
    z(t+1) ⟵ z(t)
    ΞϧΰϦζϜͷྲྀΕ
    

    View Slide

34. ϋϛϧτχΞϯϞϯςΧϧϩ๏ʢ).$๏ʣ
    Ϧʔϓϑϩοάͷύϥϝʔλʹؔͯ͠ɼҎԼͷΑ͏ͳτϨʔυΦϑ͕͋Δɽ
    ɹ).$๏͸ɼࣄޙ෼෍ͷඍ෼͑͞ܭࢉͰ͖Ε͹ద༻Ͱ͖ɼඇৗʹ൚༻తɽҰൠతͳ
    χϡʔϥϧωοτϫʔΫ͸࿈ଓͳજࡏม਺ͷΈͰ੒Γཱ͍ͬͯΔ͜ͱ͕ଟ͍ͷͰɼ).$
    ๏͸χϡʔϥϧωοτϫʔΫͷϕΠζֶशʹ࢖ΘΕ͖ͯͨɽ
    େ͖͍
    εςοϓαΠζ
    ϵ
    εςοϓ਺
    L
    খ͍͞ খ͍͞
    େ͖͍
    ड༰཰ ड༰཰
    ୳ࡧޮ཰ ܭࢉྔ
    ߴ͍ ߴ͍
    ௿͍ ௿͍
    େ͖͍ খ͍͞
    ߴ͍
    ௿͍
    

    View Slide

35. ϋϛϧτχΞϯϞϯςΧϧϩ๏ʢ).$๏ʣ
    ϥϯδϡόϯಈྗֶ๏
    ɹ ͱͨ͠৔߹ɼϥϯδϡόϯϞϯςΧϧϩ๏ɹ·ͨ͸ɹϥϯδϡόϯಈྗֶ๏ɹͱ
    ݺ͹ΕΔɽ
    
    ɹਂ૚ֶश޲͚ʹϛχόονֶश͕ߦ͑ΔΑ͏ʹͨ͠ɹ֬཰తޯ഑ϥϯδϡόϯಈྗֶ
    ๏ɹʹల։͞ΕΔɽ
    L = 1
    z*i
    = zi
    (τ + ϵ)
    = zi
    (τ) + ϵ pi
    (τ) −
    ϵ
    2
    d
    dzi
    zi
    (τ)
    = zi
    (τ) −
    ϵ2
    2
    d
    dzi
    zi
    (τ)
    + ϵpi
    (τ)
    

    View Slide

36. ໨ඪ෼෍ʢະ஌ʣ
    ɹɹɹɹɹɹɹ
    ਖ਼نԽ͞Ε͍ͯͳ͍ؔ਺ʢط஌ʣ
    
    ӡಈྔͷαϯϓϦϯάɿ
    ӡಈΤωϧΪʔɿ
    ҐஔΤωϧΪʔɿ
    ҐஔΤωϧΪʔͷภඍ෼ɿ
    p(z) = (z|μ, Σ) =
    1
    (2π)D |Σ|
    exp {−
    1
    2
    (z − μ)TΣ−1(z − μ)}
    ˜
    p(z) = exp {−
    1
    2
    (z − μ)TΣ−1(z − μ)}
    p ∼ (0, I)
    (p) =
    1
    2
    pTp
    (z) = − log (˜
    p(z)) = −
    1
    2
    (z − μ)TΣ−1(z − μ)
    ∂
    ∂z
    = − (z − μ)TΣ−1
    ϋϛϧτχΞϯϞϯςΧϧϩ๏ʢ).$๏ʣ
    ۩ମྫͰֶͿ
    https://drive.google.com/open?id=11zWctTbECXEhlHm7AqPiXC_MErAYl7hJ
    

    View Slide

37. ΪϒεαϯϓϦϯά
    ໨త
    ɹ֬཰෼෍ ͔Β௚઀ શମΛαϯϓϦϯά͢Δ͜ͱ͕೉͍͠ͱ͖ͷ୳ࡧɽ
    p(Z) Z
    લఏ
    ɹ֬཰෼෍ ͸ط஌ɽ
    p(Z)
    ख๏
    ɹɽม਺ Λ ݸͷ෦෼ू߹ʹ෼͚Δɽ
    
    ɹɽ෦෼ू߹Λஞ࣍తʹ୳ࡧ͢Δɽ
    ɹ
    Z M
    Z = {Z1
    , Z2
    , …, ZM
    }
    Z1
    ∼ p(Z1
    |Z2
    , Z3
    , …, ZM−1
    , ZM
    )
    Z2
    ∼ p(Z2
    |Z1
    , Z3
    , …, ZM−1
    , ZM
    )
    ⋮
    ZM
    ∼ p(ZM
    |Z1
    , Z2
    , …, ZM−2
    , ZM−1
    )
    

    View Slide

38. ΪϒεαϯϓϦϯά
    ɹΪϒεαϯϓϦϯάͷଥ౰ੑ͸ɼαϯϓϦϯάͷखଓ͖͕.)๏ͷҰछͱͯ͠ղऍͰ
    ͖Δ͜ͱ͕อূ͞Ε͍ͯΔɽ ͷΑ͏ʹ෼͚ɼ Λ৚݅෇͚ͨ͠΋ͱͰ ͷα
    ϯϓϦϯάΛ͢Δ͜ͱΛߟ͑ͨ৔߹ɼ ͔ͭ
    ɹൺ཰ Λܭࢉ͢ΔͱҎԼͷΑ͏ʹͳΔɽ
    
    Αͬͯɼશͯड༰͞ΕΔɽ ΋ಉ༷ɽ
    Z = {Z1
    , Z2
    } Z2
    Z1
    q(Z*
    |Z) = p(Z1*
    |Z2*
    ) Z2
    = Z2*
    r
    r =
    p(Z*
    )q(Z|Z*
    )
    p(Z)q(Z*
    |Z)
    =
    p(Z1*
    , Z2*
    )p(Z1
    |Z2*
    )
    p(Z1
    , Z2
    )p(Z1*
    |Z2
    )
    =
    p(Z1*
    |Z2*
    )p(Z2*
    )p(Z1
    |Z2*
    )
    p(Z1
    |Z2
    )p(Z2
    )p(Z1*
    |Z2
    )
    = 1
    Z2
    

    View Slide

39. ΪϒεαϯϓϦϯά
    ۩ମྫͰֶͿ
    ໨ඪ෼෍ʢط஌ʣ
    ɹɹɹɹ
    ͱ͢Δɽ
    
    p(z) = (z|μ, Σ) =
    1
    (2π)D |Σ|
    exp {−
    1
    2
    (z − μ)TΣ−1(z − μ)}
    z = (
    z1
    z2
    ), μ = (
    μ1
    μ2
    ), Σ = (
    Σ11
    Σ12
    Σ21
    Σ22
    ), Λ = Σ−1 = (
    Λ11
    Λ12
    Λ21
    Λ22
    )
    log p(z) = log p(z1
    , z2
    )
    =
    − 1
    2
    (z1
    − μ1
    )TΛ11
    (z1
    − μ1
    ) + (z1
    − μ1
    )TΛ12
    (z2
    − μ2
    ))
    − 1
    2
    (z2
    − μ2
    )TΛ22
    (z2
    − μ2
    ) + (z2
    − μ2
    )TΛ21
    (z1
    − μ1
    ))
    =
    − 1
    2 (
    zT
    1
    Λ11
    z1
    − 2z1 {Λ11
    μ1
    − 1
    2
    Λ12
    (z2
    − μ2
    )}) + C1
    − 1
    2 (
    zT
    2
    Λ22
    z2
    − 2z2 {Λ22
    μ2
    − 1
    2
    Λ21
    (z1
    − μ1
    )}) + C2
    

    View Slide

40. ΪϒεαϯϓϦϯά
    ۩ମྫͰֶͿ
    Αͬͯɼ৚݅෇͖֬཰ͷର਺͸ɼ
    
    ͱͳΔͷͰɼ৚݅෇͖֬཰͸Ψ΢ε෼෍Ͱ͋Δɼ
    
    
    log p(zi
    |zj
    ) = −
    1
    2 (
    zT
    i
    Λii
    zi
    − 2zi {
    Λii
    μi
    −
    1
    2
    Λij
    (zj
    − μj
    )})
    + C
    p(zi
    |zj
    ) = (zi
    |μi
    , Σi
    ),
    Σ−1
    i
    = Λii
    ,
    Σ−1
    i
    μi
    = Λii
    μi
    −
    1
    2
    Λij
    (zj
    − μj
    ),
    ⇔ μi
    = Σi (
    Λii
    μi
    −
    1
    2
    Λij
    (zj
    − μj
    )) .
    https://drive.google.com/open?id=1ReYNvvH-NgtsuRiDDV-lz1779sps2pT0
    

    View Slide