Share
Embed
深層学習の平均場理論 (2019/07/20)
matsuno
July 20, 2019
Transcript
57 slides
-
ਂֶशͷฏۉཧ matsuno 2019 7 ݄ 20 matsuno ਂֶशͷฏۉཧ
2019 7 ݄ 20 1 / 57 -
࣍ 1 ͡Ίʹ 2 ݯྲྀͱͳͬͨཧ ཧֶʹ͓͚Δฏۉཧ ਆܦճ࿏Ϟσϧͷฏۉཧ 3 ਂֶशͷฏۉཧ ॱൖͷฏۉཧ
ٯൖͷฏۉཧ ͦͷޙͷൃల 4 ·ͱΊ matsuno ਂֶशͷฏۉཧ 2019 7 ݄ 20 2 / 57 -
࣍ 1 ͡Ίʹ 2 ݯྲྀͱͳͬͨཧ ཧֶʹ͓͚Δฏۉཧ ਆܦճ࿏Ϟσϧͷฏۉཧ 3 ਂֶशͷฏۉཧ ॱൖͷฏۉཧ
ٯൖͷฏۉཧ ͦͷޙͷൃల 4 ·ͱΊ matsuno ਂֶशͷฏۉཧ 2019 7 ݄ 20 3 / 57 -
ͳͥ͜Μͳʹޭ͔ͨ͠ʁ දݱೳྗͷߴ͞ ύϥϝʔλͷଟ͞ ͷਂ͞ ࠷దԽख๏ͷൃల όοΫϓϩύήʔγϣϯ ֬తޯ߱Լ๏ (SGD)ɺAdam, ... ൚Խख๏ͷൃల
Dropout, batch normalization, ... matsuno ਂֶशͷฏۉཧ 2019 7 ݄ 20 5 / 57 -
ωοτϫʔΫͷબɾ։ൃ৬ਓٕʁ ωοτϫʔΫߏɺ׆ੑԽؔɺ࠷దԽख๏ɺʜ1 χϡʔϥϧωοτϫʔΫͷཧ͕ٕज़ͷൃలʹ͍͍͍ͭͯͳ͍ͨΊ χϡʔϥϧωοτϫʔΫͷҰൠతͳੑ࣭ΛΓ͍ͨ ಉ͡ૉࢠͳΒਂ͍ํ͕ྑ͍ͷ͔ʁ ܦݧతʹಋೖ͞ΕΔςΫχοΫ͕ͳͥ͏·͘ߦ͘ͷ͔ʁ ׆ੑԽؔωοτϫʔΫʹΑΒͳ͍ཧɺදݱೳྗͷࢦඪ͋ Δ͔ʁ 1࠷ۙ Neural
Architecture Search (NAS) ͱݺΕΔख๏͋Δ matsuno ਂֶशͷฏۉཧ 2019 7 ݄ 20 6 / 57 -
ࠓճͷྲྀΕ 1 ݯྲྀͱͳͬͨཧΛ؆୯ʹհ ཧֶʹ͓͚Δฏۉۙࣅ [4] ਆܦճ࿏Ϟσϧͷฏۉཧ [1, 2] 2 ࣍ʹɺ࠷ۙͷਂֶशͷฏۉཧΛհ
ॱͷฏۉཧ [5, 6] ٯͷฏۉཧ [6] ͦͷޙͷൃలΛ؆୯ʹհ [7, 8, 9] ʢಛผݴٴ͠ͳ͍ݶΓɺਤͦΕͧΕͷࢀߟจݙ͔ΒҾ༻ʣ ຊޠͷࢿྉ [10] Λେมࢀߟʹ͠·ͨ͠ matsuno ਂֶशͷฏۉཧ 2019 7 ݄ 20 9 / 57 -
ࢀߟจݙ I [1] H. Sompolinsky, A. Crisanti and H. J.
Sommers, “Chaos in random neural networks,” Physical review letters 61.3 (1988): 259. [2] རढ़Ұɺాഹɺ ʮਆܦճ࿏ͷجૅ (Խֶ 2, ֶऀͷͨΊͷ ࢠੜֶೖ-৽ֶ͍͠ΛΖ͏-)ʯ ɺੑݚڀ (2006), 87(3): 451-456, http://hdl.handle.net/2433/110690. [3] A. Crisanti and H. Sompolinsky, “Dynamics of spin systems with randomly asymmetric bonds: Langevin dynamics and a spherical model,” Physical Review A 36.10 (1987): 4922. [4] ా࡚໌ɺ ʮ౷ܭྗֶ IIʯ ɺഓ෩ؗ ৽ཧֶγϦʔζ 38. [5] B. Poole, S. Lahiri, M. Raghu, J. Sohl-Dickstein and S. Ganguli, “Exponential expressivity in deep neural networks through transient chaos,” NIPS 2016. matsuno ਂֶशͷฏۉཧ 2019 7 ݄ 20 10 / 57 -
ࢀߟจݙ II [6] S. S. Schoenholz, J. Gilmer, S. Ganguli
and J. Sohl-Dickstein, “Deep Information Propagation,” ICLR 2017. [7] G. Yang and S. S. Schoenholz, “Mean Field Residual Networks: On the Edge of Chaos,” NIPS 2017. [8] L. Xiao, Y. Bahri, J. Sohl-Dickstein, S. S. Schoenholz and J. Pennington, “Dynamical Isometry and a Mean Field Theory of CNNs: How to Train 10,000-Layer Vanilla Convolutional Neural Networks,” ICML 2018. [9] G. Yang, J. Pennington, V. Rao, J. Sohl-Dickstein and S. S. Schoenholz, “A Mean Field Theory of Batch Normalization,” ICLR 2019. [10] ా྄ɺ ʮਂχϡʔϥϧωοτϫʔΫͷཧ: ฏۉཧͷࢹ ʯ ɺ ʲୈ 25 ճ AI ηϛφʔʳ ʮਓೳͷཧʯ ɺhttps://drive. google.com/open?id=1Fhlarme8qFbhcGFLs3J8WQ3kYaJU3nbZ. matsuno ਂֶशͷฏۉཧ 2019 7 ݄ 20 11 / 57 -
࣍ 1 ͡Ίʹ 2 ݯྲྀͱͳͬͨཧ ཧֶʹ͓͚Δฏۉཧ ਆܦճ࿏Ϟσϧͷฏۉཧ 3 ਂֶशͷฏۉཧ ॱൖͷฏۉཧ
ٯൖͷฏۉཧ ͦͷޙͷൃల 4 ·ͱΊ matsuno ਂֶशͷฏۉཧ 2019 7 ݄ 20 12 / 57 -
࣍ 1 ͡Ίʹ 2 ݯྲྀͱͳͬͨཧ ཧֶʹ͓͚Δฏۉཧ ਆܦճ࿏Ϟσϧͷฏۉཧ 3 ਂֶशͷฏۉཧ ॱൖͷฏۉཧ
ٯൖͷฏۉཧ ͦͷޙͷൃల 4 ·ͱΊ matsuno ਂֶशͷฏۉཧ 2019 7 ݄ 20 13 / 57 -
ཧֶʹ͓͚Δฏۉཧ ͨ͘͞Μͷཻࢠ͕ू·Δͱɺཻࢠ 1 ͚ͭͩͰΘ͔Βͳ͍ϚΫϩͳ ໘ന͍ੑ࣭͕ݱΕΔ యܕతͳݱɿ૬సҠ ਫɾණɾਫৠؾͲΕಉ͡ਫࢠͰͰ͖͍ͯΔ͕ɺಛఆͷԹɾѹ ྗͰঢ়ଶ͕มԽ͢Δ ࣭ͷੑ࣭Λ໌Β͔ʹ͢ΔͨΊʹ࣭ΛϞσϧԽ͠ɺղੳ͢Δࡍͷ 1
ͭͷख๏͕ཧֶʹ͓͚Δฏۉཧ matsuno ਂֶशͷฏۉཧ 2019 7 ݄ 20 14 / 57 -
యܕతͳྫɿIsing ܕ લఏࣝ ʢԹҰఆʹอͨΕ͍ͯΔʣܥ͕ΤωϧΪʔ Ei Λͭঢ়ଶ i Ͱ͋Δ ֬ pi
= exp(−βEi )/Z ʢβ: ਖ਼ͷఆɺZ: ن֨Խఆʣ ઃఆ ࣭ͷϞσϧͱͯ͠ɺ֬తʹΛͱΔεϐϯ si = ±1 ͕ 2 ࣍ݩͷ ֨ࢠঢ়ʹฒΜͰ͍ͯɺ࣍ͷΤωϧΪʔΛ࣮ݱ͢Δ߹Λߟ͑Δ E = −J ∑ ⟨i,j⟩ si sj − H ∑ i si ⟨i, j⟩: ྡ͢Δ֨ࢠͷ εϐϯ͕ N ݸ͋Δͱ͢Δͱঢ়ଶͷ 2N ྫ͑εϐϯͷظ ⟨si ⟩ ΛٻΊ͍͕ͨɺେྔͷεϐϯͷ૬ޓ࡞༻ ؚ͕·ΕΔͷͰ͍͠ matsuno ਂֶशͷฏۉཧ 2019 7 ݄ 20 15 / 57 -
Ising ܕʹ͓͚Δฏۉۙࣅ ΤωϧΪʔͷ͏ͪɺ͋Δ֨ࢠ i = 0 ͕ؔΘΔ෦ʹ E0 = −J
4 ∑ i=1 s0si − Hs0 = − ( J 4 ∑ i=1 si + H ) s0 ฏۉۙࣅͰɺۙͷεϐϯͷΏΒ͗Λແࢹͯ͠ظ ψ = ⟨si ⟩ ʹஔ͖͑Δ E0 ∼ Es0 = −(4Jψ + H)s0 1 ཻࢠεϐϯͷظɺEs0 Λ༻͍ͯ؆୯ʹܭࢉͰ͖ͯ ⟨s0⟩ = p(s0 = +1) − p(s0 = −1) = tanh(β(4Jψ + H)) ͨ֨͠ࢠͱۙͷεϐϯͷҧ͍Կແ͍͔Β ⟨s0⟩ = ψ ψ = tanh(β(4Jψ + H)) ଟମܥͷεϐϯͷظ͕ɺ1 มํఔࣜͷղͰۙࣅͰ͖ͨ matsuno ਂֶशͷฏۉཧ 2019 7 ݄ 20 16 / 57 -
Ising ܕʹ͓͚Δฏۉۙࣅ ύϥϝʔλʹΑͬͯղ͕ඇ࿈ଓʹมԽ ⇒ ૬సҠ Ҿ༻ɿhttps://web.stanford.edu/~peastman/statmech/phasetransitions.html ࣮ઢɿy = tanh(β(4Jx +
H)) ͷάϥϑ ઢɿy = x ͷάϥϑ 2 ͭͷάϥϑͷަ͕ղ matsuno ਂֶशͷฏۉཧ 2019 7 ݄ 20 17 / 57 -
࣍ 1 ͡Ίʹ 2 ݯྲྀͱͳͬͨཧ ཧֶʹ͓͚Δฏۉཧ ਆܦճ࿏Ϟσϧͷฏۉཧ 3 ਂֶशͷฏۉཧ ॱൖͷฏۉཧ
ٯൖͷฏۉཧ ͦͷޙͷൃల 4 ·ͱΊ matsuno ਂֶशͷฏۉཧ 2019 7 ݄ 20 19 / 57 -
χϡʔϩϯϞσϧ ͜ͷࠒɺੜֶͷϞσϧͱͯ͠χϡʔϩϯ͕ݚڀ͞Ε͍ͯͨΑ͏ͩ จதʹ “membrane potential of the nerve cell” ʢਆܦࡉ๔ͷບిҐʣ
ͱ͍ͬͨݴ༿͕ొ͢Δ ບిҐʜࡉ๔ͷ֎ͷిҐࠩͷ͜ͱ x ∈ RN Λೖྗͱ͠ɺz ∈ RN Λग़ྗͱ͢ΔʢN ݸͷχϡʔϩϯʣ Λߟ͑Δ ॏΈ W ∈ RN×N, ൃՐͷᮢ h ∈ RN, ׆ੑԽؔ ϕ Λ༻͍Δͱɺग़ ྗ z z = ϕ(u) u = Wx − h matsuno ਂֶशͷฏۉཧ 2019 7 ݄ 20 20 / 57 -
࿈ଓ࣌ؒͷχϡʔϩϯϞσϧ ੜֶͷϞσϧͳͷͰɺೖग़ྗΛ࣌ؒͷؔʹ͢Δ Wx − h ʹΑΓൃՐ͠ɺࢦؔతʹݮਰ͢Δͱߟ͑ͯ z(t) = ϕ(u(t)) ∂tu(t)
= −u(t) + Wx(t) − h ͜ͷઅͰɺ؆୯ͷͨΊҎޙ h = 0 ͱ͢Δ matsuno ਂֶशͷฏۉཧ 2019 7 ݄ 20 21 / 57 -
ใͷϑΟʔυόοΫ χϡʔϩϯܨ͕ΓʹΑΓใͷϑΟʔυόοΫ͕͋Δ ͜͜Ͱɺग़ྗ͕ͦͷ··ೖྗͱͳΔϞσϧΛߟ͑Δ x(t) = ϕ(u(t)) ∂tu(t) = −u(t) +
Wx(t) matsuno ਂֶशͷฏۉཧ 2019 7 ݄ 20 22 / 57 -
ॏΈ͕ϥϯμϜͳ߹ ॏΈ W ͷ֤͕ɺฏۉ 0ɺࢄ σ2/N ͷਖ਼ن͔Β࣮ݱ͞Εͯ ͍Δ߹Λߟ͑Δ ҰݟҟͳΔॏΈΛͭωοτϫʔΫʹڞ௨ͷ๏ଇΛݟ͍ͩͤΔ ͜ͷωοτϫʔΫͷੑ࣭ͷղੳʹɺ֤εϐϯͷ૬ޓ࡞༻Λಠཱͳ
͔֬Βܾఆ͢ΔΑ͏ʹ Ising ܕΛ֦ுͨ͠ɺεϐϯάϥε ܕͷςΫχοΫ͕͑Δ ฏۉۙࣅʹΑΓ 1 ͭͷ ui ʹͰ͖Δ matsuno ਂֶशͷฏۉཧ 2019 7 ݄ 20 23 / 57 -
ॏΈ͕ϥϯμϜͳ߹ [1] Ͱɺui ͷࣗݾ૬ؔ ∆(τ) = ⟨ui (t)ui (t +
τ)⟩ Λௐ͍ͯΔ ∆ ҎԼͷํఔࣜΛຬͨ͢ ∂2 τ ∆ = −∂V /∂∆ V (∆) = − 1 2 ∆2 + ∫ ∞ −∞ Dz (∫ ∞ −∞ DxΦ((∆(0) − |∆|)1/2x + |∆|1/2z) )2 Dz = dz √ 2π e−z2/2, Φ(x) = ∫ ∞ 0 dyϕ(y) ∆ ͦͷఆ͔ٛΒɺҎԼͷੑ࣭Λͭ ∆(−τ) = ∆(τ) ∂τ ∆(τ)|τ=0 = 0 matsuno ਂֶशͷฏۉཧ 2019 7 ݄ 20 24 / 57 -
ॏΈ͕ϥϯμϜͳ߹ σ ͷʹΑͬͯϙςϯγϟϧ V ͕มԽ͠ɺ∆ ͷ࣌ؒൃల͕มԽ͢Δ 1 σ < 1
ͷ߹ɺ∆ = 0 ͷΈ͕ൃࢄ͠ͳ͍ղ (a) 2 σ > 1 ͷ߹ɺύϥϝʔλͱॳظʹΑͬͯෳछྨͷৼΔ͍ (b, c) matsuno ਂֶशͷฏۉཧ 2019 7 ݄ 20 25 / 57 -
࣍ 1 ͡Ίʹ 2 ݯྲྀͱͳͬͨཧ ཧֶʹ͓͚Δฏۉཧ ਆܦճ࿏Ϟσϧͷฏۉཧ 3 ਂֶशͷฏۉཧ ॱൖͷฏۉཧ
ٯൖͷฏۉཧ ͦͷޙͷൃల 4 ·ͱΊ matsuno ਂֶशͷฏۉཧ 2019 7 ݄ 20 27 / 57 -
࣍ 1 ͡Ίʹ 2 ݯྲྀͱͳͬͨཧ ཧֶʹ͓͚Δฏۉཧ ਆܦճ࿏Ϟσϧͷฏۉཧ 3 ਂֶशͷฏۉཧ ॱൖͷฏۉཧ
ٯൖͷฏۉཧ ͦͷޙͷൃల 4 ·ͱΊ matsuno ਂֶशͷฏۉཧ 2019 7 ݄ 20 28 / 57 -
ه๏ D + 1 ͷશ݁߹χϡʔϥϧωοτϫʔΫ ೖྗ͕ୈ 0 ɺग़ྗ͕ୈ D
ୈ ℓ ͷૉࢠΛ Nℓ ͱॻ͘ ୈ ℓ ʹ͢ΔͨΊͷॏΈͱόΠΞεΛ W ℓ, bℓ (ℓ = 1, . . . , D) ͱ ॻ͘ W ℓ ∈ RNℓ×Nℓ−1 , bℓ ∈ RNℓ ֤ͷग़ྗΛ xℓ ∈ RNℓ ͱ͢Δͱ xℓ = ϕ(hℓ), hℓ = W ℓxℓ−1 + bℓ x0: ೖྗ ϕ: ׆ੑԽؔʢຖʹ࡞༻ͤ͞Δʣ matsuno ਂֶशͷฏۉཧ 2019 7 ݄ 20 29 / 57 -
ઃఆ ॏΈͱόΠΞεͷ֤͕ɺҎԼͷ͔ΒͦΕͧΕϥϯμϜʹબ ͞ΕΔͱ͢Δ ॏΈɿฏۉ 0ɺࢄ σ2 w /Nℓ−1 ͷਖ਼ن όΠΞεɿฏۉ
0ɺࢄ σ2 b ͷਖ਼ن ࢄΛ Nℓ−1 Ͱׂ͍ͬͯΔͷɺ࣍ͷͷग़ྗͷεέʔϧΛଗ͑ΔͨΊ hℓ i = Nℓ−1 ∑ j=1 W ℓ ij xℓ−1 j + bℓ i ೖྗ x0 ͲͷΑ͏ʹ͢Δ͔ʁ 2 ͭͷೖྗ x0,1 ͱ x0,2 ͷ૬ؔʹΑͬͯͲͷΑ͏ʹมԽ͢Δ͔ʁ matsuno ਂֶशͷฏۉཧ 2019 7 ݄ 20 30 / 57 -
1 ͭͷೖྗͷɿܭࢉ (1/2) ֤ͷૉࢠ͕ͲΕ͘Β͍ൃՐ͍ͯ͠Δ͔Λ࣍ͷྔͰఆٛ͢Δ qℓ = 1 Nℓ Nℓ ∑
i=1 ( hℓ i )2 hℓ i (i = 1, . . . , Nℓ) ͕ͭ͘Δ֬ʢܦݧʣ Nℓ → ∞ Ͱਖ਼ن ͱͳΓɺͦͷࢄ qℓ ਖ਼ن͔Βͷେྔͷ࣮ݱ W ℓ, bℓ Ͱ༩͑ΒΕΔ ( hℓ i )2 ͷૉࢠํ ͷظɺ1 ͭͷૉࢠʹͨ͠ͱ͖ͷॏΈͱόΠΞεʹؔ͢Δ ظͱಉ͡ qℓ = ⟨( hℓ i )2 ⟩ ⟨·⟩ ɺW ℓ ͱ bℓ ʹؔ͢ΔظΛද͢ matsuno ਂֶशͷฏۉཧ 2019 7 ݄ 20 31 / 57 -
1 ͭͷೖྗͷɿܭࢉ (2/2) W ℓ, bℓ ฏۉ 0 ͷਖ਼نͷ࣮ݱͳͷͰɺަ߲ࠩͷظ͢ ͯফ͑Δ
qℓ = ⟨( hℓ i )2 ⟩ = ⟨ Nℓ−1 ∑ j=1 W ℓ ij ϕ ( hℓ−1 j ) + bℓ i 2⟩ = σ2 w Nℓ−1 Nℓ−1 ∑ i=1 ( ϕ ( hℓ−1 i ))2 + σ2 b 1 Nℓ−1 ∑ (· · · ) ܦݧΛ༻͍ͨظΛද͍ͯ͠Δ Nℓ−1 → ∞ Ͱɺਖ਼نΛ༻͍ͨظʹஔ͖͑ΒΕΔ qℓ = σ2 w ∫ ∞ −∞ dh(ϕ(h))2N ( h 0, qℓ−1 ) + σ2 b = σ2 w ∫ ∞ −∞ Dz ( ϕ (√ qℓ−1z ))2 + σ2 b , Dz = dz √ 2π e−z2/2 (1) matsuno ਂֶशͷฏۉཧ 2019 7 ݄ 20 32 / 57 -
1 ͭͷೖྗͷɿ࣮ݧ (1/2) ϕ(·) = tanh(·), σb = 0.3, Nℓ
= 1000 Ͱͷάϥϑʢೱ͍ઢཧʣ A qℓ−1 ʹର͢Δ qℓ ͷάϥϑ B ز͔ͭͷೖྗʹର͢Δ qℓ ͷมભ qℓ−1 = qℓ ͱͳΔݻఆ q∗ ͕ଘࡏʢάϥϑ A ͷˑʣ matsuno ਂֶशͷฏۉཧ 2019 7 ݄ 20 33 / 57 -
1 ͭͷೖྗͷɿ࣮ݧ (2/2) σb = 0, σw < 1 ͷ߹
ݻఆ q∗ = 0 ͷΈ ৗʹ qℓ−1 > qℓ ͱͳΓɺग़ྗ 0 ʹऩଋ͢Δ σb = 0, σw > 1 ͷ߹ ෆ҆ఆͳݻఆ q∗ = 0 ͱɺ҆ఆͳݻఆ͕ଘࡏ͢Δ େ͖ͳೖྗখ͞ͳೖྗɺʹΑΓ҆ఆͳ q∗ ʹۙͮ͘ σb ̸= 0 ͷ߹ ඇྵͷ҆ఆͳݻఆͷΈ͕ଘࡏ͢Δ σw < 1 Ͱ͋ͬͯɺόΠΞεʹΑΓग़ྗͷ 0 ͷऩଋ͕͕ΕΔ matsuno ਂֶशͷฏۉཧ 2019 7 ݄ 20 34 / 57 -
2 ͭͷೖྗͷɿܭࢉ (1/4) 2 ͭͷೖྗ x0,1, x0,2 ʹର͠ɺ֤ͷ૬ؔΛ࣍ͷྔͰఆٛ͢Δ qℓ ab
= 1 Nℓ Nℓ ∑ i=1 hℓ i ( x0,a ) hℓ i ( x0,b ) , a, b ∈ {1, 2} a = b ͷ߹ qℓ ʹͳΔͷͰɺqℓ 12 Λߟ͑Δ hℓ i ( x0,1 ) ͱ hℓ i ( x0,2 ) ͷಉ࣌ܦݧɺNℓ → ∞ Ͱ 2 ࣍ݩਖ਼ن ͱͳΓɺڞࢄ Qℓ ୈ (a, b) ͕ qℓ ab Ͱ͋ΔߦྻΛ Qℓ ͱॻ͘͜ͱʹ͢Δ matsuno ਂֶशͷฏۉཧ 2019 7 ݄ 20 35 / 57 -
2 ͭͷೖྗͷɿܭࢉ (2/4) 1 ͭͷೖྗͷ߹ͱಉ༷ʹܭࢉ͢Δͱ qℓ 12 = ⟨ hℓ
i ( x0,1 ) hℓ i ( x0,2 )⟩ = ⟨ Nℓ−1 ∑ j=1 W ℓ ij ϕ ( hℓ−1 j ( x0,1 )) + bℓ i Nℓ−1 ∑ k=1 W ℓ ik ϕ ( hℓ−1 k ( x0,1 )) + bℓ i ⟩ = σ2 w Nℓ−1 Nℓ−1 ∑ i=1 ϕ ( hℓ−1 i ( x0,1 )) ϕ ( hℓ−1 i ( x0,2 )) + σ2 b Nℓ−1→∞ − − − − − − → σ2 w ∫ dh1dh2ϕ (h1) ϕ (h2) N ( ⃗ h ⃗ 0, Qℓ−1 ) + σ2 b = σ2 w ∫ dh1dh2ϕ (h1) ϕ (h2) 1 2π √ |Qℓ−1| exp ( − 1 2 ⃗ hT ( Qℓ−1 ) −1 ⃗ h ) + σ2 b matsuno ਂֶशͷฏۉཧ 2019 7 ݄ 20 36 / 57 -
2 ͭͷೖྗͷɿܭࢉ (3/4) 2 มͷಠཱͳΨεੵͰॻ͖͍ͨ ૬ؔ cℓ = qℓ 12
/ √ qℓ 11 qℓ 22 Λ༻͍Δͱɺexp ͷத − 1 2 ⃗ hT ( Qℓ−1 ) −1 ⃗ h = − 1 2 ( 1 − (cℓ−1)2 ) h2 1 qℓ−1 11 − 2cℓ−1 √ qℓ−1 11 qℓ−1 22 h1h2 + h2 2 qℓ−1 22 มม h1 = √ qℓ−1 11 z1, h2 = √ qℓ−1 22 ( cℓ−1z1 + √ 1 − (cℓ−1)2z2 ) Λߦ͏ͱ − 1 2 ⃗ hT ( Qℓ−1 ) −1 ⃗ h = − 1 2 ( z2 1 + z2 2 ) matsuno ਂֶशͷฏۉཧ 2019 7 ݄ 20 37 / 57 -
2 ͭͷೖྗͷɿܭࢉ (4/4) มมʹΑΔϠίϏΞϯ ∂ (h1, h2) ∂ (z1, z2)
= √ qℓ−1 11 qℓ−1 22 [ 1 − (cℓ−1)2 ] = √ |Qℓ−1| ·ͱΊΔͱ qℓ 12 = σ2 w ∫ Dz1Dz2ϕ (h1) ϕ (h2) + σ2 b , h1 = √ qℓ−1 11 z1, h2 = √ qℓ−1 22 ( cℓ−1z1 + √ 1 − (cℓ−1)2z2 ) qℓ 11 , qℓ 22 ेૣ͘ q∗ ʹऩଋ͢ΔͷͰɺҎԼͷࣜͰݻఆͰͷ૬ؔ Λٞ͢Δ cℓ = 1 q∗ ( σ2 w ∫ Dz1Dz2ϕ (h1) ϕ (h2) + σ2 b ) , (2) h1 = √ q∗z1, h2 = √ q∗ ( cℓ−1z1 + √ 1 − (cℓ−1)2z2 ) matsuno ਂֶशͷฏۉཧ 2019 7 ݄ 20 38 / 57 -
૬ؔͷৼΔ͍ cℓ ( cℓ−1 = 1 ) = 1 ͳͷͰɺc∗
= 1 ݻఆ c∗ = 1 ͷ҆ఆੑɺχ1 = ∂cℓ ∂cℓ−1 cℓ−1=1 Ͱܾ·Δ χ1 < 1 ͳΒɺc∗ = 1 ͷۙ͘Ͱ cℓ−1 < cℓ ͱͳΓɺc∗ = 1 ʹऩଋ͢Δ χ1 > 1 ͳΒ c∗ = 1 ͷۙ͘Ͱ cℓ−1 > cℓ ͱͳΓɺc∗ = 1 ͔Βԕ͔͟Δ χ1 = ∂cℓ ∂cℓ−1 cℓ−1=1 = σ2 w √ q∗ ∫ Dz1Dz2ϕ (h1) ϕ′ (h2) z1 − cℓ−1 √ 1 − (cℓ−1)2 z2 cℓ−1=1 ∫ DzF(z)z = ∫ DzF′(z) Λ༻͍Δͱ χ1 = σ2 w ∫ Dz1Dz2ϕ′ (h1) ϕ′ (h2) cℓ−1=1 = σ2 w ∫ Dz ( ϕ′ (√ q∗z ))2 matsuno ਂֶशͷฏۉཧ 2019 7 ݄ 20 39 / 57 -
૬ؔͷৼΔ͍ 1 χ1 < 1: டং૬ c∗ = 1 ҆ఆͳݻఆ
ҟͳΔೖྗͰɺʹΑͬͯ૬͕ؔߴ͘ͳ͍ͬͯ͘ σw খ͍͞ 2 χ1 > 1: ΧΦε૬ c∗ = 1 ෆ҆ఆͳݻఆ ҆ఆͳݻఆ c∗ < 1 ͕ଘࡏ ҟͳΔೖྗͷࠩΛ֦େ͢Δ σw େ͖͍ matsuno ਂֶशͷฏۉཧ 2019 7 ݄ 20 40 / 57 -
ಛతͳਂ͞εέʔϧͷଘࡏ ࣮ݧͯ͠ΈΔͱɺq∗ c∗ ͷऩଋࢦؔతʹͳ͍ͬͯΔ qℓ − q∗ ∼ e−ℓ/ξq
cℓ − c∗ ∼ e−ℓ/ξc ξq, ξc ͦΕͧΕɺ୯Ұͷೖྗͷେ͖͞ͱɺ2 ͭͷೖྗͷ૬͕ؔͲΕ ͚ͩਂ͍·ͰऩଋͤͣʹͰ͖Δ͔Λද͢ matsuno ਂֶशͷฏۉཧ 2019 7 ݄ 20 41 / 57 -
ξq ͷܭࢉ (1) ࣜʹ qℓ = q∗ + ϵℓ q
Λೖ͠ɺTaylor ల։͢Δͱ ϵℓ q = ϵℓ−1 q σ2 w √ q∗ ∫ Dzϕ (√ q∗z ) ϕ′ (√ q∗z ) z + · · · = ϵℓ−1 q σ2 w √ q∗ ∫ Dz ∂ ∂z [ ϕ (√ q∗z ) ϕ′ (√ q∗z )] + · · · = ϵℓ−1 q [ χ1 + σ2 w ∫ Dzϕ (√ q∗z ) ϕ′′ (√ q∗z )] + · · · ϵℓ q ∼ e−ℓ/ξq ͱൺֱ͢Δͱ ξ−1 q = − log [ χ1 + σ2 w ∫ Dzϕ (√ q∗z ) ϕ′′ (√ q∗z )] matsuno ਂֶशͷฏۉཧ 2019 7 ݄ 20 42 / 57 -
ξc ͷܭࢉ ಉ༷ʹɺ(2) ࣜʹ cℓ = c∗ + ϵℓ c
Λೖ͠ɺTaylor ల։͢Δ ϵℓ c = ϵℓ−1 c σ2 w √ q∗ ∫ Dz1Dz2ϕ (h1) ϕ′ (h2) z1 − c∗ √ 1 − (c∗)2 z2 + · · · = ϵℓ−1 c σ2 w √ q∗ ∫ Dz1Dz2 ∂ ∂z1 − c∗ √ 1 − (c∗)2 ∂ ∂z2 [ ϕ (h1) ϕ′ (h2) ] + · · · = ϵℓ−1 c σ2 w ∫ Dz1Dz2ϕ′ (h1) ϕ′ (h2) + · · · ϵℓ c ∼ e−ℓ/ξc ͱൺֱ͢Δͱ ξ−1 c = − log [ σ2 w ∫ Dz1Dz2ϕ′ (h1) ϕ′ (h2) ] matsuno ਂֶशͷฏۉཧ 2019 7 ݄ 20 43 / 57 -
ξq , ξc ͷৼΔ͍ ࣮ݧʢ࣮ઢʣͱཧʢഁઢʣͷൺֱ σ2 b = 0.01ʢࠇʣ͔Β σ2
b = 0.3ʢʣ·Ͱม͑ͨ݁Ռ ࣮ݧͱཧͰΑ͘߹͍ͬͯΔ டং૬Ͱ c∗ = 1 ͳͷͰ ξ−1 c = − log χ1 Ͱ͋ΓɺసҠͰ χ1 = 1 ͔ͩΒ ξc ൃࢄ͢Δ ࣮ݧతʹൃࢄ͕Α͘ݟ͍͑ͯΔ σʔλؒͷ૬͕ؔऩଋͤͣʹΘ͍ͬͯ͘ matsuno ਂֶशͷฏۉཧ 2019 7 ݄ 20 44 / 57 -
࣍ 1 ͡Ίʹ 2 ݯྲྀͱͳͬͨཧ ཧֶʹ͓͚Δฏۉཧ ਆܦճ࿏Ϟσϧͷฏۉཧ 3 ਂֶशͷฏۉཧ ॱൖͷฏۉཧ
ٯൖͷฏۉཧ ͦͷޙͷൃల 4 ·ͱΊ matsuno ਂֶशͷฏۉཧ 2019 7 ݄ 20 45 / 57 -
ٯͷฏۉཧ ͔͜͜Βٯʹରͯ͠ಉ༷ͷٞΛߦͬͯΈΔ ଛࣦؔ E ͷඍ δℓ i = ∂E/∂hℓ i
͕ͲΕ͚ͩൃՐ͍ͯ͠Δ͔Λߟ͑Δ qℓ = ⟨( δℓ i )2 ⟩ ॱൖͷ߹ͷ qℓ ʹରԠ qℓ ͕ඇྵ͔ͭ༗ݶʹͱͲ·ΔύϥϝʔλྖҬ͕Γ͍ͨ ௨ৗͷ chain rule ʹΑΓɺδℓ i ࣍ͷੑ࣭Λͭ δℓ i = ∂E ∂hℓ i = Nℓ+1 ∑ j=1 ∂E ∂hℓ+1 j ∂hℓ+1 j ∂hℓ i = ϕ′ ( hℓ i ) Nℓ+1 ∑ j=1 δℓ+1 j W ℓ+1 ji matsuno ਂֶशͷฏۉཧ 2019 7 ݄ 20 46 / 57 -
ٯͷฏۉཧɿܭࢉ ·ͣ δℓ i ͷදࣜΛೖ qℓ = ⟨( δℓ i
)2 ⟩ = ⟨ ϕ′ ( hℓ i ) Nℓ+1 ∑ j=1 δℓ+1 j W ℓ+1 ji 2⟩ ͜͜Ͱɺٯܭࢉ࣌ʹ༻͢ΔॏΈɺॱʹ༻͢ΔॏΈͱ ಠཱʹਖ਼ن͔Βऔ͖͍ͬͯͯΔͱԾఆ͢Δͱɺ֤߲͕Ͱ ͖ͯ qℓ = ⟨( ϕ′ ( hℓ i ))2 ⟩ Nℓ+1 ∑ j=1 ⟨( δℓ+1 j )2 ⟩ ⟨( W ℓ+1 ji )2 ⟩ = ∫ dh ( ϕ′(h) )2 N ( h 0, qℓ ) × qℓ+1 Nℓ+1 ∑ j=1 σ2 w Nℓ qℓ→q∗ − − − − → qℓ+1 Nℓ+1 Nℓ χ1 matsuno ਂֶशͷฏۉཧ 2019 7 ݄ 20 47 / 57 -
ಛతͳਂ͞εέʔϧ ॱͷ߹ͱಉ༷ʹɺಛతͳਂ͞εέʔϧ ξ∇ ΛܭࢉͰ͖Δ ؆୯ͷͨΊ Nℓ = Nℓ+1 ͱ͢Δͱɺqℓ =
qℓ+1χ1 ͳͷͰ qℓ ∼ qDe−(D−ℓ)/ξ∇ , ξ−1 ∇ = − log χ1 ॱͷ߹ʹࣔͨ͠ 2 ͭͷ૬ͱసҠ্ͰɺޯͷৼΔ͍͕ େ͖͘มΘΔ டং૬ (χ1 < 1) Ͱ ξ∇ > 0 ͱͳΓɺ|ξ∇ | ͘Β͍ͷٯͰޯ͕ ফࣦ ΧΦε૬ (χ1 > 1) Ͱ ξ∇ < 0 ͱͳΓɺ|ξ∇ | ͘Β͍ͷٯͰޯ͕ ൃࢄ సҠ্ (χ1 = 1) Ͱޯ͕༗ݶʹͱͲ·Δ matsuno ਂֶशͷฏۉཧ 2019 7 ݄ 20 48 / 57 -
࣮ݧɿMNIST ʹ͓͚Δޯ (a) ύϥϝʔλʹΑͬͯɺೖྗଆʹۙͮ͘΄Ͳޯ͕ൃࢄ·ͨ 0 ʹऩ ଋ͍ͯ͠Δ (b) ξ∇ ͕ཧͱ࣮ݧͰΑ͘߹͍ͬͯΔ
matsuno ਂֶशͷฏۉཧ 2019 7 ݄ 20 49 / 57 -
࣮ݧɿֶशՄೳੑ ϥϯμϜωοτϫʔΫֶ͕शͰ͖ΔͨΊʹɺೖྗ͞Εͨใ͕࠷ ޙ·Ͱ͠ɺޯ͕ઌ಄·Ͱ͢Δ͜ͱ͕ඞཁͰͳ͍͔ L, σw Λมֶ͑ͯशͯ͠ΈΔͱɺωοτϫʔΫͷਫ਼ͱ ξc ͷڍಈ͕ Ұக͢Δ ৭ೱ͍΄Ͳֶशσʔλʹର͢Δਫ਼͕ߴ͍͜ͱΛද͢
matsuno ਂֶशͷฏۉཧ 2019 7 ݄ 20 50 / 57 -
࣍ 1 ͡Ίʹ 2 ݯྲྀͱͳͬͨཧ ཧֶʹ͓͚Δฏۉཧ ਆܦճ࿏Ϟσϧͷฏۉཧ 3 ਂֶशͷฏۉཧ ॱൖͷฏۉཧ
ٯൖͷฏۉཧ ͦͷޙͷൃల 4 ·ͱΊ matsuno ਂֶशͷฏۉཧ 2019 7 ݄ 20 51 / 57 -
Residual Network ͷฏۉཧ શ݁߹ residual network ʹର͢ΔฏۉཧΛߏங [7] શ݁߹ residual
network: xℓ = V ℓϕ ( hℓ ) + xℓ−1 + aℓ, hℓ = W ℓxℓ−1 + bℓ ओͳࢦඪɿ eℓ = ⟨ hℓ,1 i hℓ,2 i ⟩ √⟨( xℓ,1 i )2 ⟩ ⟨( xℓ,2 i )2 ⟩, χℓ = ⟨( ∂E ∂xℓ i )2 ⟩ ϕ(·) = tanh(·) ͷ߹ eℓ − e∗ ∼ ℓ−δ∗ , χm ∼ eA( √ ℓ− √ m)χℓ (A ∝ σw ) ී௨ͷωοτϫʔΫΑΓऩଋ͕͘ɺΑΓใ͕ΘΓ͍͢ matsuno ਂֶशͷฏۉཧ 2019 7 ݄ 20 52 / 57 -
Residual Network ͷฏۉཧɿ࣮ݧ MNIST Ͱֶशͨ͠ͱ͖ͷɺςετσʔλʹର͢Δਫ਼ ઢɿlog ( χ0/χℓ ) ∼
σw √ L ͕Ұఆ tanh ͷ߹ʹ eℓ − e∗ ΑΓ χm ͷํ͕ൃࢄ͍͢͠ͷͰɺχm ͷ ൃࢄΛ͑ΔϋΠύʔύϥϝʔλͰ͋Δ͜ͱ͕ॏཁ ޯͷେ͖͕͞มΘΒͳ͍͜ͱ͕ཧͳΒɺlog ( χ0/χℓ ) = 0 ͕ཧʁ → σw ͕খ͍͞ͱҟͳΔೖྗͷͷ͕͕ࠩΒͣɺදݱೳྗ্͕͕ Βͣʹ͏·͍͔͘ͳ͍ͱߟ͑ΒΕΔ matsuno ਂֶशͷฏۉཧ 2019 7 ݄ 20 53 / 57 -
ΈࠐΈχϡʔϥϧωοτϫʔΫ (CNN) ͷฏۉཧ CNN ʹର͢ΔฏۉཧΛߏங [8] पظڥք݅Λͭ 1 ࣍ݩ CNN
νϟϯωϧํेେ͖͍ͱߟ͑Δ͕ɺۭؒํ༗ݶͰΑ͍ 2 ࣍ݩ CNN ʹର͢Δ৽͍͠ॏΈॳظԽํ๏ΛఏҊ͠ɺ10,000 ͷ CNN ͷֶशʹޭ Λਂ͘͢Δ͚ͩͰ಄ଧͪʹͳ͍ͬͯΔͷͰɺresidual connection batch normalization ͷΑ͏ͳߏֶ͕शͷޮੑҎ֎ͷ؍͔Β ॏཁ͔͠Εͳ͍ ࠨɿMNISTɺӈɿCIFAR-10 matsuno ਂֶशͷฏۉཧ 2019 7 ݄ 20 54 / 57 -
Batch normalization (batchnorm) ͷฏۉཧ Batchnorm ΛऔΓೖΕͨશ݁߹ωοτϫʔΫʹର͢Δฏۉཧ [9] Batchnorm Λಋೖ͢Δͱɺޯ͕ඞͣൃࢄ͢Δ →
ֶशՄೳͳωοτϫʔΫͷਂ͞ʹ্ݶ͕͋Δ matsuno ਂֶशͷฏۉཧ 2019 7 ݄ 20 55 / 57 -
࣍ 1 ͡Ίʹ 2 ݯྲྀͱͳͬͨཧ ཧֶʹ͓͚Δฏۉཧ ਆܦճ࿏Ϟσϧͷฏۉཧ 3 ਂֶशͷฏۉཧ ॱൖͷฏۉཧ
ٯൖͷฏۉཧ ͦͷޙͷൃల 4 ·ͱΊ matsuno ਂֶशͷฏۉཧ 2019 7 ݄ 20 56 / 57