Kento Nozawa
May 23, 2017
450

# Deep Learning 6.3-6.4

PDFでダウンロードするとリンクをクリックできます．

May 23, 2017

## Transcript

x + b )
4. ### ӅΕ૚ͷ׆ੑԽؔ਺ g • ΑΓ༏Εͨ׆ੑԽؔ਺Λݟ͚ͭΕ͹࿦จʹ • validation error౳Λݟܾͯఆ • g͸element-wiseʹద༻ •

ྫ֎: softmax h = g ( zj) = g ( X i ( wj,i ⇥ xi) + bj)
5. ### ׆ੑԽؔ਺ͷඍ෼ • ޯ഑Լ๏ʹΑΔߋ৽৚݅ɿඍ෼Մೳͳؔ਺ • ྫ֎ɿ g(x) = relu(x) ͷ x=0

• ࣮༻্͸໰୊ʹͳΒͳ͍ • ޯ഑͕0ʹͳΔ఺͸ೖྗʹདྷͳ͍ • ࣮૷্Ͳ͔ͬͪͷඍ෼஋Λฦ͢
6. ### • ໎ͬͨΒ͜Ε • ϝϦοτ • ඍ෼஋͕ఆ਺ɿ0 or 1 • ΄΅ઢܗͰɼ࠷దԽ͠΍͍͢

• σϝϦοτ • ෛͷ৔߹ɼޯ഑͕ͳ͍ • ֦ுͨؔ͠਺ • ELU, PReLU, Leakly ReLU 6.3.1 Rectiﬁed Linear UnitsʢReLUʣ
7. ### Maxout units • ׆ੑԽؔ਺ࣗମΛֶशɿ೚ҙͷತؔ਺ͷۙࣅ • 1Ϣχοτʹ͖ͭkݸͷύϥϝʔλ • k=2, d=1, m=1,

w=[1, 0], b=[0, 0] → ReLU • Dropoutͱͷซ༻Λਪ঑ʢఏҊ࿦จʣ • ReLU&Dropout: ͲͬͪͰ0ʹͳΔ͔ᐆດ • Catastrophic forgettingΛ؇࿨ [Ian Goodfellow, et al., 2015]͔Β
8. ### Catastrophic forgetting ୯ҰͷDNNͷϞσϧXΛߟ͑Δ 1. λεΫAΛֶश 2. λεΫAͷॏΈͷ··ɼผλεΫBΛֶश 3. λεΫAʹ͓͚Δੑೳ͸ग़ͳ͍ •

Aͷͱ͖ͷॏΈΛ๨Ε͍ͯΔ • ॏཁ౓ͷߴ͍ॏΈͷֶश཰͸Լ͛Δ͜ͱͰ๷͙ • J. Kirkpatrick et al., Overcoming Catastrophic Forgetting in Neural Networks. PNAS, 2017.
9. ### 6.3.2 Logistic Sigmoid and Hyperbolic Tangent • ޯ഑ϕʔεͷֶश๏Ͱ͸ඍົͳؔ਺ • ೖྗ஋͕0෇͚ۙͩහײ

• ͦΕҎ֎͸ανΔ: ޯ഑0ʹͳΓ΍͍͢ • 2୒ͳΒsigmoidΑΓtanh͕͍͍ • ཧ༝: 0෇ۙͰ܏͖1
10. ### 6.3.3 Other Hidden Units • MNISTΛ෼ྨ͢ΔMLPsͷ׆ੑԽؔ਺ͱͯ͠cosine • ࢼͨ͠Βreluͱಉ͘͡Β͍ • 11૚͘Β͍ॏͶΔͱޯ഑͕ফ͑·ͨ͠

• CIFAR10ͷCNNͩͱReLUʹෛ͚Δ • Notebooks: MNIST, CNN, CNN&BN • ͳͥ࿦จͰग़͍ͯͳ͍ʁ • ReLU΍ͦͷվྑʹউͨͳ͍ͱ͍͚ͳ͍

12. ### Architecture • ૚ͷਂ͞ • Ϣχοτ਺ • Ϣχοτؒͷ݁߹ • Convolution/Pooling/Skip-connection (Residual

Block) • GoogleͷAutoMLɿߏ଄΍׆ੑؔ਺ͷࣗಈܾఆɹ • Barret Zoph and Quoc Le. Neural Architecture Search with Reinforcement Learning. In Proc. ICLR, 2017. • ڧԽֶश • GPU਺: 800 • Esteban Real et al., Large-Scale Evolution of Image Classiﬁers. In Proc. ICML, 2017. • ਐԽతΞϧΰϦζϜ • Ϟσϧ਺: 1000
13. ### 6.4.1 Universal Approximation Properties and Depth • Universal approximation theorem

[1989] • squashing activation functionͰ1૚Ҏ্ͷதؒ૚Λ΋ͭNNs • ࣮਺ۭؒͷ෦෼ू߹Ͱ཈͑ΒΕ͔ͯͭดͨ͡࿈ଓؔ਺ΛۙࣅͰ͖Δ • ଌ౓͕Ͱͯ͘ΔͷͰɼຊॻͰ͸ৄࡉͳ࿩͸লུ • ʮۙࣅͰ͖Δʯ≠ʮֶशͰ͖Δʯ 1. ࠷దԽͰύϥϝʔλΛݟ͚ͭΔอূ͕ͳ͍ 2. ֶशΞϧΰϦζϜ͕ޡͬͨؔ਺ΛબͿ (ྫ: աֶश) • Ϣχοτ਺͕๲େʹඞཁ • ਂ͍ํ͕ύϥϝʔλ਺͸গͳ͘ࡁΉ: See Fig. 6.5 — 6.7.
14. ### Universal Approximation Theoremʹؔ࿈ͯ͠ • ࠷ۙͷ൚Խʹؔ͢Δ࿦จ • Chiyuan Zhang et al.,

Understanding Deep Learning Requires Rethinking Generalization. In Proc. ICLR, 2017. Best Paper. • ڭࢣϥϕϧΛshufﬂeͯ͠΋loss͸ͪΌΜͱԼ͕Δ