Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Features
Speaker Deck
PRO
Sign in
Sign up for free
Search
Search
Problems of Neural Networks and its solutions
Search
izuna385
June 21, 2018
Technology
0
130
Problems of Neural Networks and its solutions
Residual Connections とBatch Normalizationがメイン
izuna385
June 21, 2018
Tweet
Share
More Decks by izuna385
See All by izuna385
jel: japanese entity linker
izuna385
0
330
Firebase-React-App
izuna385
0
230
React+FastAPIを用いた簡単なWebアプリ作製
izuna385
0
1.6k
UseCase of Entity Linking
izuna385
0
530
Unofficial slides: From Zero to Hero: Human-In-The-Loop Entity Linking in Low Resource Domains (ACL 2020)
izuna385
1
640
Poly-encoders: Transformer Architectures and Pre-training Strategies for Fast and Accurate Multi-sentence Scoring
izuna385
0
810
Zero-shot Entity Linking with Dense Entity Retrieval (Unofficial slides) and Entity Linking future directions
izuna385
3
1k
Entity representation with relational attention
izuna385
0
76
Zero-Shot Entity Linking by Reading Entity Descriptions
izuna385
0
530
Other Decks in Technology
See All in Technology
MIMEと文字コードの闇
hirachan
2
1.4k
Change Managerを活用して本番環境へのセキュアなGUIアクセスを統制する / Control Secure GUI Access to the Production Environment with Change Manager
yuj1osm
0
110
いまからでも遅くない!コンテナでWebアプリを動かしてみよう!コンテナハンズオン編
nomu
0
170
Potential EM 制度を始めた理由、そして2年後にやめた理由 - EMConf JP 2025
hoyo
2
2.9k
What's new in Go 1.24?
ciarana
1
110
大規模アジャイルフレームワークから学ぶエンジニアマネジメントの本質
staka121
PRO
3
1.4k
サイト信頼性エンジニアリングとAmazon Web Services / SRE and AWS
ymotongpoo
7
1.8k
リクルートのエンジニア組織を下支えする 新卒の育成の仕組み
recruitengineers
PRO
1
140
"TEAM"を導入したら最高のエンジニア"Team"を実現できた / Deploying "TEAM" and Building the Best Engineering "Team"
yuj1osm
1
230
Exadata Database Service on Cloud@Customer セキュリティ、ネットワーク、および管理について
oracle4engineer
PRO
2
1.6k
アジャイルな開発チームでテスト戦略の話は誰がする? / Who Talks About Test Strategy?
ak1210
1
710
フォーイット_エンジニア向け会社紹介資料_Forit_Company_Profile.pdf
forit_tech
1
1.7k
Featured
See All Featured
The Cult of Friendly URLs
andyhume
78
6.2k
4 Signs Your Business is Dying
shpigford
183
22k
Art, The Web, and Tiny UX
lynnandtonic
298
20k
Reflections from 52 weeks, 52 projects
jeffersonlam
348
20k
Practical Orchestrator
shlominoach
186
10k
Gamification - CAS2011
davidbonilla
80
5.2k
Docker and Python
trallard
44
3.3k
Adopting Sorbet at Scale
ufuk
75
9.2k
A designer walks into a library…
pauljervisheath
205
24k
Understanding Cognitive Biases in Performance Measurement
bluesmoon
27
1.6k
The MySQL Ecosystem @ GitHub 2015
samlambert
250
12k
CSS Pre-Processors: Stylus, Less & Sass
bermonpainter
356
29k
Transcript
1 / 18 Neural Networks
2 / 18 1. NN !
• Residual Network • Batch Normalization 2. 1. • •
3 / 18 Plain NNs(&) ' pros #%
" (ex. CNN, RNN, ...) cons ! $ $
4 / 18 RNN RNN [1] P. Razvan et
al ,"On the difficulty of training recurrent neural networks." International Conference on Machine Learning. 2013. !"#$ !" %"&$ %"#$ %" %"&$ '() '() '() '*+, '*+, -!"# = /(!!"# ) -! -!$# %! : input !! : hidden state '%&' : '() : input / !" = '*+, 2 !"#$ + '() %"
5 / 18 !" !# !$ %" %# %$ &'(
&'( &'( &)*+ &)*+ ,! = .(!! ) ," ,# RNN 3 1, 12 = 1," 12 + 1,# 12 + 1,$ 12 1,$ 12 = 4 "565$ 1,$ 1!$ 7 1!$ 1!6 7 18!6 12 1!$ 1!" = 1!$ 1!# 7 1!# 1!" = &)*+ 9 :;<= >? !# 7 &)*+ 9 :;<= >? !" @A!B @C : !" ~!6E" fix !6
6 / 18 RNN Vanishing/Exploding Gradient : !"#$ !%&
'( )( … … )* '* ………… ………… +( +* !"#$ (-) !%& (-) '% …… '/ )/ +/
7 / 18 ,$+ /' !"#$ !- !"#$ 2 %
× '()* + ×%,- → # !"#$ !"#$ . 2 % × '()*(+).,-×%,- 1%input or 1)* Loss( RNN ."0& Vanishing/Exploding Gradient
8 / 18 +$ DeepNN( ! +
" )*&!/#% ' (→ ! Loss func ! Loss func → Residual Connection, Batch No malization
9 / 18 0), : Residual Connection – -– F(x)
"/#2 → "/ F(x) + x → (4 '$"/ Identity Mapping +%*1&: 3 . ! 3 Identity – [1] He, Kaiming, et al. "Identity mappings in deep residual networks." European Conference on Computer Vision. Springer, Cham, 2016.
10 / 18 : Residual Connection –– ' Forward
$#& Backward !$"& Deep % & input
11 / 18 Residual Connection –– https://icml.cc/2016/tutorials/icml2016_tutorial_deep_residual_networks_kaiminghe.pdf
12 / 18 ResNet Batch Normalization ResNet Residual Block
• ImplementationBatch Normalization NN ! $# • Batch Normalization" ## http://torch.ch/blog/2016/02/04/resnets.html Plain
13 / 18 ( ) 1 2
( ) n … Batch Normalization –Revisit Gaussian-
14 / 18 Batch Normalization -Input Data distribution
- (Convergence) !! Input NN → input
15 / 18 Batch Normalization -distribution - !"#$% & '
= ) & ' ← ' − , - ~/(,, -2) input
16 / 18 Batch Normalization Data distribution •
=(!, ")fix • Batch Normalization Batch Normalization
17 / 18 Batch Normalization – [2]Ioffe, Sergey,
and Christian Szegedy. "Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift." (2015). !, # !%$( → normalize scaling '"&# nomalize
18 / 18 DeepNN+ ! /
& -"#.#)%/'( *$ +!→ , Identity – normalize scaling implement Deep Net