Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Features
Speaker Deck
PRO
Sign in
Sign up for free
Search
Search
Problems of Neural Networks and its solutions
Search
izuna385
June 21, 2018
Technology
0
110
Problems of Neural Networks and its solutions
Residual Connections とBatch Normalizationがメイン
izuna385
June 21, 2018
Tweet
Share
More Decks by izuna385
See All by izuna385
jel: japanese entity linker
izuna385
0
290
Firebase-React-App
izuna385
0
210
React+FastAPIを用いた簡単なWebアプリ作製
izuna385
0
1.3k
UseCase of Entity Linking
izuna385
0
400
Unofficial slides: From Zero to Hero: Human-In-The-Loop Entity Linking in Low Resource Domains (ACL 2020)
izuna385
1
580
Poly-encoders: Transformer Architectures and Pre-training Strategies for Fast and Accurate Multi-sentence Scoring
izuna385
0
630
Zero-shot Entity Linking with Dense Entity Retrieval (Unofficial slides) and Entity Linking future directions
izuna385
3
850
Entity representation with relational attention
izuna385
0
69
Zero-Shot Entity Linking by Reading Entity Descriptions
izuna385
0
450
Other Decks in Technology
See All in Technology
知識と実践を紡ぐGenAI / Connecting Knowledge and experience with GenAI
aki_moon
2
160
kcp: Kubernetes APIs Are All You Need #techfeed_live / TechFeed Experts Night 28th
ytaka23
1
180
1Q86
kawaguti
PRO
2
170
TypescriptでのContextualな構造化ロギングと社内全体への導入
leveragestech
3
540
技術力の伸ばし方を考える
khirata
0
140
【TSkaigi】2024/05/11 当日スライド
kimitashoichi
14
3.8k
Kaggleで学ぶ系列データのための深層学習モデリング
yu4u
7
1.7k
日本が誇るイタリアのダンスミュージック!? ユーロビートって何??
minorun365
PRO
1
130
DevRelによる信頼構築とデータ駆動で変わるエンジニア採用 / DevRel Trust Building to Data Driven Engineering Hiring
bobtani
1
120
スクラムに出会って「できた」を実感できるようになってきた話 / Scrum makes me feel like I can do it
yayoi_dd
2
110
Google Cloudを組織(企業)で運用する時のベストプラクティス × 健康の環境分離戦略 #まるクラ勉強会
yasumuusan
0
150
[2024년 5월 세미나] 생성형 AI와 함께하는 데이터 분석가 커리어
datarian
0
1.1k
Featured
See All Featured
Code Reviewing Like a Champion
maltzj
515
39k
How GitHub (no longer) Works
holman
305
140k
What the flash - Photography Introduction
edds
64
11k
Gamification - CAS2011
davidbonilla
77
4.6k
ピンチをチャンスに:未来をつくるプロダクトロードマップ #pmconf2020
aki_iinuma
81
44k
Raft: Consensus for Rubyists
vanstee
133
6.3k
実際に使うSQLの書き方 徹底解説 / pgcon21j-tutorial
soudai
123
39k
GitHub's CSS Performance
jonrohan
1025
450k
Understanding Cognitive Biases in Performance Measurement
bluesmoon
12
1.1k
Design by the Numbers
sachag
274
18k
Scaling GitHub
holman
457
140k
Side Projects
sachag
451
41k
Transcript
1 / 18 Neural Networks
2 / 18 1. NN !
• Residual Network • Batch Normalization 2. 1. • •
3 / 18 Plain NNs(&) ' pros #%
" (ex. CNN, RNN, ...) cons ! $ $
4 / 18 RNN RNN [1] P. Razvan et
al ,"On the difficulty of training recurrent neural networks." International Conference on Machine Learning. 2013. !"#$ !" %"&$ %"#$ %" %"&$ '() '() '() '*+, '*+, -!"# = /(!!"# ) -! -!$# %! : input !! : hidden state '%&' : '() : input / !" = '*+, 2 !"#$ + '() %"
5 / 18 !" !# !$ %" %# %$ &'(
&'( &'( &)*+ &)*+ ,! = .(!! ) ," ,# RNN 3 1, 12 = 1," 12 + 1,# 12 + 1,$ 12 1,$ 12 = 4 "565$ 1,$ 1!$ 7 1!$ 1!6 7 18!6 12 1!$ 1!" = 1!$ 1!# 7 1!# 1!" = &)*+ 9 :;<= >? !# 7 &)*+ 9 :;<= >? !" @A!B @C : !" ~!6E" fix !6
6 / 18 RNN Vanishing/Exploding Gradient : !"#$ !%&
'( )( … … )* '* ………… ………… +( +* !"#$ (-) !%& (-) '% …… '/ )/ +/
7 / 18 ,$+ /' !"#$ !- !"#$ 2 %
× '()* + ×%,- → # !"#$ !"#$ . 2 % × '()*(+).,-×%,- 1%input or 1)* Loss( RNN ."0& Vanishing/Exploding Gradient
8 / 18 +$ DeepNN( ! +
" )*&!/#% ' (→ ! Loss func ! Loss func → Residual Connection, Batch No malization
9 / 18 0), : Residual Connection – -– F(x)
"/#2 → "/ F(x) + x → (4 '$"/ Identity Mapping +%*1&: 3 . ! 3 Identity – [1] He, Kaiming, et al. "Identity mappings in deep residual networks." European Conference on Computer Vision. Springer, Cham, 2016.
10 / 18 : Residual Connection –– ' Forward
$#& Backward !$"& Deep % & input
11 / 18 Residual Connection –– https://icml.cc/2016/tutorials/icml2016_tutorial_deep_residual_networks_kaiminghe.pdf
12 / 18 ResNet Batch Normalization ResNet Residual Block
• ImplementationBatch Normalization NN ! $# • Batch Normalization" ## http://torch.ch/blog/2016/02/04/resnets.html Plain
13 / 18 ( ) 1 2
( ) n … Batch Normalization –Revisit Gaussian-
14 / 18 Batch Normalization -Input Data distribution
- (Convergence) !! Input NN → input
15 / 18 Batch Normalization -distribution - !"#$% & '
= ) & ' ← ' − , - ~/(,, -2) input
16 / 18 Batch Normalization Data distribution •
=(!, ")fix • Batch Normalization Batch Normalization
17 / 18 Batch Normalization – [2]Ioffe, Sergey,
and Christian Szegedy. "Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift." (2015). !, # !%$( → normalize scaling '"&# nomalize
18 / 18 DeepNN+ ! /
& -"#.#)%/'( *$ +!→ , Identity – normalize scaling implement Deep Net