Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Features
Speaker Deck
PRO
Sign in
Sign up for free
Search
Search
Problems of Neural Networks and its solutions
Search
izuna385
June 21, 2018
Technology
0
110
Problems of Neural Networks and its solutions
Residual Connections とBatch Normalizationがメイン
izuna385
June 21, 2018
Tweet
Share
More Decks by izuna385
See All by izuna385
jel: japanese entity linker
izuna385
0
280
Firebase-React-App
izuna385
0
210
React+FastAPIを用いた簡単なWebアプリ作製
izuna385
0
1.3k
UseCase of Entity Linking
izuna385
0
390
Unofficial slides: From Zero to Hero: Human-In-The-Loop Entity Linking in Low Resource Domains (ACL 2020)
izuna385
1
570
Poly-encoders: Transformer Architectures and Pre-training Strategies for Fast and Accurate Multi-sentence Scoring
izuna385
0
620
Zero-shot Entity Linking with Dense Entity Retrieval (Unofficial slides) and Entity Linking future directions
izuna385
3
840
Entity representation with relational attention
izuna385
0
69
Zero-Shot Entity Linking by Reading Entity Descriptions
izuna385
0
440
Other Decks in Technology
See All in Technology
Cloud Native Java with Spring Boot (CNCF Aarhus, April 2024)
thomasvitale
1
110
AWS を使う上で知っておきたいオンプレミス知識/aws-on-premise-essentials
emiki
1
4.2k
Databricks における 『MLOps』
databricksjapan
2
140
Tebiki株式会社 エンジニア採用資料
tebiki
0
4.1k
オーナーシップを持つ領域を明確にする
konifar
11
2.5k
マルチアカウント環境への発見的統制の導入
ch1aki
1
1.3k
Algyan イベント振り返り
linyixian
0
190
HEXA OSINT CTF V3 作戦会議
meow_noisy
0
110
PHPカンファレンス小田原2024
ysknsid25
3
660
シン・Kafka / shin-kafka
oracle4engineer
PRO
7
2.7k
巨大なテーブルのテーブル定義を無停止で安全に誰でも変更できるようにする / Table-definitions-for-huge-tables-can-be-modified-by-anyone-safely-and-non-disruptively
freee
1
740
Autonomous Database Cloud 技術詳細 / adb-s_technical_detail_jp
oracle4engineer
PRO
14
35k
Featured
See All Featured
Adopting Sorbet at Scale
ufuk
67
8.6k
Intergalactic Javascript Robots from Outer Space
tanoku
266
26k
How to name files
jennybc
64
92k
Large-scale JavaScript Application Architecture
addyosmani
503
110k
The Art of Programming - Codeland 2020
erikaheidi
41
12k
Design and Strategy: How to Deal with People Who Don’t "Get" Design
morganepeng
115
18k
ReactJS: Keep Simple. Everything can be a component!
pedronauck
658
120k
What's in a price? How to price your products and services
michaelherold
237
11k
Learning to Love Humans: Emotional Interface Design
aarron
266
39k
Agile that works and the tools we love
rasmusluckow
324
20k
ピンチをチャンスに:未来をつくるプロダクトロードマップ #pmconf2020
aki_iinuma
76
41k
Side Projects
sachag
451
41k
Transcript
1 / 18 Neural Networks
2 / 18 1. NN !
• Residual Network • Batch Normalization 2. 1. • •
3 / 18 Plain NNs(&) ' pros #%
" (ex. CNN, RNN, ...) cons ! $ $
4 / 18 RNN RNN [1] P. Razvan et
al ,"On the difficulty of training recurrent neural networks." International Conference on Machine Learning. 2013. !"#$ !" %"&$ %"#$ %" %"&$ '() '() '() '*+, '*+, -!"# = /(!!"# ) -! -!$# %! : input !! : hidden state '%&' : '() : input / !" = '*+, 2 !"#$ + '() %"
5 / 18 !" !# !$ %" %# %$ &'(
&'( &'( &)*+ &)*+ ,! = .(!! ) ," ,# RNN 3 1, 12 = 1," 12 + 1,# 12 + 1,$ 12 1,$ 12 = 4 "565$ 1,$ 1!$ 7 1!$ 1!6 7 18!6 12 1!$ 1!" = 1!$ 1!# 7 1!# 1!" = &)*+ 9 :;<= >? !# 7 &)*+ 9 :;<= >? !" @A!B @C : !" ~!6E" fix !6
6 / 18 RNN Vanishing/Exploding Gradient : !"#$ !%&
'( )( … … )* '* ………… ………… +( +* !"#$ (-) !%& (-) '% …… '/ )/ +/
7 / 18 ,$+ /' !"#$ !- !"#$ 2 %
× '()* + ×%,- → # !"#$ !"#$ . 2 % × '()*(+).,-×%,- 1%input or 1)* Loss( RNN ."0& Vanishing/Exploding Gradient
8 / 18 +$ DeepNN( ! +
" )*&!/#% ' (→ ! Loss func ! Loss func → Residual Connection, Batch No malization
9 / 18 0), : Residual Connection – -– F(x)
"/#2 → "/ F(x) + x → (4 '$"/ Identity Mapping +%*1&: 3 . ! 3 Identity – [1] He, Kaiming, et al. "Identity mappings in deep residual networks." European Conference on Computer Vision. Springer, Cham, 2016.
10 / 18 : Residual Connection –– ' Forward
$#& Backward !$"& Deep % & input
11 / 18 Residual Connection –– https://icml.cc/2016/tutorials/icml2016_tutorial_deep_residual_networks_kaiminghe.pdf
12 / 18 ResNet Batch Normalization ResNet Residual Block
• ImplementationBatch Normalization NN ! $# • Batch Normalization" ## http://torch.ch/blog/2016/02/04/resnets.html Plain
13 / 18 ( ) 1 2
( ) n … Batch Normalization –Revisit Gaussian-
14 / 18 Batch Normalization -Input Data distribution
- (Convergence) !! Input NN → input
15 / 18 Batch Normalization -distribution - !"#$% & '
= ) & ' ← ' − , - ~/(,, -2) input
16 / 18 Batch Normalization Data distribution •
=(!, ")fix • Batch Normalization Batch Normalization
17 / 18 Batch Normalization – [2]Ioffe, Sergey,
and Christian Szegedy. "Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift." (2015). !, # !%$( → normalize scaling '"&# nomalize
18 / 18 DeepNN+ ! /
& -"#.#)%/'( *$ +!→ , Identity – normalize scaling implement Deep Net