Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Features
Speaker Deck
PRO
Sign in
Sign up for free
Search
Search
Problems of Neural Networks and its solutions
Search
izuna385
June 21, 2018
Technology
0
150
Problems of Neural Networks and its solutions
Residual Connections とBatch Normalizationがメイン
izuna385
June 21, 2018
Tweet
Share
More Decks by izuna385
See All by izuna385
jel: japanese entity linker
izuna385
0
380
Firebase-React-App
izuna385
0
250
React+FastAPIを用いた簡単なWebアプリ作製
izuna385
0
1.7k
UseCase of Entity Linking
izuna385
0
580
Unofficial slides: From Zero to Hero: Human-In-The-Loop Entity Linking in Low Resource Domains (ACL 2020)
izuna385
1
660
Poly-encoders: Transformer Architectures and Pre-training Strategies for Fast and Accurate Multi-sentence Scoring
izuna385
0
870
Zero-shot Entity Linking with Dense Entity Retrieval (Unofficial slides) and Entity Linking future directions
izuna385
3
1.1k
Entity representation with relational attention
izuna385
0
83
Zero-Shot Entity Linking by Reading Entity Descriptions
izuna385
0
560
Other Decks in Technology
See All in Technology
Claude CodeでKiroの仕様駆動開発を実現させるには...
gotalab555
3
900
猫でもわかるQ_CLI(CDK開発編)+ちょっとだけKiro
kentapapa
0
3.4k
AI によるドキュメント処理を加速するためのOCR 結果の永続化と再利用戦略
tomoaki25
0
410
Jamf Connect ZTNAとMDMで実現! 金融ベンチャーにおける「デバイストラスト」実例と軌跡 / Kyash Device Trust
rela1470
0
150
Rubyの国のPerlMonger
anatofuz
3
730
マルチモーダル基盤モデルに基づく動画と音の解析技術
lycorptech_jp
PRO
4
520
【CEDEC2025】『Shadowverse: Worlds Beyond』二度目のDCG開発でゲームをリデザインする~遊びやすさと競技性の両立~
cygames
PRO
1
290
JAWS AI/ML #30 AI コーディング IDE "Kiro" を触ってみよう
inariku
3
290
2025-07-31: GitHub Copilot Agent mode at Vibe Coding Cafe (15min)
chomado
2
380
Strands Agents & Bedrock AgentCoreを1分でおさらい
minorun365
PRO
6
240
オブザーバビリティプラットフォーム開発におけるオブザーバビリティとの向き合い / Hatena Engineer Seminar #34 オブザーバビリティの実現と運用編
arthur1
0
350
✨敗北解法コレクション✨〜Expertだった頃に足りなかった知識と技術〜
nanachi
1
510
Featured
See All Featured
Embracing the Ebb and Flow
colly
86
4.8k
Java REST API Framework Comparison - PWX 2021
mraible
32
8.8k
Principles of Awesome APIs and How to Build Them.
keavy
126
17k
Faster Mobile Websites
deanohume
308
31k
The Straight Up "How To Draw Better" Workshop
denniskardys
235
140k
Build your cross-platform service in a week with App Engine
jlugia
231
18k
GraphQLの誤解/rethinking-graphql
sonatard
71
11k
Automating Front-end Workflow
addyosmani
1370
200k
Building Better People: How to give real-time feedback that sticks.
wjessup
367
19k
RailsConf & Balkan Ruby 2019: The Past, Present, and Future of Rails at GitHub
eileencodes
139
34k
CSS Pre-Processors: Stylus, Less & Sass
bermonpainter
357
30k
個人開発の失敗を避けるイケてる考え方 / tips for indie hackers
panda_program
110
19k
Transcript
1 / 18 Neural Networks
2 / 18 1. NN !
• Residual Network • Batch Normalization 2. 1. • •
3 / 18 Plain NNs(&) ' pros #%
" (ex. CNN, RNN, ...) cons ! $ $
4 / 18 RNN RNN [1] P. Razvan et
al ,"On the difficulty of training recurrent neural networks." International Conference on Machine Learning. 2013. !"#$ !" %"&$ %"#$ %" %"&$ '() '() '() '*+, '*+, -!"# = /(!!"# ) -! -!$# %! : input !! : hidden state '%&' : '() : input / !" = '*+, 2 !"#$ + '() %"
5 / 18 !" !# !$ %" %# %$ &'(
&'( &'( &)*+ &)*+ ,! = .(!! ) ," ,# RNN 3 1, 12 = 1," 12 + 1,# 12 + 1,$ 12 1,$ 12 = 4 "565$ 1,$ 1!$ 7 1!$ 1!6 7 18!6 12 1!$ 1!" = 1!$ 1!# 7 1!# 1!" = &)*+ 9 :;<= >? !# 7 &)*+ 9 :;<= >? !" @A!B @C : !" ~!6E" fix !6
6 / 18 RNN Vanishing/Exploding Gradient : !"#$ !%&
'( )( … … )* '* ………… ………… +( +* !"#$ (-) !%& (-) '% …… '/ )/ +/
7 / 18 ,$+ /' !"#$ !- !"#$ 2 %
× '()* + ×%,- → # !"#$ !"#$ . 2 % × '()*(+).,-×%,- 1%input or 1)* Loss( RNN ."0& Vanishing/Exploding Gradient
8 / 18 +$ DeepNN( ! +
" )*&!/#% ' (→ ! Loss func ! Loss func → Residual Connection, Batch No malization
9 / 18 0), : Residual Connection – -– F(x)
"/#2 → "/ F(x) + x → (4 '$"/ Identity Mapping +%*1&: 3 . ! 3 Identity – [1] He, Kaiming, et al. "Identity mappings in deep residual networks." European Conference on Computer Vision. Springer, Cham, 2016.
10 / 18 : Residual Connection –– ' Forward
$#& Backward !$"& Deep % & input
11 / 18 Residual Connection –– https://icml.cc/2016/tutorials/icml2016_tutorial_deep_residual_networks_kaiminghe.pdf
12 / 18 ResNet Batch Normalization ResNet Residual Block
• ImplementationBatch Normalization NN ! $# • Batch Normalization" ## http://torch.ch/blog/2016/02/04/resnets.html Plain
13 / 18 ( ) 1 2
( ) n … Batch Normalization –Revisit Gaussian-
14 / 18 Batch Normalization -Input Data distribution
- (Convergence) !! Input NN → input
15 / 18 Batch Normalization -distribution - !"#$% & '
= ) & ' ← ' − , - ~/(,, -2) input
16 / 18 Batch Normalization Data distribution •
=(!, ")fix • Batch Normalization Batch Normalization
17 / 18 Batch Normalization – [2]Ioffe, Sergey,
and Christian Szegedy. "Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift." (2015). !, # !%$( → normalize scaling '"&# nomalize
18 / 18 DeepNN+ ! /
& -"#.#)%/'( *$ +!→ , Identity – normalize scaling implement Deep Net