Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Features
Speaker Deck
PRO
Sign in
Sign up for free
Search
Search
Problems of Neural Networks and its solutions
Search
izuna385
June 21, 2018
Technology
0
140
Problems of Neural Networks and its solutions
Residual Connections とBatch Normalizationがメイン
izuna385
June 21, 2018
Tweet
Share
More Decks by izuna385
See All by izuna385
jel: japanese entity linker
izuna385
0
350
Firebase-React-App
izuna385
0
240
React+FastAPIを用いた簡単なWebアプリ作製
izuna385
0
1.7k
UseCase of Entity Linking
izuna385
0
560
Unofficial slides: From Zero to Hero: Human-In-The-Loop Entity Linking in Low Resource Domains (ACL 2020)
izuna385
1
650
Poly-encoders: Transformer Architectures and Pre-training Strategies for Fast and Accurate Multi-sentence Scoring
izuna385
0
840
Zero-shot Entity Linking with Dense Entity Retrieval (Unofficial slides) and Entity Linking future directions
izuna385
3
1.1k
Entity representation with relational attention
izuna385
0
82
Zero-Shot Entity Linking by Reading Entity Descriptions
izuna385
0
550
Other Decks in Technology
See All in Technology
自動化の第一歩 -インフラ環境構築の自動化について-
smt7174
1
140
Amplifyとゼロからはじめた AIコーディング。失敗と気づき
mkdev10
1
160
MagicPod MCPサーバー開発の裏側とAIエージェント活用の展望
magicpod
0
290
Kaigi Effect 2025 #rubykaigi2025_after
sue445
0
200
AIフレンドリーなプロダクト開発を目指して 〜MCPを橋渡しにした環境移行〜
shinpr
0
130
encoding/json v2を予習しよう!
yuyu_hf
PRO
1
220
ITベンダーから見る内製化支援の本質/in-house-dev
slsops
1
160
最近のRedmineの開発動向と次期バージョン6.1.0
vividtone
0
110
激動の一年を通じて見えてきた「技術でリードする」ということ
ktr_0731
8
8.3k
GPU 클라우드 환경에서의 회복탄력적 AI 운영 : 훈련 및 추론을 위한 견고한 아키텍처와 전략
inureyes
PRO
0
130
Serverlessだからこそコードと設計にはこだわろう
kenichirokimura
3
1.1k
MCP でモノが動くとおもしろい/It is interesting when things move with MCP
bitkey
3
610
Featured
See All Featured
RailsConf & Balkan Ruby 2019: The Past, Present, and Future of Rails at GitHub
eileencodes
137
33k
The MySQL Ecosystem @ GitHub 2015
samlambert
251
12k
Helping Users Find Their Own Way: Creating Modern Search Experiences
danielanewman
29
2.6k
Testing 201, or: Great Expectations
jmmastey
42
7.5k
The Psychology of Web Performance [Beyond Tellerrand 2023]
tammyeverts
47
2.8k
I Don’t Have Time: Getting Over the Fear to Launch Your Podcast
jcasabona
32
2.3k
Fantastic passwords and where to find them - at NoRuKo
philnash
51
3.2k
Creating an realtime collaboration tool: Agile Flush - .NET Oxford
marcduiker
30
2k
The Cost Of JavaScript in 2023
addyosmani
49
7.8k
Principles of Awesome APIs and How to Build Them.
keavy
126
17k
Templates, Plugins, & Blocks: Oh My! Creating the theme that thinks of everything
marktimemedia
30
2.4k
How to Ace a Technical Interview
jacobian
276
23k
Transcript
1 / 18 Neural Networks
2 / 18 1. NN !
• Residual Network • Batch Normalization 2. 1. • •
3 / 18 Plain NNs(&) ' pros #%
" (ex. CNN, RNN, ...) cons ! $ $
4 / 18 RNN RNN [1] P. Razvan et
al ,"On the difficulty of training recurrent neural networks." International Conference on Machine Learning. 2013. !"#$ !" %"&$ %"#$ %" %"&$ '() '() '() '*+, '*+, -!"# = /(!!"# ) -! -!$# %! : input !! : hidden state '%&' : '() : input / !" = '*+, 2 !"#$ + '() %"
5 / 18 !" !# !$ %" %# %$ &'(
&'( &'( &)*+ &)*+ ,! = .(!! ) ," ,# RNN 3 1, 12 = 1," 12 + 1,# 12 + 1,$ 12 1,$ 12 = 4 "565$ 1,$ 1!$ 7 1!$ 1!6 7 18!6 12 1!$ 1!" = 1!$ 1!# 7 1!# 1!" = &)*+ 9 :;<= >? !# 7 &)*+ 9 :;<= >? !" @A!B @C : !" ~!6E" fix !6
6 / 18 RNN Vanishing/Exploding Gradient : !"#$ !%&
'( )( … … )* '* ………… ………… +( +* !"#$ (-) !%& (-) '% …… '/ )/ +/
7 / 18 ,$+ /' !"#$ !- !"#$ 2 %
× '()* + ×%,- → # !"#$ !"#$ . 2 % × '()*(+).,-×%,- 1%input or 1)* Loss( RNN ."0& Vanishing/Exploding Gradient
8 / 18 +$ DeepNN( ! +
" )*&!/#% ' (→ ! Loss func ! Loss func → Residual Connection, Batch No malization
9 / 18 0), : Residual Connection – -– F(x)
"/#2 → "/ F(x) + x → (4 '$"/ Identity Mapping +%*1&: 3 . ! 3 Identity – [1] He, Kaiming, et al. "Identity mappings in deep residual networks." European Conference on Computer Vision. Springer, Cham, 2016.
10 / 18 : Residual Connection –– ' Forward
$#& Backward !$"& Deep % & input
11 / 18 Residual Connection –– https://icml.cc/2016/tutorials/icml2016_tutorial_deep_residual_networks_kaiminghe.pdf
12 / 18 ResNet Batch Normalization ResNet Residual Block
• ImplementationBatch Normalization NN ! $# • Batch Normalization" ## http://torch.ch/blog/2016/02/04/resnets.html Plain
13 / 18 ( ) 1 2
( ) n … Batch Normalization –Revisit Gaussian-
14 / 18 Batch Normalization -Input Data distribution
- (Convergence) !! Input NN → input
15 / 18 Batch Normalization -distribution - !"#$% & '
= ) & ' ← ' − , - ~/(,, -2) input
16 / 18 Batch Normalization Data distribution •
=(!, ")fix • Batch Normalization Batch Normalization
17 / 18 Batch Normalization – [2]Ioffe, Sergey,
and Christian Szegedy. "Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift." (2015). !, # !%$( → normalize scaling '"&# nomalize
18 / 18 DeepNN+ ! /
& -"#.#)%/'( *$ +!→ , Identity – normalize scaling implement Deep Net