Problems of Neural Networks and its solutions

Residual Connections とBatch Normalizationがメイン

1. 1 / 18 Neural Networks
       
2. 2 / 18 1. NN  !   
    • Residual Network • Batch Normalization 2. 1. 
   •   •  
3. 3 / 18 Plain NNs(&) '   pros #% "  (ex. CNN, RNN, ...) cons !  $
   $ 
4. 4 / 18 RNN
     RNN [1] P. Razvan et al ,"On the difficulty of training recurrent neural networks." International Conference on Machine Learning. 2013. !"#$ !" %"&$ %"#$ %" %"&$ '() '() '() '*+, '*+, -!"# = /(!!"# ) -! -!$# %! : input !! :   hidden state '%&' :   '() : input /   !" = '*+, 2 !"#$ + '() %"
5. 5 / 18 !" !# !$ %" %# %$ &'( &'( &'( &)*+ &)*+ ,! = .(!! ) ," ,# RNN
     3 1, 12 = 1," 12 + 1,# 12 + 1,$ 12 1,$ 12 = 4 "565$ 1,$ 1!$ 7 1!$ 1!6 7 18!6 12 1!$ 1!" = 1!$ 1!# 7 1!# 1!" = &)*+ 9 :;<= >? !# 7 &)*+ 9 :;<= >? !" @A!B @C : !" ~!6E" fix !6  
6. 6 / 18 RNN
     Vanishing/Exploding Gradient : !"#$ !%&  '( )( … … )* '* ………… ………… +( +* !"#$ (-) !%& (-) '% …… '/ )/ +/
7. 7 / 18 ,$+ /' !"#$ !-
   !"#$ 2 % × '()* + ×%,- → # !"#$    
   !"#$ . 2 % × '()*(+).,-×%,-  
   1%input   or 1)* 
   Loss(  RNN ."0& Vanishing/Exploding Gradient
8. 8 / 18 +$   DeepNN(  ! + " )*&!/#% ' 
   (→ ! Loss func ! Loss func   → Residual Connection, Batch No malization 
   
9. 9 / 18 0), : Residual Connection – -– F(x) "/#2 → "/ F(x) + x  → (4 '$"/ Identity Mapping +%*1&: 3 . 
   !   3  Identity – [1] He, Kaiming, et al. "Identity mappings in deep residual networks." European Conference on Computer Vision. Springer, Cham, 2016.
10. 10 / 18  : Residual Connection –
    – ' Forward   $#& Backward  !
    $"& Deep  %   & input
11. 11 / 18 Residual Connection –
    – https://icml.cc/2016/tutorials/icml2016_tutorial_deep_residual_networks_kaiminghe.pdf
12. 12 / 18 ResNet
    Batch Normalization ResNet Residual Block • ImplementationBatch Normalization NN ! $# • Batch Normalization"  
      ## http://torch.ch/blog/2016/02/04/resnets.html Plain
13. 13 / 18  (  ) 1  2  (
    ) n   … Batch Normalization –Revisit Gaussian-   
    
14. 14 / 18 Batch Normalization -Input Data distribution    -  (Convergence) 
    !! Input 
    NN  →  input 
     
15. 15 / 18 Batch Normalization -distribution 
    - !"#$% & ' = ) & ' ← ' − , - ~/(,, -2)   
      input  
16. 16 / 18 Batch Normalization Data distribution •  
    =(!, ")fix • Batch Normalization   
       Batch Normalization    
17. 17 / 18 Batch Normalization –
      [2]Ioffe, Sergey, and Christian Szegedy. "Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift." (2015). !, # !%$( →  normalize
     scaling '"&# 
    nomalize
18. 18 / 18    DeepNN+  ! / & -"#.#)%/'( *$ 
    +!→   ,  Identity – normalize  scaling implement  Deep Net