OpenTalks.AI - Дмитрий Ветров, НейроДифуры: Что будет, если устремить число слоев в ResNet'ах к бесконечности или для чего в машинном обучении может пригодиться теория оптимального управления.

Transcript

1. NeuroDiffEquations: What will happen if the number of ResNet's layers will go to infinity? Dmitry Vetrov Head of Bayesian methods research group Research professor at HSE Head of ML lab in Samsung AI Center, Moscow http://bayesgroup.ru
2. Residual Networks K. He, X. Zhang, S. Ren, J. Sun. Deep Residual Learning for Image Recognition. CVPR 2016.
3. Analogy with Euler’s scheme
4. Neural ODE T. Chen, Y. Rubanova, J. Bettencourt, D. Duvenaud. Neural Ordinary Differential Equations. NIPS 2018.
5. Neural ODE T. Chen, Y. Rubanova, J. Bettencourt, D. Duvenaud. Neural Ordinary Differential Equations. NIPS 2018. Best Paper Award at NIPS 2018!
6. Neural ODE
7. Neural ODE Pontryagin theorem!
8. Continuous back-propagation
9. Continuous back-propagation
10. Continuous back-propagation
11. Continuous back-propagation
12. Continuous back-propagation
13. Continuous back-propagation
14. Discrete vs continuous back-prop
15. Discrete vs continuous back-prop
16. Differentiation w.r.t
17. Differentiation w.r.t
18. Advantages of neural ODE • Adaptive computation time
19. Advantages of neural ODE • Euler’s scheme is known to be unstable in numerical methods • In Neural ODE one can use more robust numerical schemes e.g. Runge-Kutta or implicit methods • This becomes possible after we realized that ResNets are simply neural ODEs
20. Advantages of neural ODE • Normalizing flows
21. Extensions of neural ODE