Deep Learning Frameworks: A journey through the many tools available in the Python ecosystem for Deep Learning.

This course is meant for research and practitioners who are interested in understand commonalities and differences of the many different tools nowadays available in the Python ecosystem for Machine and Deep Learning. Among the analysed frameworks, this talk will examine Theano (and Lasagne), TensorFlow, Keras, PyTorch and mxnet from a very technical perspective. This main goal of this talk is to introduce the many features of the frameworks, emphasising their pros and cons, as well as suggested use cases in which those tools may express their best.

Transcript

1. LOREM I P S U M DEEP LEARNING FRAMEWORKS A journey through the many tools available in the 
 Python ecosystem for Deep Learning vmaggio@fbk.eu @leriomaggio @leriomaggio
3. BACK T O T H E F U T U R E
4. SHAME
5. PYTHON & THE WEB
7. THE E V I L 
 E N E M Y
8. PYLAB
9. # PYLAB
12. MACHINE LEARNING IN PYTHON
13. ML PYTHON POWERED MACHINE LEARNING P Y T H O N github.com/josephmisiti/awesome-machine-learning#python
14. MACHINE LEARNING P Y T H O N Scala PyWrap + C++ PyWrap + Python Powered
15. Pure Python Numpy + Scipy Python + Cython
17. NLP & Machine Learning Supervised Learning (SVM) Neural Networks General Purpose
18. (General purpose) MACHINE LEARNING LIBRARIESwritten in Python
19. MACHINE LEARNING IN PYTHON +Deep
20. MACHINE (+DEEP) LEARNING LIBRARIES ZOO DEEP LEARNING P Y T H O N All the same?
21. MACHINE (+DEEP) LEARNING LIBRARIES ZOO NO All the same? HARDWARE
22. MACHINE (+DEEP) LEARNING LIBRARIES ZOO HARDWARE -ish All the same?
23. DEEP LEARNING HARDWARE
24. DEEP LEARNING HARDWARE
25. DL FRAMEWORKS
26. deeplearning.net/software/theano/
27. • Theano is the numerical computing workhorse that powers many of the other deep learning frameworks. • It was built by Frédéric Bastien and the excellent research team behind the University of Montreal’s lab, MILA. • Its API is quite low level • to write effective Theano you need to be quite familiar with the algorithms that are hidden away behind the scenes in other frameworks. • Theano is a go-to library if: • you have substantial academic machine learning expertise • you are looking for very fine grained control of your models, or want to implement a novel or unusual model. • In general, Theano trades ease of use for flexibility.
28. L O G I S T I C R E G R E S S I O N https://github.com/Newmu/Theano-Tutorials
29. • PROS • Flexibility • Performance (if used properly) • Inspirational for many other frameworks 
 (see next) • Framework of reference till ~2016 • CONS • No Longer Maintained • Substantial learning curve • Lower level API • Compiling complex symbolic graphs can be slow
30. B A S E D F R A M E W O R K S https://blocks.readthedocs.io/en/latest/
31. Pylearn2: a machine learning research library https://arxiv.org/abs/1308.4214
32. bit.ly/pydata2015-intro-to-lasagne lasagne.readthedocs.io/en/latest/ github.com/Lasagne/Recipes
33. bit.ly/pydata2015-intro-to-lasagne lasagne.readthedocs.io/en/latest/ github.com/Lasagne/Recipes
34. • PROS • Still very flexible • Higher layer of abstraction than Theano • Very well tested • CONS • Substantial learning curve (still) • Smaller Community
35. tensorflow.org/api_docs/python/
36. • TensorFlow is a blend between lower level, symbolic computation 
 (like Theano), and higher level, network specification libraries like Blocks and Lasagne. • (so far) TF has the largest active community - backed by Google Brain • It offers support for running machine/deep learning models: • across multiple GPUs • provides utilities for efficient data pipelining • visualisation, and serialisation of models. • (More recently), the TensorFlow team decided to incorporate support for Keras
39. L O G I S T I C R E G R E S S I O N
40. • PROS • Very large community • Low level & high level interfaces to network training • Faster model compilation than Theano-based options • Clean multi-GPU support • Distributed Computation/Training + Model Serving • Ready for Production • CONS • Steep learning curve • (quite) bad documentation • Non Numpy-friendly API • Difficult to Debug
41. TENSORBOARD
42. TENSORBOARD
43. TENSORBOARD: HISTOGRAMS
44. TENSORBOARD: GRAPH
45. DISTRIBU TED
46. KERAS Functional API
47. pyimagesearch.com/2018/10/08/keras-vs-tensorflow-which-one-is-better-and-which-one-should-i-learn/
49. pytorch.org
51. https://deepsense.ai/keras-vs-pytorch-avp-transfer-learning/ https://github.com/stared/Keras-PyTorch-AvP-transfer-learning
60. TOWARDS A (UNIQUE) DL API
65. LOREM I P S U M
67. Thanks a lot for you 
 kind attention @leriomaggio vmaggio@fbk.eu @leriomaggio