Introduction to Deep Learning & NLP - PyData London 2016

We take a crack at explaining the following topics:
1. What is deep learning?
2. Motivation: Some use cases where it has produced state-of-art results
3. Basic building blocks of Neural networks (Neuron, activation function, back propagation algorithm, gradient descent algorithm)
4. Supervised learning (multi-layer perceptron, recurrent neural network)
5. Introduction to word2vec
6. Introduction to Recurrent Neural Networks
7. Text classification using RNN
8. Impact of GPUs (Some practical thoughts on hardware and software)

Transcript

1. Introduction to Deep Learning Raghotham S Nischal HP www.unnati.xyz
2. AGENDA Motivation to Learning Introduction to Artificial Neural Networks Multi Layer Perceptron Deep Learning in Natural Language Processing Impact of GPU
3. MACHINE LEARNING
4. Machine Learning
5. What is Machine Learning ? Extracting features from data to solve predictive tasks
6. Use Cases Forecasting Recommendations Anomalies Classification Ranking Summarizing Decision making
7. LEARNING
8. How do we recognize digits? What is Learning?
9. 8 Computation Outputs Model source: http://www.slideshare.net/indicods/deep-learning-with-python-and-the-theano-library Inputs Machine Learning Framework
10. Recognizing Digit – An algorithm Use functions that compute relevant information to solve the problem
11. For each image, find “most similar” image. Guess that as the label. k Nearest-Neighbors Recognizing Digit – An algorithm Use functions that compute relevant information to solve the problem
12. Recognizing Digit – An algorithm Hard coded rules are brittle. Mostly better than a coin-toss. But far from being effective and accurate
13. LEARNING Biological Inspiration
14. Biological Inspiration Connected network of neurons Communicate by electrical and chemical signals ~ 1011 neurons ~1000 synapses per neuron
15. How is information detected? How is it stored? How does it influence recognition? Questions about Learning
16. Learnings from Neuro & Cognitive Science
17. Learnings from Neuro & Cognitive Science Kids talk grammatically correct sentences even before they are taught formal language. Kids learn after listening to a lot of sentences. See/hear/feel first. Assimilate. Build the context hierarchically. Recognize. Respond. Associations and Structural inferences. Understand context. Eg: Drinking water Vs River Vs Ocean
18. Simplified Brain’s Visual System source: ASDM Summer School on Deep Learning 2014 Hierarchical Structure: Several layers of representation Each layer builds on top of the previous one to obtain more complex representations
19. Importance of Connectionism Simple units interacting in a complex network Distributed representation of knowledge Parallelism Threshold mechanism for robust classification Mechanism for learning – adjusting synaptic weights Comprehend inner structure of observed data Lessons from Biological Learning
20. Example for Learning
21. Classify points to a curve source: http://colah.github.io/posts/2014-03-NN-Manifolds-Topology/ Two Curves on a plane
22. source: http://colah.github.io/posts/2014-03-NN-Manifolds-Topology/ Classify points to a curve Two Curves on a plane A simple model
23. source: http://colah.github.io/posts/2014-03-NN-Manifolds-Topology/ Classify points to a curve Two Curves on a plane A better model
24. Classify points to a curve source: http://colah.github.io/posts/2014-03-NN-Manifolds-Topology/ Obtained by transforming to a higher dimension A better model Two Curves on a plane
25. Visualization on learning separation source: http://colah.github.io/posts/2014-03-NN-Manifolds-Topology/
26. Now – a Bummer ☺ source: http://colah.github.io/posts/2014-03-NN-Manifolds-Topology/
27. Neural Network Building Blocks
29. INPUT ACTIVATION FUNCTION OUTPUT NEURON WEIGHTS Neuron, Activation Function
30. Single neuron is not enough INPUT OUTPUT Individual elements are weak computational elements Need: Networked elements
31. A Simple Neural Network source: https://en.wikipedia.org/wiki/Artificial_neural_network
32. Feed-forward Neural Network source: https://en.wikipedia.org/wiki/Artificial_neural_network Input layer of source nodes One or several hidden layers of processing neurons Output layer of processing neuron(s) Connections only between adjacent layers There are no feedback connections The network is formed by
33. Activation Function source: https://en.wikibooks.org/wiki/Artificial_Neural_Networks/Activation_Functions activation function of a node defines the output of that node given inputs
34. Perceptron A A a source: ASDM Summer School on Deep Learning 2014 Invented in 1957 Classifies input data into one of the output classes. If the weighted input is more than the threshold, classify as 1. Else 0 Online learning possible
35. Sigmoid/Logistic Output is bounded between 0 & 1 Domain: Complete set of Real numbers Smooth and continuous function Symmetric Derivative can be quickly calculated Positive, Bounded, strictly positive
36. Generalization of logistic regression for Multi-class classification Softmax
37. Rectified Linear Units Cheap to compute (no exponentials) Faster training Sparser networks Bounded below 0 Strictly increasing
38. source: ASDM Summer School on Deep Learning 2014 Max-out Generalization of Rectified Linear Max of k linear functions -> piece-wise linear At large k, can approximate a nonlinear function
39. Learning in ANN
40. Learning in ANN - Backpropagation source: ASDM Summer School on Deep Learning 2014
41. Backpropagation Algorithm - Math source: ASDM Summer School on Deep Learning 2014 M..AA...AA…TTTT.. HHH..
42. Backpropagation Algorithm Recursively and iteratively, the weight of the neuron that contributed most to the error gets penalized the most
43. Backpropagation Algorithm
44. Backpropagation Algorithm
45. Backpropagation Algorithm
46. Backpropagation Algorithm
47. Backpropagation Algorithm Computes gradient of the loss function w.r. to the weights. Backpropagate training error to generate deltas of all the neurons from hidden layers to output layer Use gradient descent to update weights
48. Goal: To find minimum of the loss function (minimize error of the model) Learning in ANN – Gradient Descent
49. Learning in ANN – Gradient Descent Depending on where we start, we can end up in different places.
50. Er…. How to compute gradient?
51. SGD/Mini-Batch Instead of using all of the training data, train iteratively on “mini-batches”
52. Online Learning Mini-batch size is 1. Weights are adjusted for every single data point.
53. Multi-Layer Perceptron Feedforward ANN Mostly sigmoid Can classify data that aren’t linearly separable
54. HANDS ON - MLP
55. Overfitting source:http://mathbabe.org/2012/11/20/columbia-data-science-course-week-12-predictive-modeling-data-leakage-model-evaluation/ Underfitted Model Good Model Overfitted Model
56. Some ways to address Overfitting Weight Decay L1/L2 Regularization Suitable Model Architectures Unsupervised Pre-training Dropout Data Augmentation
57. Dropout Cripple the network by removing hidden units stochastically In practice, probability of 0.5 works well. BEFORE DROPOUT source:http://info.usherbrooke.ca/hlarochelle/neural_networks/content.html
58. Dropout Cripple the network by removing hidden units stochastically In practice, probability of 0.5 works well. BEFORE DROPOUT source:http://info.usherbrooke.ca/hlarochelle/neural_networks/content.html AFTER DROPOUT
59. Natural Language Processing
60. Source: https://gist.github.com/nylki/1efbaa36635956d35bcc Automatic Generation of Cooking Recipe
61. Existing ML framework for NLP Input Text Linear Model (SVM, Softmax) Features (BOW, TFIDF, etc)
62. A Sentence source: http://web.stanford.edu/class/cs224n/handouts/CS224N_DeepNLP_Week7_lecture2.pdf
63. Structure important for humor & sarcasm Structure is very important A small crowd quietly enters the historic church A Historic crowd enters the small quietly church
64. Representing structure is hard n-grams quickly explodes
65. Limitation of the architectures so far Fixed Size Input (Eg: Image) Fixed computational steps (Eg: Number of layers) Fixed Size Output (Eg: probabilities of different classes) Hierarchy captured. But context, structure ? – mostly NOT !
66. Solution?
67. Solution? Vectorization
68. Convert text to numbers Vectorization One Hot Encoding WordNet Co-occurrence Matrix
69. Word2Vec Window based vectorization Learn surrounding words The model learns using the data instead of relying on any other kind of corpus Represents discrete state of the word Similar words are clustered together
70. Word2Vec - Skip Gram source: https://blog.acolyer.org/2016/04/21/the-amazing-power-of-word-vectors/
71. Word2Vec - CBoW Continuous Bag of Words source: https://blog.acolyer.org/2016/04/21/the-amazing-power-of-word-vectors/
72. Word2Vec source: http://deeplearning4j.org/word2vec
73. Hands on Vectorization
74. Recurrent Neural Networks
75. Recurrent Neural Network
76. One-to-One Sequence Vanilla-model. No RNN source: http://karpathy.github.io/2015/05/21/rnn-effectiveness/
77. One-to-Many Sequence Sequence output (Recognize image and explain it in words) source: http://karpathy.github.io/2015/05/21/rnn-effectiveness/
78. Many-to-One Sequence source: http://karpathy.github.io/2015/05/21/rnn-effectiveness/ Sequence input (Sentiment analysis of a sentence)
79. Many-to-Many Sequence source: http://karpathy.github.io/2015/05/21/rnn-effectiveness/ Sequence input (Machine Translation – English to Spanish)
80. RNN
81. RNN
82. LSTM Long Short Term Memory
83. LSTM
84. Effect of Dataset Size 1 thousand labeled examples perform 25-50% worse than linear model 1 Million labeled examples perform 0-30% better than linear model RNNs have poor generalization properties on small datasets RNNs have better generalization on large datasets.
85. Hands On - RNN
86. Graphical Processing Unit
88. Impact of GPU Compared to CPUs 20x speedups are typical Source: http://www.nvidia.com/object/what-is-gpu-computing.html GPUs have thousands of cores to process parallel workloads effectively
89. Impact of GPU Accelerated computations on float32 data Matrix multiplication, convolution, and large element-wise operations can be accelerated a lot Difficult to parallelize dense neural networks on multiple GPU efficiently (Active area of research) Copying of large quantities of data to and from a device is relatively slow CUDA has released cuDNN
90. SOME DEEP LEARNING EXAMPLES
91. Video Classification
92. Image Recognition source: http://www.youtube.com
93. Image Generation – Google Inceptionism source: http://googleresearch.blogspot.in/2015/06/inceptionism-going-deeper-into-neural.html
94. Natural Language Processing Handwriting Generator
95. Thank you www.unnati.xyz @unnati_xyz