Deep Learning for Data Scientists

Neural networks have seen renewed interest from data scientists and machine learning researchers for their ability to accurately classify high-dimensional data, including images, sounds and text. In this session we will discuss the fundamental algorithms behind neural networks, such as back-propogation and gradient descent. We will develop an intuition for how to train a deep neural network using large data sets. We will then use the algorithms we have developed to train a simple handwritten digit recognizer, and illustrate how to generalize this same technique across larger types of images using convolutional neural nets. In the second and final part of this presentation, we will show you how to apply the same algorithms we have implemented using Keras and TensorFlow, a Python library for deep learning on large datasets. Attendees will learn how to implement a simple neural network, monitor its training progress and test for accuracy over time. Prior experience with Python and some basic algebra is a pre-requirement.

Transcript

1. Deep Learning for Data Scientists by Breandan Considine @breandan GIDS 2017
2. What is “three”?
3. Size Shape Distance Similarity Separation Orientation 3
4. What is “dog”?
6. Early Speech Recognition • Requires lots of handmade feature engineering • Poor results: >25% WER for HMM architectures
7. Automatic speech recognition in 2011
8. Year over year Top-5 Recognition Error
9. What happened? • Bigger data • Faster hardware • Smarter algorithms
10. Handwriting recognition
11. Handwriting recognition http://genekogan.com/works/a-book-from-the-sky/
12. Speech recognition
13. Speech Verification / Recitation
14. Speech Generation
15. https://erikbern.com/2016/01/21/analyzing-50k-fonts-using-deep-neural-networks/
16. https://handong1587.github.io/deep_learning/2015/10/09/image-generation.html
18. https://arxiv.org/abs/1609.04802
19. Machine learning, for humans • Self-improvement • Language learning • Computer training • Special education • Reading comprehension • Content generation
20. • A “tensor’ is just an n-dimensional array • Useful for working with complex data • We use (tiny) tensors every day! What’s a Tensor?
21. 't' What’s a Tensor? • A “tensor’ is just an n-dimensional array • Useful for working with complex data • We use (tiny) tensors every day!
22. 't' What’s a Tensor? • A “tensor’ is just an n-dimensional array • Useful for working with complex data • We use (tiny) tensors every day!
23. 't' What’s a Tensor? • A “tensor’ is just an n-dimensional array • Useful for working with complex data • We use (tiny) tensors every day!
24. 't' What’s a Tensor? • A “tensor’ is just an n-dimensional array • Useful for working with complex data • We use (tiny) tensors every day!
25. What’s a Tensor? 't' • A “tensor’ is just an n-dimensional array • Useful for working with complex data • We use (tiny) tensors every day!
26. NxM image is a point in RNM
27. https://inst.eecs.berkeley.edu/~cs194-26/fa14/upload/files/proj5/cs194-dm/
28. http://ai.stanford.edu/~wzou/emnlp2013_ZouSocherCerManning.pdf
29. http://www.snee.com/bobdc.blog/2016/09/semantic-web-semantics-vs-vect.html
30. https://arxiv.org/pdf/1301.3781.pdf
31. Types of machine learning
32. A quick taste of supervised learning
37. 0 1
38. Cool learning algorithm def classify(datapoint, weights):
39. Cool learning algorithm def classify(datapoint, weights): prediction = sum(x * y for x, y in zip([1] + datapoint, weights))
40. Cool learning algorithm def classify(datapoint, weights): prediction = sum(x * y for x, y in zip([1] + datapoint, weights)) if prediction < 0: return 0 else: return 1
41. Cool learning algorithm def classify(datapoint, weights): prediction = sum(x * y for x, y in zip([1] + datapoint, weights)) if prediction < 0: return 0 else: return 1
42. Cool learning algorithm def train(data_set):
43. Cool learning algorithm def train(data_set):
44. Cool learning algorithm def train(data_set): class Datum: def __init__(self, features, label): self.features = [1] + features self.label = label
45. Cool learning algorithm def train(data_set): weights = [0] * len(data_set[0].features) [0, 0, 0]
46. Cool learning algorithm def train(data_set): weights = [0] * len(data_set[0].features) total_error = threshold + 1
47. Cool learning algorithm def train(data_set): weights = [0] * len(data_set[0].features) total_error = threshold + 1 while total_error > threshold: total_error = 0 for item in data_set: error = item.label – classify(item.features, weights) weights = [w + RATE * error * i for w, i in zip(weights, item.features)] total_error += abs(error)
48. Cool learning algorithm def train(data_set): weights = [0] * len(data_set[0].features) total_error = threshold + 1 while total_error > threshold: total_error = 0 for item in data_set: error = item.label – classify(item.features, weights) weights = [w + RATE * error * i for w, i in zip(weights, item.features)] total_error += abs(error)
49. Cool learning algorithm def train(data_set): weights = [0] * len(data_set[0].features) total_error = threshold + 1 while total_error > threshold: total_error = 0 for item in data_set: error = item.label – classify(item.features, weights) weights = [w + RATE * error * i for w, i in zip(weights, item.features)] total_error += abs(error)
50. weights = [w + RATE * error * i for w, i in zip(weights, item.features)] Cool learning algorithm * 1 i1 i2 in * * * w0 w1 w2 wn Σ
51. Cool learning algorithm def train(data_set): weights = [0] * len(data_set[0].features) total_error = threshold + 1 while total_error > threshold: total_error = 0 for item in data_set: error = item.label – classify(item.features, weights) weights = [w + RATE * error * i for w, i in zip(weights, item.features)] total_error += abs(error)
52. Cool learning algorithm def train(data_set): weights = [0] * len(data_set[0].features) total_error = threshold + 1 while total_error > threshold: total_error = 0 for item in data_set: error = item.label – classify(item.features, weights) weights = [w + RATE * error * i for w, i in zip(weights, item.features)] total_error += abs(error)
59. Even Cooler Algorithm! (Backprop) train(trainingSet) : initialize network weights randomly until average error stops decreasing (or you get tired): for each sample in trainingSet: prediction = network.output(sample) compute error (prediction – sample.output) compute error of (hidden -> output) layer weights compute error of (input -> hidden) layer weights update weights across the network save the weights
60. Gradient Descent http://cs231n.github.io/
63. “Deep” neural networks
65. ImageNet LSVR Competition
66. What is a kernel? • A kernel is just a matrix • Used for edge detection, blurs, filters
67. Image Convolved Feature
73. Pooling (Downsampling)
74. Low level features
78. Convolutional neural network
79. Google Inception Model
80. Google Inception Model
81. Google Inception Model
83. “A Neural Network Zoo,” Fjdor Van Neen http://www.asimovinstitute.org/neural-network-zoo/
84. Data Science/Engineering • Data selection • Data processing • Formatting & Cleaning • Sampling • Data transformation • Feature scaling & Normalization • Decomposition & Aggregation • Dimensionality reduction
86. Common Mistakes • Training set – 70%/30% split • Test set – Do not show this to your model! • Sensitivity vs. specificity • Overfitting
88. Training your own model •Requirements • Clean, labeled data set • Clear decision problem • Patience and/or GPUs •Before you start, ask yourself: • Can I solve this problem more easily?
89. Preparing data for ML •Generating Labels •Dimensionality reduction •Determining salient features •Visualizing the shape of your data •Correcting statistical bias •Getting data in the right format
91. A brief look at unsupervised learning • Where did my labels go? • Mostly clustering, separation, association • Many different methods • Self organizing map • Expectation-maximization • Association rule learning • Reccomender systems
99. Data pre-processing • Data selection • Data processing • Formatting & Cleaning • Sampling • Data transformation • Feature scaling & Normalization • Decomposition & Aggregation • Dimensionality reduction
101. Principal Component Analysis
103. • CS231 Course Notes • Deeplearning4j Examples • Visualizing MNIST • Neural Networks and Deep Learning • Andrew Ng’s Machine Learning class • Awesome Public Datasets • Hackers Guide to Neural Networks Further resources
104. Further resources • Code for slides github.com/breandan/ml-exercises • Hacker’s Guide to Neural Networks, Andrej Karpathy • Neural Networks Demystified, Stephen Welch, • Machine Learning, Andrew Ng https://www.coursera.org/learn/machine-learnin • Awesome public data sets github.com/caesar0301/awesome-public-datasets
105. Special thanks to: • Saltmarch • O’Reilly Media • TensorFlow • Hanneli •