Deep Learning: An Introduction [Devoxx Belgium 2016]

Neural networks have seen renewed interest from data scientists and machine learning experts for their ability to accurately classify high-dimensional data. In this session we will discuss the fundamental algorithms behind neural networks, and develop an intuition for how to train a deep neural network using large data sets. We will then use the algorithms we have developed to train a simple handwritten digit recognizer, and illustrate how to generalize this technique to different images. In the second and final part, we will show you how to apply the same algorithms using DL4J, a Spark-based Java library for deep learning. You will learn how to implement a neural network, monitor it's training progress and test its accuracy over time. Prior experience with Java and some basic algebra is a pre-requirement.

Transcript

1. Deep Learning: An Introduction Breandan Considine Devoxx Belgium ’16
2. Who am I? • Background in Computer Science, Machine Learning • Worked for a small ad-tech startup out of university • Spent two years as Developer Advocate @JetBrains • Interested in machine learning and speech recognition • Enjoy writing code, traveling to conferences, reading • Say hello! @breandan | breandan.net | bre@ndan.co
3. Who am I? • Background in Computer Science, Machine Learning • Worked for a small ad-tech startup out of university • Spent two years as Developer Advocate @JetBrains • Interested in machine learning and speech recognition • Enjoy writing code, traveling to conferences, reading • Say hello! @breandan | breandan.net | bre@ndan.co
4. ImageNet Large Scale Visual Recognition
5. Year over year Top-5 Recognition Error
6. Automatic speech recognition in 2011
7. Automatic speech recognition in 2015 This image cannot currently be displayed.
8. What happened? • Bigger data • Faster hardware • Smarter algorithms
9. Traditional ASR • Requires lots of handmade feature engineering • Poor results: >25% WER for HMM architectures
10. What is Machine Learning?
11. Why machine learning? • Prediction • Categorization • Anomaly detection • Personalization • Adaptive control • Playing games
12. Traditional childhood education • One-size-fits-all curriculum • Teaching process is repetitive • Students are not fully engaged • Memorization over understanding • Encouragement can be inconsistent • Teaches to the test (not the real world)
13. How can we disrupt early education? • Personalized learning • Teaching assistance • Adaptive feedback • Active engagement • Spaced repetition • Assistive technology
15. How can we disrupt early education? • Personalized learning • Teaching assistance • Adaptive feedback • Active engagement • Spaced repetition • Assistive technology
18. Recall
21. How can we disrupt early education? • Personalized learning • Teaching assistance • Adaptive feedback • Active engagement • Spaced repetition • Assistive technology
23. Handwriting recognition
24. Speech recognition
25. How can we disrupt early education? • Personalized learning • Teaching assistance • Adaptive feedback • Active engagement • Spaced repetition • Assistive technology
27. How can we disrupt early education? • Personalized learning • Teaching assistance • Adaptive feedback • Active engagement • Spaced repetition • Assistive technology
28. Spaced repetition
29. How can we disrupt early education? • Personalized learning • Teaching assistance • Adaptive feedback • Active engagement • Spaced repetition • Assistive technology
30. Speech Verification / Recitation
31. Language learning
33. Machine learning, for humans • Self improvement • Language learning • Computer training • Special education • Reading comprehension • Content generation
34. Machine learning fundamentals • Tensors • Supervised learning • Unsupervised learning
35. So what’s a Tensor? • A “tensor’ is just an n-dimensional array • Useful for working with complex data • We use (tiny) tensors every day!
36. So what’s a Tensor anyway? • A “tensor’ is just an n-dimensional array • Useful for working with complex data • We use (tiny) tensors every day! 't'
37. So what’s a Tensor anyway? • A “tensor’ is just an n-dimensional array • Useful for working with complex data • We use (tiny) tensors every day! 't'
38. So what’s a Tensor anyway? • A “tensor’ is just an n-dimensional array • Useful for working with complex data • We use (tiny) tensors every day! 't'
39. So what’s a Tensor anyway? • A “tensor’ is just an n-dimensional array • Useful for working with complex data • We use (tiny) tensors every day! 't'
40. So what’s a Tensor anyway? • A “tensor’ is just an n-dimensional array • Useful for working with complex data • We use tiny (and large) tensors every day! 't'
41. Amplitude Frequency Time
42. NxM image is a point in RNM
43. https://inst.eecs.berkeley.edu/~cs194-26/fa14/upload/files/proj5/cs194-dm/
44. https://inst.eecs.berkeley.edu/~cs194-26/fa14/upload/files/proj5/cs194-dm/
45. https://inst.eecs.berkeley.edu/~cs194-26/fa14/upload/files/proj5/cs194-dm/
46. What are they good for? • Modeling complex systems, data sets • Capturing higher order correlations • Representing dynamic relationships • Doing machine learning!
47. Types of machine learning
48. A quick taste of supervised learning
49. Let’s have a look at linear regression! • • < Live Code >
50. Classification in a nutshell
55. 0 1
56. Cool learning algorithm def classify(datapoint, weights):
57. Cool learning algorithm def classify(datapoint, weights): prediction = sum(x * y for x, y in zip([1] + datapoint, weights))
58. Cool learning algorithm def classify(datapoint, weights): prediction = sum(x * y for x, y in zip([1] + datapoint, weights)) if prediction < 0: return 0 else: return 1
59. Cool learning algorithm def classify(datapoint, weights): prediction = sum(x * y for x, y in zip([1] + datapoint, weights)) if prediction < 0: return 0 else: return 1
60. Cool learning algorithm def train(data_set):
61. Cool learning algorithm def train(data_set):
62. Cool learning algorithm def train(data_set): class Datum: def __init__(self, features, label): self.features = [1] + features self.label = label
63. Cool learning algorithm def train(data_set): weights = [0] * len(data_set[0].features) [0, 0, 0]
64. Cool learning algorithm def train(data_set): weights = [0] * len(data_set[0].features) total_error = threshold + 1
65. Cool learning algorithm def train(data_set): weights = [0] * len(data_set[0].features) total_error = threshold + 1 while total_error > threshold: total_error = 0 for item in data_set: error = item.label – classify(item.features, weights) weights = [w + RATE * error * i for w, i in zip(weights, item.features)] total_error += abs(error)
66. Cool learning algorithm def train(data_set): weights = [0] * len(data_set[0].features) total_error = threshold + 1 while total_error > threshold: total_error = 0 for item in data_set: error = item.label – classify(item.features, weights) weights = [w + RATE * error * i for w, i in zip(weights, item.features)] total_error += abs(error)
67. Cool learning algorithm def train(data_set): weights = [0] * len(data_set[0].features) total_error = threshold + 1 while total_error > threshold: total_error = 0 for item in data_set: error = item.label – classify(item.features, weights) weights = [w + RATE * error * i for w, i in zip(weights, item.features)] total_error += abs(error)
68. weights = [w + RATE * error * i for w, i in zip(weights, item.features)] Cool learning algorithm * 1 i1 i2 in * * * w0 w1 w2 wn Σ
69. Cool learning algorithm def train(data_set): weights = [0] * len(data_set[0].features) total_error = threshold + 1 while total_error > threshold: total_error = 0 for item in data_set: error = item.label – classify(item.features, weights) weights = [w + RATE * error * i for w, i in zip(weights, item.features)] total_error += abs(error)
70. Cool learning algorithm def train(data_set): weights = [0] * len(data_set[0].features) total_error = threshold + 1 while total_error > threshold: total_error = 0 for item in data_set: error = item.label – classify(item.features, weights) weights = [w + RATE * error * i for w, i in zip(weights, item.features)] total_error += abs(error)
74. Even Cooler Algorithm! (Backprop) train(trainingSet) : initialize network weights randomly until average error stops decreasing (or you get tired): for each sample in trainingSet: prediction = network.output(sample) compute error (prediction – sample.output) compute error of (hidden -> output) layer weights compute error of (input -> hidden) layer weights update weights across the network save the weights
79. Gradient Descent http://cs231n.github.io/
81. “A Neural Network Zoo,” Fjdor Van Neen http://www.asimovinstitute.org/neural-network-zoo/
83. A brief look at unsupervised learning • Where did my labels go? • Mostly clustering, separation, association • Many different methods • Self organizing map • Expectation-maximization • Association rule learning • Reccomender systems
86. Cool clustering algorithm def cluster(data, k, max_it=1000):
87. Cool clustering algorithm def cluster(data, k, max_it=1000): labeled = np.append(data, np.zeros((len(data), 1)), axis=1) random_pts = np.random.choice(len(labeled), k, replace=False
88. Cool clustering algorithm def cluster(data, k, max_it=1000): labeled = np.append(data, np.zeros((len(data), 1)), axis=1) random_pts = np.random.choice(len(labeled), k, replace=False centers = labeled[random_pts] centers[:, -1] = range(1, k + 1) # Assign labels
89. Cool clustering algorithm def cluster(data, k, max_it=1000): labeled = np.append(data, np.zeros((len(data), 1)), axis=1) random_pts = np.random.choice(len(labeled), k, replace=False centers = labeled[random_pts] centers[:, -1] = range(1, k + 1) # Assign labels it = 0 old_centers = None while it < max_it and not np.array_equal(old_centers, centers): it += 1
90. Cool clustering algorithm def cluster(data, k, max_it=1000): labeled = np.append(data, np.zeros((len(data), 1)), axis=1) random_pts = np.random.choice(len(labeled), k, replace=False centers = labeled[random_pts] centers[:, -1] = range(1, k + 1) # Assign labels it = 0 old_centers = None while it < max_it and not np.array_equal(old_centers, centers): it += 1 old_centers = np.copy(centers)
91. Cool clustering algorithm def cluster(data, k, max_it=1000): labeled = np.append(data, np.zeros((len(data), 1)), axis=1) random_pts = np.random.choice(len(labeled), k, replace=False centers = labeled[random_pts] centers[:, -1] = range(1, k + 1) # Assign labels it = 0 old_centers = None while it < max_it and not np.array_equal(old_centers, centers): it += 1 old_centers = np.copy(centers) update_labels(labeled, centers)
92. Cool clustering algorithm def update_labels(data, centers): for datum in data: datum[-1] = centers[0, -1] min = distance.euclidean(datum[:-1], centers[0, :-1]) for center in centers: dist = distance.euclidean(datum[:-1], center[:-1]) if dist < min: min = dist datum[-1] = center[-1]
93. Cool clustering algorithm def update_labels(data, centers): for datum in data: datum[-1] = centers[0, -1] min = distance.euclidean(datum[:-1], centers[0, :-1]) for center in centers: dist = distance.euclidean(datum[:-1], center[:-1]) if dist < min: min = dist datum[-1] = center[-1]
94. Cool clustering algorithm def update_labels(data, centers): for datum in data: datum[-1] = centers[0, -1] min = distance.euclidean(datum[:-1], centers[0, :-1]) for center in centers: dist = distance.euclidean(datum[:-1], center[:-1]) if dist < min: min = dist datum[-1] = center[-1]
95. Cool clustering algorithm def update_labels(data, centers): for datum in data: datum[-1] = centers[0, -1] min = distance.euclidean(datum[:-1], centers[0, :-1]) for center in centers: dist = distance.euclidean(datum[:-1], center[:-1]) if dist < min: min = dist datum[-1] = center[-1]
96. Cool clustering algorithm def update_labels(data, centers): for datum in data: datum[-1] = centers[0, -1] min = distance.euclidean(datum[:-1], centers[0, :-1]) for center in centers: dist = distance.euclidean(datum[:-1], center[:-1]) if dist < min: min = dist datum[-1] = center[-1]
97. Cool clustering algorithm def cluster(data, k, max_it=1000): labeled = np.append(data, np.zeros((len(data), 1)), axis=1) random_pts = np.random.choice(len(labeled), k, replace=False centers = labeled[random_pts] centers[:, -1] = range(1, k + 1) # Assign labels it = 0 old_centers = None while it < max_it and not np.array_equal(old_centers, centers): it += 1 old_centers = np.copy(centers) update_labels(labeled, centers) update_centers(labeled, centers)
98. Cool clustering algorithm def update_centers(data, centers): k = len(centers) for i in range(1, k + 1): cluster = data[data[:, -1] == i, :-1] centers[i - 1, :-1] = np.mean(cluster, axis=0)
99. Cool clustering algorithm def update_centers(data, centers): k = len(centers) for i in range(1, k + 1): cluster = data[data[:, -1] == i, :-1] centers[i - 1, :-1] = np.mean(cluster, axis=0)
100. Cool clustering algorithm def update_centers(data, centers): k = len(centers) for i in range(1, k + 1): cluster = data[data[:, -1] == i, :-1] centers[i - 1, :-1] = np.mean(cluster, axis=0)
101. Cool clustering algorithm def update_centers(data, centers): k = len(centers) for i in range(1, k + 1): cluster = data[data[:, -1] == i, :-1] centers[i - 1, :-1] = np.mean(cluster, axis=0)
102. Cool clustering algorithm def update_centers(data, centers): k = len(centers) for i in range(1, k + 1): cluster = data[data[:, -1] == i, :-1] centers[i - 1, :-1] = np.mean(cluster, axis=0)
103. Cool clustering algorithm def cluster(data, k, max_it=1000): labeled = np.append(data, np.zeros((len(data), 1)), axis=1) random_pts = np.random.choice(len(labeled), k, replace=False centers = labeled[random_pts] centers[:, -1] = range(1, k + 1) # Assign labels it = 0 old_centers = None while it < max_it and not np.array_equal(old_centers, centers): it += 1 old_centers = np.copy(centers) update_labels(labeled, centers) update_centers(labeled, centers) return labeled
104. Cool clustering algorithm def cluster(data, k, max_it=1000): labeled = np.append(data, np.zeros((len(data), 1)), axis=1) random_pts = np.random.choice(len(labeled), k, replace=False centers = labeled[random_pts] centers[:, -1] = range(1, k + 1) # Assign labels it = 0 old_centers = None while it < max_it and not np.array_equal(old_centers, centers): it += 1 old_centers = np.copy(centers) update_labels(labeled, centers) update_centers(labeled, centers) return labeled
110. Data pre-processing • Data selection • Data processing • Formatting & Cleaning • Sampling • Data transformation • Feature scaling & Normalization • Decomposition & Aggregation • Dimensionality reduction
112. Principal Component Analysis
114. Reinforcement Learning • Agent has a context and set of choices • Each choice has an (unknown) reward • Goal: Maximize cumulative reward
115. Further resources • CS231 Course Notes • TensorFlow Models • Visualizing MNIST • Neural Networks and Deep Learning • Andrew Ng’s Machine Learning class • Awesome Public Datasets • Amy Unruh & Eli Bixby's TensorFlow Workshop
116. Thank You! Mary, Mark, Margaret, Hanneli