Tutorial: Deep Learning on Java (Jfokus 2017)

Machine learning has recently made enormous progress in a variety of real-world applications, such as computer vision, speech recognition and language processing. And now, Java has the libraries to help you apply these techniques to large data sets, with new Spark-based tools like deeplearning4java (DL4J). In this workshop, you will learn the basic building blocks for deep learning: gradient descent, backpropogation, model training and evaluation. We'll cover how to build and train supervised machine learning models, give you an overview of deep learning and show you how to recognize handwritten digits. You will gain an intuition for how to develop custom models to discover new insights and untapped patterns big data. No prior experience in machine learning is required.

Transcript

1. Deep  Learning on  Java Breandan Considine   JFokus  2017
2. Who  am I? • Background  in  Computer  Science,  Machine Learning • Worked  for  a  small  ad-­‐tech  startup  out  of university • Spent  two  years  as  Developer  Advocate @JetBrains • Interested  in  machine  learning  and  speech recognition • Enjoy  writing  code,  traveling  to  conferences, reading • Say  hello!  @breandan |  breandan.net | bre@ndan.co
3. What  is  learning?
4. What  is  “three”?
5. Size Shape Distance Similarity Separation Orientation 3
6. What  is  “dog”?
8. What  is  “Swedish”?
9. Early  Speech  Recognition • Requires  lots  of  handmade  feature engineering • Poor  results:  >25%  WER  for  HMM architectures
10. Automatic  speech  recognition  in 2011
11. Year  over  year  Top-­‐5  Recognition Error
12. What happened? • Bigger data • Faster hardware • Smarter algorithms
13. What  is  machine learning? • Prediction • Categorization • Anomaly detection • Personalization • Adaptive control • Playing games
15. Traditional education • One-­‐size-­‐fits-­‐all curriculum • Teaching  process  is repetitive • Students  are  not  fully engaged • Memorization  over understanding • Encouragement  can  be inconsistent • Teaches  to  the  test  (not  the  real  world)
16. How can we improve  education? • Personalized learning • Teaching assistance • Adaptive feedback • Active engagement • Spaced repetition • Assistive technology
18. How can we improve  education? • Personalized learning • Teaching assistance • Adaptive feedback • Active engagement • Spaced repetition • Assistive technology
21. Recall
24. How can we improve  education? • Personalized learning • Teaching assistance • Adaptive feedback • Active engagement • Spaced repetition • Assistive technology
26. Handwriting recognition
27. Speech recognition
28. • Personalized learning • Teaching assistance • Adaptive feedback • Active engagement • Spaced repetition • Assistive technology How can we improve  education?
30. • Personalized learning • Teaching assistance • Adaptive feedback • Active engagement • Spaced repetition • Assistive technology How can we impvove  education?
31. Spaced repetition
32. How can we impvove  education? • Personalized learning • Teaching assistance • Adaptive feedback • Active engagement • Spaced repetition • Assistive technology
33. Speech  Verification  / Recitation
34. Language learning
36. Machine  learning,  for humans • Self improvement • Language learning • Computer training • Special education • Reading comprehension • Content generation
37. • A  “tensor’  is  just  an  n-­‐dimensional array • Useful  for  working  with  complex data • We  use  (tiny)  tensors  every day! What’s  a  Tensor?
38. 't' What’s  a  Tensor? • A  “tensor’  is  just  an  n-­‐dimensional array • Useful  for  working  with  complex data • We  use  (tiny)  tensors  every day!
39. 't' What’s  a  Tensor? • A  “tensor’  is  just  an  n-­‐dimensional array • Useful  for  working  with  complex data • We  use  (tiny)  tensors  every day!
40. 't' What’s  a  Tensor? • A  “tensor’  is  just  an  n-­‐dimensional array • Useful  for  working  with  complex data • We  use  (tiny)  tensors  every day!
41. 't' What’s  a  Tensor? • A  “tensor’  is  just  an  n-­‐dimensional array • Useful  for  working  with  complex data • We  use  (tiny)  tensors  every day!
42. What’s  a  Tensor? 't' • A  “tensor’  is  just  an  n-­‐dimensional array • Useful  for  working  with  complex data • We  use  (tiny)  tensors  every day!
43. Types  of  machine learning
44. Supervised  Learning
45. Supervised  Learning
50. 0 1
51. y  =  mx  +  b
52. z  =  mx  +  ny  +  b
53. Cool  learning algorithm def classify(datapoint, weights):
54. Cool  learning algorithm def classify(datapoint, weights): y for x, y in prediction = sum(x * zip([1] + datapoint, weights))
55. Cool  learning algorithm def classify(datapoint, weights): y for x, y in prediction = sum(x * zip([1] + datapoint, weights)) if prediction < 0: return 0 else: return 1
57. Cool  learning algorithm def classify(datapoint, weights): y for x, y in prediction = sum(x * zip([1] + datapoint, weights)) if prediction < 0: return 0 else: return 1
58. Cool  learning algorithm def train(data_set):
59. Cool  learning algorithm def train(data_set): class Datum: def init (self, features, label): self.features = [1] + features self.label = label
60. Cool  learning algorithm def train(data_set): weights = [0] * len(data_set[0].features) [0, 0, 0]
61. Cool  learning algorithm def train(data_set): weights = [0] * len(data_set[0].features) total_error = threshold + 1
62. Cool  learning algorithm def train(data_set): weights = [0] * len(data_set[0].features) total_error = threshold + 1 while total_error > threshold: total_error = 0 for item in data_set: weights) error = item.label – classify(item.features, weights = [w + RATE for w, i * error * i in zip(weights, item.features)] total_error += abs(error)
63. Cool  learning algorithm def train(data_set): weights = [0] * len(data_set[0].features) total_error = threshold + 1 while total_error > threshold: total_error = 0 for item in data_set: weights) error = item.label – classify(item.features, weights = [w + RATE for w, i * error * i in zip(weights, item.features)] total_error += abs(error)
64. Cool  learning algorithm def train(data_set): weights = [0] * len(data_set[0].features) total_error = threshold + 1 while total_error > threshold: total_error = 0 for item in data_set: weights) error = item.label – classify(item.features, weights = [w + RATE for w, i * error * i in zip(weights, item.features)] total_error += abs(error)
65. weights zip(weights, item.features)] Cool  learning algorithm 1 i1 i2 in * * * * = [w + RATE * error * i for w, i in w0 w1 w2 wn Σ
66. Cool  learning algorithm def train(data_set): weights = [0] * len(data_set[0].features) total_error = threshold + 1 while total_error > threshold: total_error = 0 for item in data_set: weights) error = item.label – classify(item.features, weights = [w + RATE for w, i * error * i in zip(weights, item.features)] total_error += abs(error)
67. Cool  learning algorithm def train(data_set): weights = [0] * len(data_set[0].features) total_error = threshold + 1 while total_error > threshold: total_error = 0 for item in data_set: weights) error = item.label – classify(item.features, weights = [w + RATE for w, i * error * i in zip(weights, item.features)] total_error += abs(error)
75. Backpropogation train(trainingSet) : initialize network weights randomly until average error stops decreasing (or you get tired): for each sample in trainingSet: prediction = network.output(sample) compute error (prediction – sample.output) compute error of (hidden -> output) layer weights compute error of (input -> hidden) layer weights update weights across the network save the weights
76. Gradient  Descent http://cs231n.github.io/
77. “Deep”  neural  networks
79. NxM  image  is  a  point  in RNM
80. https://inst.eecs.berkeley.edu/~cs194-­‐26/fa14/upload/files/proj5/cs194-­‐dm/
81. https://inst.eecs.berkeley.edu/~cs194-­‐26/fa14/upload/files/proj5/cs194-­‐dm/
82. https://inst.eecs.berkeley.edu/~cs194-­‐26/fa14/upload/files/proj5/cs194-­‐dm/
83. ImageNet  LSVR  Competition
84. What  is  a  kernel? • A  kernel  is  just  a  matrix • Used  for  edge  detection,  blurs,  filters
89. Image Convolved Feature
90. Max  Pooling  (downsampling)
91. Low  level  features
93. Convolutional  neural  network
96. Multi-­‐layer  Network  Configuration MultiLayerConfiguration conf = new NeuralNetConfiguration.Builder() .seed(12345).optimizationAlgo(STOCHASTIC_GRADIENT_DESCENT) .iterations(1).learningRate(0.006).updater(NESTEROVS).momentum(0.9) .regularization(true).l2(1e-4).list().layer(0, new DenseLayer.Builder() .nIn(28 * 28) // Number of input datapoints. .nOut(1000) // Number of output datapoints. .activation(Activation.RELU).weightInit(XAVIER) .build()) .layer(1, new OutputLayer.Builder(NEGATIVELOGLIKELIHOOD) .nIn(1000) .nOut(10) .activation(SOFTMAX).weightInit(XAVIER).build()) .pretrain(false).backprop(true) .build();
97. Model  Initialization MultiLayerNetwork mlpNet = new MultiLayerNetwork(conf); mlpNet.init();
98. Training  the  model DataSetIterator dataSetIterator = ... for(int i=0; i < numEpochs; i++) { model.fit(dataSetIterator); }
99. Evaluation log.info("Evaluating model...."); evaluator = new Evaluation(outputNum); while(dataSetIterator.hasNext()){ DataSet next = dataSetIterator.next(); evalaluator.eval(next.getLabels(), output); } log.info(eval.stats());
101. “A  Neural  Network  Zoo,”  Fjdor  Van  Neen http://www.asimovinstitute.org/neural-­‐network-­‐zoo/
102. Google  Inception  Model
103. Google  Inception  Model
104. Google  Inception  Model
105. Data Science/Engineering • Data selection • Data processing • Formatting  & Cleaning • Sampling • Data transformation • Feature  scaling  & Normalization • Decomposition  & Aggregation • Dimensionality reduction
108. Common  Mistakes •Training  set  – 70%/30%  split •Test  set  – Do  not  show  this  to  your  model! •Sensitivity  vs.  specificity •Overfitting
109. Training  your  own  model •Requirements •Clean,  labeled  data  set •Clear  decision  problem •Patience  and/or  GPUs •Before  you  start
110. Preparing  data  for  ML •Generating  Labels •Dimensionality  reduction •Determining  salient  features •Visualizing  the  shape  of  your  data •Correcting  statistical  bias •Getting  data  in  the  right  format
111. Cool  clustering algorithm def cluster(data, k, max_it=1000):
112. Cool  clustering algorithm def cluster(data, k, max_it=1000): labeled = np.append(data, np.zeros((len(data), 1)), axis=1) random_pts = np.random.choice(len(labeled), k, replace=False
113. Cool  clustering algorithm def cluster(data, k, max_it=1000): labeled = np.append(data, np.zeros((len(data), 1)), axis=1) random_pts = np.random.choice(len(labeled), k, replace=False centers = labeled[random_pts] centers[:, -1] = range(1, k + 1) # Assign labels
114. Cool  clustering algorithm def cluster(data, k, max_it=1000): labeled = np.append(data, np.zeros((len(data), 1)), axis=1) random_pts = np.random.choice(len(labeled), k, replace=False centers = labeled[random_pts] centers[:, -1] = range(1, k + 1) # Assign labels it = 0 old_centers = None < max_it and not np.array_equal(old_centers, centers): while it it += 1
115. Cool  clustering algorithm def cluster(data, k, max_it=1000): labeled = np.append(data, np.zeros((len(data), 1)), axis=1) random_pts = np.random.choice(len(labeled), k, replace=False centers = labeled[random_pts] centers[:,-1] = range(1, k + 1) # Assign labels it = 0 old_centers = None not np.array_equal(old_centers, centers): while it < max_it and it += 1 old_centers = np.copy(centers)
116. Cool  clustering algorithm def cluster(data, k, max_it=1000): labeled = np.append(data, np.zeros((len(data), 1)), axis=1) random_pts = np.random.choice(len(labeled), k, replace=False centers = labeled[random_pts] centers[:, -1] = range(1, k + 1) # Assign labels it = 0 old_centers = None max_it and not np.array_equal(old_centers, centers): while it < it += 1 old_centers = np.copy(centers) update_labels(labeled, centers)
117. Cool  clustering algorithm def update_labels(data, centers): for datum in data: datum[-1] = centers[0, -1] centers[0, :-1]) min = distance.euclidean(datum[:-1], for center in centers: dist = distance.euclidean(datum[:-1], center[:-1]) if dist min = < min: dist datum[-1] = center[-1]
118. Cool  clustering algorithm def update_labels(data, centers): for datum in data: datum[-1] = centers[0, -1] centers[0, :-1]) min = distance.euclidean(datum[:-1], for center in centers: dist = distance.euclidean(datum[:-1], center[:-1]) if dist min = < min: dist datum[-1] = center[-1]
119. Cool  clustering algorithm def update_labels(data, centers): for datum in data: datum[-1] = centers[0, -1] min = distance.euclidean(datum[:-1], centers[0, :-1]) for center in centers: dist = distance.euclidean(datum[:-1], center[:-1]) if dist min = < min: dist datum[-1] = center[-1]
120. Cool  clustering algorithm def update_labels(data, centers): for datum in data: datum[-1] = centers[0, -1] min = distance.euclidean(datum[:-1], centers[0, :-1]) for center in centers: dist = distance.euclidean(datum[:-1], center[:-1]) if dist min = < min: dist datum[-1] = center[-1]
121. Cool  clustering algorithm def update_labels(data, centers): for datum in data: datum[-1] = centers[0, -1] centers[0, :-1]) min = distance.euclidean(datum[:-1], for center in centers: dist = distance.euclidean(datum[:-1], center[:-1]) if dist min = < min: dist datum[-1] = center[-1]
122. Cool  clustering algorithm def cluster(data, k, max_it=1000): labeled = np.append(data, np.zeros((len(data), 1)), axis=1) random_pts = np.random.choice(len(labeled), k, replace=False centers = labeled[random_pts] centers[:, -1] = range(1, k + 1) # Assign labels it = 0 old_centers = None max_it and not np.array_equal(old_centers, centers): while it < it += 1 old_centers = np.copy(centers) update_labels(labeled, centers) update_centers(labeled,centers)
123. Cool  clustering algorithm def update_centers(data, centers): k = len(centers) for i in range(1, k + 1): cluster = centers[i data[data[:, -1] == i, :-1] - 1, :-1] = np.mean(cluster, axis=0)
124. Cool  clustering algorithm def update_centers(data, centers): k = len(centers) for i in range(1, k + 1): cluster = centers[i data[data[:, -1] == i, :-1] - 1, :-1] = np.mean(cluster, axis=0)
125. Cool  clustering algorithm def update_centers(data, centers): k = len(centers) for i in range(1, k + 1): cluster = centers[i data[data[:, -1] == i, :-1] - 1, :-1] = np.mean(cluster, axis=0)
126. Cool  clustering algorithm def update_centers(data, centers): k = len(centers) for i in range(1, k + 1): cluster = centers[i data[data[:, -1] == i, :-1] - 1, :-1] = np.mean(cluster, axis=0)
127. Cool  clustering algorithm def update_centers(data, centers): k = len(centers) for i in range(1, k + 1): cluster = centers[i data[data[:, -1] == i, :-1] - 1, :-1] = np.mean(cluster, axis=0)
128. Cool  clustering algorithm def cluster(data, k, max_it=1000): labeled = np.append(data, np.zeros((len(data), 1)), axis=1) random_pts = np.random.choice(len(labeled), k, replace=False centers = labeled[random_pts] centers[:, -1] = range(1, k + 1) # Assign labels it = 0 old_centers = None max_it and not np.array_equal(old_centers, centers): while it < it += 1 old_centers = np.copy(centers) update_labels(labeled, centers) update_centers(labeled,centers) return labeled
129. Cool  clustering algorithm def cluster(data, k, max_it=1000): labeled = np.append(data, np.zeros((len(data), 1)), axis=1) random_pts = np.random.choice(len(labeled), k, replace=False centers = labeled[random_pts] centers[:, -1] = range(1, k + 1) # Assign labels it = 0 old_centers = None max_it and not np.array_equal(old_centers, centers): while it < it += 1 old_centers = np.copy(centers) update_labels(labeled, update_centers(labeled, centers) centers) return labeled
134. Further resources • CS231  Course Notes • Deeplearning4j  Examples • Visualizing MNIST • Neural  Networks  and  Deep  Learning • Andrew  Ng’s  Machine  Learning class • Awesome  Public Datasets
135. Thank You! Mary,  Mark,  Margaret, Hanneli