Deep Learning on Java [DevNexus 2017]

Machine learning has recently made enormous progress in a variety of real-world applications, such as computer vision, speech recognition and language processing. And now, Java has the libraries to help you apply these techniques to large data sets, with new Spark-based tools like deeplearning4java (DL4J). In this workshop, you will learn the basic building blocks for deep learning: gradient descent, backpropogation, model training and evaluation. We’ll cover how to build and train supervised machine learning models, give you an overview of deep learning and show you how to recognize handwritten digits. You will gain an intuition for how to develop custom models to discover new insights and untapped patterns big data. No prior experience in machine learning is required.

Transcript

1. Deep Learning on Java Breandan Considine DevNexus 2017
2. Who am I? • Background in Computer Science, Machine Learning • Worked for a small ad-tech startup out of university • Spent two years as Developer Advocate @JetBrains • Interested in machine learning and speech recognition • Enjoy writing code, traveling to conferences, reading • Say hello! @breandan | breandan.net | bre@ndan.co
3. What is “three”?
4. Size Shape Distance Similarity Separation Orientation 3
5. What is “dog”?
7. Early Speech Recognition • Requires lots of handmade feature engineering • Poor results: >25% WER for HMM architectures
8. Automatic speech recognition in 2011
9. Year over year Top-5 Recognition Error
10. What happened? • Bigger data • Faster hardware • Smarter algorithms
11. What is machine learning? • Prediction • Categorization • Anomaly detection • Personalization • Adaptive control • Playing games
12. Traditional education • One-size-fits-all curriculum • Teaching process is repetitive • Students are not fully engaged • Memorization over understanding • Encouragement can be inconsistent • Teaches to the test (not the real world)
13. How can we improve education? • Personalized learning • Teaching assistance • Adaptive feedback • Active engagement • Spaced repetition • Assistive technology
19. Handwriting recognition
20. Handwriting recognition http://genekogan.com/works/a-book-from-the-sky/
21. Speech recognition
22. Speech Verification / Recitation
23. Speech Generation
24. https://erikbern.com/2016/01/21/analyzing-50k-fonts-using-deep-neural-networks/
25. https://handong1587.github.io/deep_learning/2015/10/09/image-generation.html
27. https://arxiv.org/abs/1609.04802
28. Machine learning, for humans • Self-improvement • Language learning • Computer training • Special education • Reading comprehension • Content generation
29. • A “tensor’ is just an n-dimensional array • Useful for working with complex data • We use (tiny) tensors every day! What’s a Tensor?
30. 't' What’s a Tensor? • A “tensor’ is just an n-dimensional array • Useful for working with complex data • We use (tiny) tensors every day!
31. 't' What’s a Tensor? • A “tensor’ is just an n-dimensional array • Useful for working with complex data • We use (tiny) tensors every day!
32. 't' What’s a Tensor? • A “tensor’ is just an n-dimensional array • Useful for working with complex data • We use (tiny) tensors every day!
33. 't' What’s a Tensor? • A “tensor’ is just an n-dimensional array • Useful for working with complex data • We use (tiny) tensors every day!
34. What’s a Tensor? 't' • A “tensor’ is just an n-dimensional array • Useful for working with complex data • We use (tiny) tensors every day!
35. NxM image is a point in RNM
36. https://inst.eecs.berkeley.edu/~cs194-26/fa14/upload/files/proj5/cs194-dm/
37. http://ai.stanford.edu/~wzou/emnlp2013_ZouSocherCerManning.pdf
38. http://www.snee.com/bobdc.blog/2016/09/semantic-web-semantics-vs-vect.html
39. https://arxiv.org/pdf/1301.3781.pdf
40. Types of machine learning
41. Supervised Learning
42. Supervised Learning
47. 0 1
48. y = mx + b
49. z = mx + ny + b
50. Cool learning algorithm def classify(datapoint, weights):
51. Cool learning algorithm def classify(datapoint, weights): y for x, y in prediction = sum(x * zip([1] + datapoint, weights))
52. Cool learning algorithm def classify(datapoint, weights): y for x, y in prediction = sum(x * zip([1] + datapoint, weights)) if prediction < 0: return 0 else: return 1
54. Cool learning algorithm def classify(datapoint, weights): y for x, y in prediction = sum(x * zip([1] + datapoint, weights)) if prediction < 0: return 0 else: return 1
55. Cool learning algorithm def train(data_set):
56. Cool learning algorithm def train(data_set): class Datum: def init (self, features, label): self.features = [1] + features self.label = label
57. Cool learning algorithm def train(data_set): weights = [0] * len(data_set[0].features) [0, 0, 0]
58. Cool learning algorithm def train(data_set): weights = [0] * len(data_set[0].features) total_error = threshold + 1
59. Cool learning algorithm def train(data_set): weights = [0] * len(data_set[0].features) total_error = threshold + 1 while total_error > threshold: total_error = 0 for item in data_set: weights) error = item.label – classify(item.features, weights = [w + RATE for w, i * error * i in zip(weights, item.features)] total_error += abs(error)
60. Cool learning algorithm def train(data_set): weights = [0] * len(data_set[0].features) total_error = threshold + 1 while total_error > threshold: total_error = 0 for item in data_set: weights) error = item.label – classify(item.features, weights = [w + RATE for w, i * error * i in zip(weights, item.features)] total_error += abs(error)
61. Cool learning algorithm def train(data_set): weights = [0] * len(data_set[0].features) total_error = threshold + 1 while total_error > threshold: total_error = 0 for item in data_set: weights) error = item.label – classify(item.features, weights = [w + RATE for w, i * error * i in zip(weights, item.features)] total_error += abs(error)
62. weights zip(weights, item.features)] Cool learning algorithm 1 i1 i2 in * * * * = [w + RATE * error * i for w, i in w0 w1 w2 wn Σ
63. Cool learning algorithm def train(data_set): weights = [0] * len(data_set[0].features) total_error = threshold + 1 while total_error > threshold: total_error = 0 for item in data_set: weights) error = item.label – classify(item.features, weights = [w + RATE for w, i * error * i in zip(weights, item.features)] total_error += abs(error)
64. Cool learning algorithm def train(data_set): weights = [0] * len(data_set[0].features) total_error = threshold + 1 while total_error > threshold: total_error = 0 for item in data_set: weights) error = item.label – classify(item.features, weights = [w + RATE for w, i * error * i in zip(weights, item.features)] total_error += abs(error)
66. Gradient Descent http://cs231n.github.io/
73. Backpropogation train(trainingSet) : initialize network weights randomly until average error stops decreasing (or you get tired): for each sample in trainingSet: prediction = network.output(sample) compute error (prediction – sample.output) compute error of (hidden -> output) layer weights compute error of (input -> hidden) layer weights update weights across the network save the weights
75. “Deep” neural networks
77. ImageNet LSVR Competition
78. What is a kernel? • A kernel is just a matrix • Used for edge detection, blurs, filters
79. Image Convolved Feature
85. Pooling (Downsampling)
86. Low level features
90. Convolutional neural network
91. Google Inception Model
92. Google Inception Model
93. Google Inception Model
96. Network Configuration MultiLayerConfiguration mlc = new NeuralNetConfiguration.Builder() .seed(12345) .optimizationAlgo(STOCHASTIC_GRADIENT_DESCENT) .iterations(1) .learningRate(0.006) .updater(NESTEROVS) .momentum(0.9) .regularization(true) .l2(1e-4) .list() …
97. Network Configuration … .layer(0, new DenseLayer.Builder() .nIn(28 * 28) // Number of input datapoints. .nOut(1000) // Number of output datapoints. .activation(Activation.RELU).weightInit(XAVIER) .build()) .layer(1, new OutputLayer.Builder(NEGATIVELOGLIKELIHOOD) .nIn(1000).nOut(10) .activation(SOFTMAX).weightInit(XAVIER).build()) .pretrain(false) .backprop(true) .build();
98. Model Initialization MultiLayerNetwork mlpNet = new MultiLayerNetwork(conf); mlpNet.init();
99. Training the model DataSetIterator dataSetIterator = ... for(int i=0; i < numEpochs; i++) { model.fit(dataSetIterator); }
100. Evaluation evaluator = new Evaluation(outputNum); while(testSetIterator.hasNext()){ DataSet next = dataSetIterator.next(); INDArray guesses = model.output(next.getFeatureMatrix(),false); INDArray realOutcomes = next.getLabels(); evalaluator.eval(, output); } log.info(eval.stats());
102. “A Neural Network Zoo,” Fjdor Van Neen http://www.asimovinstitute.org/neural-network-zoo/
103. Data Science/Engineering • Data selection • Data processing • Formatting & Cleaning • Sampling • Data transformation • Feature scaling & Normalization • Decomposition & Aggregation • Dimensionality reduction
105. Common Mistakes • Training set – 70%/30% split • Test set – Do not show this to your model! • Sensitivity vs. specificity • Overfitting
107. Training your own model •Requirements • Clean, labeled data set • Clear decision problem • Patience and/or GPUs •Before you start
108. Preparing data for ML •Generating Labels •Dimensionality reduction •Determining salient features •Visualizing the shape of your data •Correcting statistical bias •Getting data in the right format
109. Further resources • CS231 Course Notes • Deeplearning4j Examples • Visualizing MNIST • Neural Networks and Deep Learning • Andrew Ng’s Machine Learning class • Awesome Public Datasets • Hackers Guide to Neural Networks
110. Thank You! Mary, Mark, Margaret, Hanneli