Deep Learning on Java

Machine learning has undergone a flurry of progress recently thanks to the growth of big data, fast hardware, and clever algorithms. And with tools such as Spark and Hadoop, the JVM is no stranger to machine learning, using tools including h2o, dl4j and other Spark-based libraries. With these tools, developers can harness the power of distributed hardware and deep learning to discover new and untapped patterns and relationships in big data. In this session, learn how to train a classifier to recognize handwritten digits and how you can build your own models using open source data sets. No prior experience is required.

Transcript

1. Deep Learning on Java Breandan Considine JavaOne 2016
6. 0 1
7. Cool learning algorithm classify(datapoint, weights) :
8. Cool learning algorithm classify(datapoint, weights) : prediction = 0 for i from 0 to weights.size: prediction += weights[i] * datapoint[i]
9. Cool learning algorithm classify(datapoint, weights) : prediction = 0 for i from 0 to weights.size: prediction += weights[i] * datapoint[i] if prediction < 0 return 0 else return 1
10. Cool learning algorithm classify(datapoint, weights) : prediction = 0 for i from 0 to weights.size: prediction += weights[i] * datapoint[i] if prediction < 0 return 0 else return 1
11. Cool learning algorithm train(List of samples) : [x=1, y=0], output=1
12. Cool learning algorithm train(List of samples) : weights = array[samples[0].inputs.length + 1] [0, 0, 0]
13. Cool learning algorithm train(List of samples) : weights = array[samples[0].inputs.length + 1] while totalError is less than some threshold: totalError = 0 for each sample in samples :
14. Cool learning algorithm train(List of samples) : weights = array[samples[0].inputs.length + 1] while totalError is less than some threshold: totalError = 0 for each sample in samples : sample.input.prepend(1) // “Bias”
15. Cool learning algorithm train(List of samples) : weights = array[samples[0].inputs.length + 1] while totalError is less than some threshold: totalError = 0 for each sample in samples : sample.input.prepend(1) // “Bias” error = sample.output - classify(sample.input, weights)
16. Cool learning algorithm train(List of samples) : weights = array[samples[0].inputs.length + 1] while totalError is less than some threshold: totalError = 0 for each sample in samples : sample.input.prepend(1) // “Bias” error = sample.output - classify(sample.input, weights) for i from 0 to weights.length : weights[i] += RATE * error * sample.inputs[i]
17. Cool learning algorithm train(List of samples) : weights = array[samples[0].inputs.length + 1] while totalError is less than some threshold: totalError = 0 for each sample in samples : sample.input.prepend(1) // “Bias” error = sample.output - classify(sample.input, weights) for i from 0 to weights.length : weights[i] += RATE * error * sample.inputs[i] totalError += |error|
24. Even Cooler Algorithm! train(trainingSet) : initialize network weights randomly until average error stops decreasing (or you get tired): for each sample in trainingSet: prediction = network.output(sample) compute error (prediction – sample.output) compute error of (hidden -> output) layer weights compute error of (input -> hidden) layer weights update weights across the network save the weights
25. Gradient Descent http://cs231n.github.io/
30. Google Inception Model
31. Google Inception Model
32. Google Inception Model
33. ImageNet Large Scale Visual Recognition
34. Year over year Top-5 Recognition Error
35. Training your own model •Requirements •Clean, labeled data set •Clear decision problem •Patience and/or GPUs •Before you start
36. Common Mistakes •Training set – 70%/30% split •Test set – Do not show this to your model! •Sensitivity vs. specificity •Overfitting
39. Multi-layer Network Configuration MultiLayerConfiguration conf = new NeuralNetConfiguration.Builder() .optimizationAlgo(STOCHASTIC_GRADIENT_DESCENT) .learningRate(0.006) .list() .layer(0, new DenseLayer.Builder() .nIn(numRows * numColumns).nOut(1000) .activation("relu") .weightInit(WeightInit.XAVIER).build()) .layer(1, new Builder(NEGATIVELOGLIKELIHOOD) .nIn(1000).nOut(outputNum).activation("softmax") .weightInit(WeightInit.XAVIER).build()) .pretrain(false).backprop(true)
40. Model Initialization MultiLayerNetwork model = new MultiLayerNetwork(conf); model.init(); model.setListeners(Arrays.asList( new ScoreIterationListener(1), new HistogramIterationListener(1)));
42. Training the model DataSetIterator dataSetIterator = ... log.info("Training model..."); for(int i=0; i < numEpochs; i++) { model.fit(dataSetIterator); }
43. Evaluation log.info("Evaluating model...."); Evaluation evaluator = new Evaluation(outputNum); while(dataSetIterator.hasNext()){ DataSet next = dataSetIterator.next(); evalaluator.eval(next.getLabels(), output); } log.info(eval.stats());
44. Demo time!
46. References • Andrej Karpathy, CS231 Course Notes: http://cs231n.github.io/ • DL4j: https://github.com/deeplearning4j/deeplearning4j • Michael Nielsen, Neural Networks and Deep Learning, http://neuralnetworksanddeeplearning.com/ • Andrew Ng, Machine Learning class, Stanford/Coursera https://class.coursera.org/ml-003/lecture
47. Special Thanks JavaOne Program Committee Sharat Chandler Hanneli Tavante