Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Enjoy Deep Learning
 by JavaScript

yujiosaka
August 12, 2016

Enjoy Deep Learning
 by JavaScript

yujiosaka

August 12, 2016
Tweet

More Decks by yujiosaka

Other Decks in Technology

Transcript

  1. Yuji Isobe Enjoy Deep Learning
 by JavaScript TokyoJS 2.1@ Abeja

    INC https://speakerdeck.com/yujiosaka/hitasurale-sitedeipuraningu
  2. ZenClerk provides online customers
 with exciting shopping experience, personalized by

    machine learning
 to detect their growing desire to buy.
  3. OK! OK! OK! I understand… Sounds Cool!(ɾ㱼ɾ) Bayesian probability k-nearest

    neighbors Generalized linear model Neural Network Support Vector Machine Great Wall
  4. ✓ Classification of MNIST (handwritten digits data) ✓ 28 x

    28px ✓ 60,000 training data ✓ 10,000 test data 101 Digit Recognizer
  5. 1.Do not battle if not necessary 2.Do not steal items

    3.Do not pick up items Let’s Play FF6 with rules:
  6. 1.Use Deep Learning 2.Use Only JavaScript 3.Do not use machine

    learning libraries Let’s Play Kaggle with rules:
  7. ✓ Onlinebook ✓ History from Neural Network to Deep Learning

    ✓ Example implementation by Python on GitHub Neural Networks and Deep Learning
  8. Python→CoffeeScript→ES2015 sed & manual replacement Decaf JS &
 manual replacement

    Deep Learning Library Written in ES2015 JavaScript Babel first in NPM
  9. Python def update_mini_batch(self, mini_batch, eta): nabla_b = [np.zeros(b.shape) for b

    in self.biases] nabla_w = [np.zeros(w.shape) for w in self.weights] for x, y in mini_batch: delta_nabla_b, delta_nabla_w = self.backprop(x, y) nabla_b = [nb+dnb for nb, dnb in zip(nabla_b, delta_nabla_b)] nabla_w = [nw+dnw for nw, dnw in zip(nabla_w, delta_nabla_w)] self.weights = [w-(eta/len(mini_batch))*nw for w, nw in zip(self.weights, nabla_w)] self.biases = [b-(eta/len(mini_batch))*nb for b, nb in zip(self.biases, nabla_b)]
  10. CoffeeScript updateMiniBatch: (miniBatch, eta) -> nablaB = (Matrix.zeros(b.rows, b.cols) for

    b in @biases) nablaW = (Matrix.zeros(w.rows, w.cols) for w in @weights) for [x, y] in miniBatch [deltaNablaB, deltaNablaW] = @backprop(x, y) nablaB = (nb.plus(dnb) for [nb, dnb] in _.zip(nablaB, deltaNablaB)) nablaW = (nw.plus(dnw) for [nw, dnw] in _.zip(nablaW, deltaNablaW)) @weights = (w.minus(nw.mulEach(eta / miniBatch.length))
 for [w, nw] in _.zip(@weights, nablaW)) @biases = (b.minus(nb.mulEach(eta / miniBatch.length))
 for [b, nb] in _.zip(@biases, nablaB))
  11. numpy.nan_to_num nanToNum() { let thisData = this.data, rows = this.rows,

    cols = this.cols; let row, col, result = new Array(rows); for (row=0; row<rows; ++row) { result[row] = new Array(cols); for (col=0; col<cols; ++col) { result[row][col] = n2n(thisData[row][col]); } } return new Matrix(result); };
  12. numpy.ravel ravel() { let thisData = this.data, rows = this.rows,

    cols = this.cols; let a = new Array(rows * cols); for (let i = 0, jBase = 0; i<rows; ++i, jBase += cols) { for (let j = 0; j<cols; ++j) { a[jBase + j] = thisData[i][j]; } } return a; };
  13. b Perceptron Neuron Model x1 x2 x3 output w1 w2

    w3 PVUQVU JGЄKXKYKC≤
 JGЄKXKYKC
  14. 5 Perceptron Neuron Model Is the weather good? Does your


    girlfriend come? Is the place
 near stations? Go to the fest. 6 2 2 No Yes Yes No   ≤
  15. b Sigmoid Neuron Model x1 x2 x3 w1 w2 w3

    PVUQVU 
  FYQ ЄKXKYKC output
  16. ✓ Sigmoid function can produce 0 to 1 output ✓

    Small difference of input makes that of output ✓ In other words sigmoid function is differentiable What’s the difference?
  17. ✓ Improve accuracy by modifying weights (w) and bias (b)

    of each neuron ( ) ✓ Techniques like Back Propagation was invented
 for that purpose. Training neurons
  18. Why so popular? ✓ New techniques has been invented recently

    ✓ It can avoid overfitting when adding layers ✓ It can improve expression by adding layers
  19. ✓ Other Activation FunctionʢSoftmax/ReLUʣ ✓ Regularization (L2 Regularization/Dropout) ✓ Cross

    Entropy Cost Function ✓ Improving weight initialization Other techniques
  20. Deep Learning is
 a set of techniques There is no

    “Deep Learning Algorithm” You can improve accuracy by assembling many techniques
 like a jigsaw puzzle
  21. Problem 1
 Allergy to mathematical expression Once I wrote the

    code, It was actually easy to understand. function sigmoid(z) { return 1 / (1 + Math.exp(-z)); } let output = sigmoid(w.dot(a).plus(b)); 
  FYQ ЄKXKYKC
  22. I copied and pasted from StackOverflow answers,
 and it actually

    worked. costDelta(y) { this.outputDropout.minus(y); } Problem 2
 I didn’t know differentiation formula
  23. Softmax causes digit overflow if you follow textbooks.
 Again, I

    got answers from StackOverflow, and it worked.
 Problem 3
 Textbook didn’t tell me let max = _.max(vector), tmp = _.map(vector, (v) => { return Math.exp(v - max); }), sum = _.sum(tmp); return _.map(tmp, (v) => { return v / sum; });
  24. It only takes 1 hour by Python reference implementation,
 but

    mine by Node.js takes more than 24 hours. I learned that Numpy does some crazy tricks for you. Problem 4
 My computing speed is too slow I used small data set in development environment
  25. Implementations with Theano and TensorFlow are
 hard to reference because

    their API’s are too advanced. WTH is automatic differentiation!? Problem 5
 Python libraries are too sophisticated I became familiar with Python libraries
  26. WIP

  27. 1.To get GitHub stars (of course!) 2.To understand how deep

    learning works My initial motivations
  28. 1.You don’t have to train by JavaScript,
 but you may

    want to predict by it You can load data trained by Python,
 and use that for the prediction on browsers Promise.all([ jsmind.Netrowk.load(‘/path/to/layers.json'), jsmind.MnistLoader.loadTestDataWrapper() ]).spread(function(net, testData) { var accuracy = net.accuracy(testData); console.log('Test accuracy ' + accuracy); }); Load trained layers’ data