$30 off During Our Annual Pro Sale. View Details »

Enjoy Deep Learning
 by JavaScript

yujiosaka
August 12, 2016

Enjoy Deep Learning
 by JavaScript

yujiosaka

August 12, 2016
Tweet

More Decks by yujiosaka

Other Decks in Technology

Transcript

  1. Yuji Isobe
    Enjoy Deep Learning

    by JavaScript
    TokyoJS 2.1@ Abeja INC
    https://speakerdeck.com/yujiosaka/hitasurale-sitedeipuraningu

    View Slide

  2. [
    “Node.js”,
    “MongoDB”,
    “AngularJS”,
    “socket.io”,
    “React.js”,
    “Emotion Intelligence “

    ]
    @yujiosaka
    +BWB4DSJQU

    View Slide

  3. emin = Emotion Intelligence
    ؾ࣋ͪΛղ͢ΔςΫϊϩδʔͷ୳ڀ
    Emotion Intelligence͸ɺʮແҙࣝͷߦಈ͔Βɺ
    ਓͷؾ࣋ͪͷػඍΛղ͢Δ஌ੑʯΛɺਓ޻஌ೳ͓Αͼػցֶ
    शͷԠ༻ٕज़Λ༻͍ͯ
    ։ൃ͠ɺϏδωεʹԠ༻͍ͯ͠·͢ɻ
    In search for technology to

    understand human emotion

    View Slide

  4. ZenClerk Series
    [FODMFSLMJUF
    ;FODMFSL *OUFSFTU8JEHFU

    View Slide

  5. ZenClerk provides online customers

    with exciting shopping experience,
    personalized by machine learning

    to detect their growing desire to buy.

    View Slide

  6. I haven’t studied ML

    before… (´ɾωɾʆ)

    View Slide

  7. Introduction to ML

    View Slide

  8. OK!
    OK!
    OK!
    I understand…
    Sounds Cool!(ɾ㱼ɾ)
    Bayesian probability
    k-nearest neighbors
    Generalized linear model
    Neural Network
    Support Vector Machine
    Great Wall

    View Slide

  9. %BUB7JTVBMJ[BUJPO
    .BDIJOF-FBSOJOH
    .BUIFNBUJDT
    4UBUJTUJDT
    $PNQVUFS4DJFODF
    $PNNVOJDBUJPO
    %PNBJO,OPXMFEHF
    My skill set
    XBOUUPEFWFMPQ

    View Slide

  10. View Slide

  11. View Slide

  12. -FU`TUSZUIJT

    View Slide

  13. ✓ Classification of MNIST (handwritten digits data)
    ✓ 28 x 28px
    ✓ 60,000 training data
    ✓ 10,000 test data
    101 Digit Recognizer

    View Slide

  14. Aim for 99% accuracy
    http://yann.lecun.com/exdb/mnist/

    View Slide

  15. But it didn’t look fun at all

    View Slide

  16. I really wanted to enjoy it

    View Slide

  17. View Slide

  18. ͻͨ͢Βָͯ͠FF6

    View Slide

  19. 1.Do not battle if not necessary
    2.Do not steal items
    3.Do not pick up items
    Let’s Play FF6 with rules:

    View Slide

  20. 1.Use Deep Learning
    2.Use Only JavaScript
    3.Do not use machine learning libraries
    Let’s Play Kaggle with rules:

    View Slide

  21. Begin with Google Search

    View Slide

  22. IUUQOFVSBMOFUXPSLTBOEEFFQMFBSOJOHDPNJOEFYIUNM

    View Slide

  23. ✓ Onlinebook
    ✓ History from Neural Network to Deep Learning
    ✓ Example implementation by Python on GitHub
    Neural Networks and Deep Learning

    View Slide

  24. Make strategy

    View Slide

  25. Python→CoffeeScript→ES2015
    sed & manual replacement Decaf JS &

    manual replacement
    Deep Learning Library Written in ES2015
    JavaScript
    Babel
    first in NPM

    View Slide

  26. Isn’t Python and CoffeeScript

    very similar?

    View Slide

  27. Python
    def update_mini_batch(self, mini_batch, eta):
    nabla_b = [np.zeros(b.shape) for b in self.biases]
    nabla_w = [np.zeros(w.shape) for w in self.weights]
    for x, y in mini_batch:
    delta_nabla_b, delta_nabla_w = self.backprop(x, y)
    nabla_b = [nb+dnb for nb, dnb in zip(nabla_b, delta_nabla_b)]
    nabla_w = [nw+dnw for nw, dnw in zip(nabla_w, delta_nabla_w)]
    self.weights = [w-(eta/len(mini_batch))*nw
    for w, nw in zip(self.weights, nabla_w)]
    self.biases = [b-(eta/len(mini_batch))*nb
    for b, nb in zip(self.biases, nabla_b)]

    View Slide

  28. CoffeeScript
    updateMiniBatch: (miniBatch, eta) ->
    nablaB = (Matrix.zeros(b.rows, b.cols) for b in @biases)
    nablaW = (Matrix.zeros(w.rows, w.cols) for w in @weights)
    for [x, y] in miniBatch
    [deltaNablaB, deltaNablaW] = @backprop(x, y)
    nablaB = (nb.plus(dnb) for [nb, dnb] in _.zip(nablaB, deltaNablaB))
    nablaW = (nw.plus(dnw) for [nw, dnw] in _.zip(nablaW, deltaNablaW))
    @weights = (w.minus(nw.mulEach(eta / miniBatch.length))

    for [w, nw] in _.zip(@weights, nablaW))
    @biases = (b.minus(nb.mulEach(eta / miniBatch.length))

    for [b, nb] in _.zip(@biases, nablaB))

    View Slide

  29. Implement Numpy’s API

    View Slide

  30. numpy.nan_to_num
    nanToNum() {
    let thisData = this.data,
    rows = this.rows,
    cols = this.cols;
    let row, col, result = new Array(rows);
    for (row=0; rowresult[row] = new Array(cols);
    for (col=0; colresult[row][col] = n2n(thisData[row][col]);
    }
    }
    return new Matrix(result);
    };

    View Slide

  31. numpy.ravel
    ravel() {
    let thisData = this.data,
    rows = this.rows,
    cols = this.cols;
    let a = new Array(rows * cols);
    for (let i = 0, jBase = 0; ifor (let j = 0; ja[jBase + j] = thisData[i][j];
    }
    }
    return a;
    };

    View Slide

  32. https://github.com/juliankrispel/decaf

    View Slide

  33. Manual Replacement

    View Slide


  34. View Slide

  35. It worked…lol

    View Slide

  36. It’s about time to study

    View Slide

  37. χϡʔϥϧωοτϫʔΫ ਆܦճ࿏໢ɺӳOFVSBMOFUXPSL //
    ͸ɺ೴ػೳʹݟΒΕΔ͍͔ͭ͘ͷಛੑΛܭࢉػ
    ্ͷγϛϡϨʔγϣϯʹΑͬͯදݱ͢Δ͜ͱΛ໨ࢦͨ͠਺ֶϞσϧͰ͋Δɻ
    χϡʔϥϧωοτϫʔΫ8JLJQFEJB
    IUUQTKBXJLJQFEJBPSHXJLJχϡʔϥϧωοτϫʔΫ
    What is Neural Network?

    View Slide

  38. b
    Perceptron Neuron Model
    x1
    x2
    x3
    output
    w1
    w2
    w3
    PVUQVU
    JGЄKXKYKC≤

    JGЄKXKYKC

    View Slide

  39. 5
    Perceptron Neuron Model
    Is the weather good?
    Does your

    girlfriend come?
    Is the place

    near stations?
    Go to the fest.
    6
    2
    2
    No
    Yes
    Yes
    No

    View Slide

  40. b
    Sigmoid Neuron Model
    x1
    x2
    x3
    w1
    w2
    w3
    PVUQVU

    FYQ ЄKXKYKC

    output

    View Slide

  41. Step Function (Perceptron)

    View Slide

  42. Sigmoid Function

    View Slide

  43. ✓ Sigmoid function can produce 0 to 1 output
    ✓ Small difference of input makes that of output
    ✓ In other words sigmoid function is differentiable
    What’s the difference?

    View Slide

  44. Structure
    w + Δw

    b + Δb
    output + Δoutput

    View Slide

  45. ✓ Improve accuracy by modifying weights (w) and
    bias (b) of each neuron ( )
    ✓ Techniques like Back Propagation was invented

    for that purpose.
    Training neurons

    View Slide

  46. What is Deep Learning?

    View Slide

  47. Neural Network

    View Slide

  48. Deep Learning

    View Slide

  49. Why so popular?
    ✓ New techniques has been invented recently
    ✓ It can avoid overfitting when adding layers
    ✓ It can improve expression by adding layers

    View Slide

  50. Let’s implement it

    View Slide

  51. Convolutional Neural Network

    View Slide

  52. Problem
    The two images are recognized different to each other
    1px

    View Slide

  53. Solution

    View Slide

  54. Structure
    convolutional layer
    pooling layer

    View Slide

  55. ✓ Other Activation FunctionʢSoftmax/ReLUʣ
    ✓ Regularization (L2 Regularization/Dropout)
    ✓ Cross Entropy Cost Function
    ✓ Improving weight initialization
    Other techniques

    View Slide

  56. Deep Learning is

    a set of techniques
    There is no “Deep Learning Algorithm”
    You can improve accuracy by assembling many techniques

    like a jigsaw puzzle

    View Slide

  57. Problems I encountered

    and how I overcame it

    View Slide

  58. Problem 1

    Allergy to mathematical expression
    Once I wrote the code, It was actually easy to understand.
    function sigmoid(z) {
    return 1 / (1 + Math.exp(-z));
    }
    let output = sigmoid(w.dot(a).plus(b));

    FYQ ЄKXKYKC

    View Slide

  59. I copied and pasted from StackOverflow answers,

    and it actually worked.
    costDelta(y) {
    this.outputDropout.minus(y);
    }
    Problem 2

    I didn’t know differentiation formula

    View Slide

  60. Softmax causes digit overflow if you follow textbooks.

    Again, I got answers from StackOverflow, and it worked.

    Problem 3

    Textbook didn’t tell me
    let max = _.max(vector),
    tmp = _.map(vector, (v) => { return Math.exp(v - max); }),
    sum = _.sum(tmp);
    return _.map(tmp, (v) => { return v / sum; });

    View Slide

  61. It only takes 1 hour by Python reference implementation,

    but mine by Node.js takes more than 24 hours.
    I learned that Numpy does some crazy tricks for you.
    Problem 4

    My computing speed is too slow
    I used small data set in development environment

    View Slide

  62. Implementations with Theano and TensorFlow are

    hard to reference because their API’s are too advanced.
    WTH is automatic differentiation!?
    Problem 5

    Python libraries are too sophisticated
    I became familiar with Python libraries

    View Slide


  63. View Slide

  64. WIP

    View Slide

  65. IUUQTHJUIVCDPNZVKJPTBLBKTNJOE

    View Slide

  66. Demo

    View Slide

  67. 99.1% accuracy
    but it takes 24 hours to run

    View Slide

  68. Why did I do this?

    View Slide

  69. 1.To get GitHub stars (of course!)
    2.To understand how deep learning works
    My initial motivations

    View Slide

  70. I didn’t think it was useful

    but my mind changed

    View Slide

  71. Sometimes you want to

    do prediction on browsers

    View Slide

  72. 1.You don’t have to train by JavaScript,

    but you may want to predict by it
    You can load data trained by Python,

    and use that for the prediction on browsers
    Promise.all([
    jsmind.Netrowk.load(‘/path/to/layers.json'),
    jsmind.MnistLoader.loadTestDataWrapper()
    ]).spread(function(net, testData) {
    var accuracy = net.accuracy(testData);
    console.log('Test accuracy ' + accuracy);
    });
    Load trained layers’ data

    View Slide

  73. It is useful when you do
    online learning on Node.js

    View Slide

  74. I personally don’t like
    language lock-in (ʆ^´)

    View Slide

  75. Anyone should be able to

    do ML by any languages.

    View Slide

  76. Let’s Enjoy Deep Learning!

    View Slide