Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Deep Learning - an Introduction for Ruby Developers

Geoffrey Litt
September 13, 2016

Deep Learning - an Introduction for Ruby Developers

Deep learning is exploding in popularity these days, powering everything from photo tagging to self driving cars – and you don’t need to hire a data science team or learn C++ to use machine learning models in your application. We'll learn some modern deep learning techniques, targeted at Ruby developers with no prior experience in the field. We’ll also cover the various tools available in the Ruby machine learning ecosystem.

Geoffrey Litt

September 13, 2016
Tweet

More Decks by Geoffrey Litt

Other Decks in Programming

Transcript

  1. Deep Learning:
    An Introduction for
    Ruby Developers

    View Slide

  2. @geoffreylitt

    View Slide

  3. View Slide

  4. View Slide

  5. View Slide

  6. Machine learning = learning
    functions from data
    Machine learning

    View Slide

  7. View Slide

  8. View Slide

  9. View Slide

  10. View Slide

  11. What we’ll cover today
    • Some fundamental intuition for how machine
    learning works
    • Recent developments in deep neural networks
    • Ruby tools for machine learning, including a
    new emerging project

    View Slide

  12. View Slide

  13. Apartment  size  (sq ft) Monthly rent  ($)
    500 1000
    650 1200
    800 1650
    1000 2200
    1050 2250
    1200 2350
    1300 2600

    View Slide

  14. 0
    500
    1000
    1500
    2000
    2500
    3000
    400 600 800 1000 1200 1400
    Monthly  rent
    Apartment  size

    View Slide

  15. View Slide

  16. View Slide

  17. 0
    500
    1000
    1500
    2000
    2500
    3000
    400 600 800 1000 1200 1400
    Monthly  rent
    Apartment  size
    m    =    -­‐1
    b      =    2500
    Error = High

    View Slide

  18. 0
    500
    1000
    1500
    2000
    2500
    3000
    400 600 800 1000 1200 1400
    Monthly  rent
    Apartment  size
    m    =    0
    b      =    1750
    Error = Lower

    View Slide

  19. 0
    500
    1000
    1500
    2000
    2500
    3000
    400 600 800 1000 1200 1400
    Monthly  rent
    Apartment  size
    m    =      2.2
    b =      -­‐200
    Error = Low

    View Slide

  20. m
    b
    Cost
    (aka how far off are we?)

    View Slide

  21. View Slide

  22. View Slide

  23. View Slide

  24. View Slide

  25. Image from http://neuralnetworksanddeeplearning.com/

    View Slide

  26. View Slide

  27. inputs[0]
    inputs[1]
    inputs[2]
    output
    output

    View Slide

  28. inputs[0]
    inputs[1]
    inputs[2]
    output
    output
    weights[0]
    weights[1]
    weights[2]

    View Slide

  29. inputs[0]
    inputs[1]
    inputs[2]
    output
    output
    weights[0]
    weights[1]
    weights[2]

    View Slide

  30. inputs[0]
    inputs[1]
    inputs[2]
    output
    output
    weights[0]
    weights[1]
    weights[2]

    View Slide

  31. inputs[0]
    inputs[1]
    inputs[2]
    output
    output
    weights[0]
    weights[1]
    weights[2]

    View Slide

  32. inputs[0]
    inputs[1]
    inputs[2]
    output
    output
    weights[0]
    weights[1]
    weights[2]

    View Slide

  33. Image from http://cs231n.github.io/
    0.1    0.8    0.9    0.1
    0.0    0.2    0.8    0.1
    0.0    0.4    0.3    0.0
    0.1    0.5    0.1    0.0
    P(cat)
    P(dog)
    Learning Weights Weight updates flow backwards
    through the network
    Error = High

    View Slide

  34. Image from http://cs231n.github.io/
    0.1    0.5    0.1    0.0
    0.1    0.2    0.3    0.6
    0.0    0.2    0.8    0.1
    0.5    0.7    0.7    0.8
    P(cat)
    P(dog)
    Learning Weights

    View Slide

  35. Image from http://cs231n.github.io/
    0.1    0.5    0.1    0.0
    0.1    0.2    0.3    0.6
    0.0    0.2    0.8    0.1
    0.5    0.7    0.7    0.8
    P(cat)
    P(dog)
    Learning Weights
    Error = Lower

    View Slide

  36. Image from http://cs231n.github.io/
    0.0    0.3    0.5    0.1
    0.1    0.2    0.1    0.0
    0.1    0.9    0.5    0.6
    0.5    0.9    0.7    0.8
    P(cat)
    P(dog)
    Learning Weights
    Error = Lower

    View Slide

  37. Image from http://cs231n.github.io/
    P(cat)
    P(dog)
    Learning Weights
    Error = Low
    0.1    0.5    0.1    0.0
    0.1    0.2    0.3    0.6
    0.0    0.2    0.8    0.1
    0.5    0.7    0.7    0.8

    View Slide

  38. “Deep Learning” =
    more layers
    = more data and compute
    needed to train
    = more parameters to train
    = things we have a ton
    of in 2016
    neural nets with…

    View Slide

  39. Convolutional neural networks

    View Slide

  40. Convolutional neural networks
    0%
    5%
    10%
    15%
    20%
    25%
    30%
    2011 2012 2013 2014 2015
    ImageNet Error  rate
    Human  benchmark
    AlexNet,  2012
    GoogLeNet,  2014

    View Slide

  41. Recurrent neural networks
    Vinyals et  al  (2014).  Show  and  Tell:  A  Neural  
    Image  Caption  Generator. CoRR,  abs/1411.4555,  .

    View Slide

  42. Recurrent neural networks
    Karpathy (2015).  Deep  Visual-­‐Semantic  Alignments  
    for  Generating  Image  Descriptions. CVPR

    View Slide

  43. Recurrent neural networks
    http://karpathy.github.io/2015/05/21/rnn-­‐effectiveness/

    View Slide

  44. Recurrent neural networks
    http://karpathy.github.io/2015/05/21/rnn-­‐effectiveness/

    View Slide

  45. Deep Learning =
    Less Feature Engineering
    Fancy audio
    feature
    engineering
    Fancy text
    feature
    engineering

    View Slide

  46. Deep Learning =
    Less Feature Engineering
    Early part of network
    learns features better
    than humans can create

    View Slide

  47. View Slide

  48. Option 1:
    Roll your own

    View Slide

  49. Source: https://blog.intercom.io/machine-learning-way-easier-than-it-looks/

    View Slide

  50. -
    +
    • Avoid new dependencies
    • Flexibility
    • Understand your solution
    • Lots of work
    • Testing/debugging
    • Performance

    View Slide

  51. Option 2:
    Use a high-level
    library

    View Slide

  52. https://davidcel.is/recommendable/

    View Slide

  53. View Slide

  54. -
    +
    • Quick results
    • No knowledge needed
    • Often basic algorithms
    • Only set use cases

    View Slide

  55. Option 3:
    Use a low-level
    library

    View Slide

  56. 0
    20
    40
    60
    80
    100
    120
    140
    160
    180
    200
    Number of machine learning libraries
    • scipy
    • numpy
    • NLTK
    • scikit-learn
    • keras
    • theano
    • caffe
    • tensorflow
    Source: https://github.com/josephmisiti/awesome-machine-learning

    View Slide

  57. View Slide

  58. ruby-fann (thin Ruby wrapper)
    libfann (written in C)

    View Slide

  59. 0
    2000
    4000
    6000
    8000
    10000
    12000
    14000
    fann theano caffe
    Github Stars (9/5/16)

    View Slide

  60. 0
    5000
    10000
    15000
    20000
    25000
    30000
    35000
    fann theano caffe tensorflow
    Github Stars (9/5/16) ?

    View Slide

  61. View Slide

  62. View Slide

  63. View Slide

  64. View Slide

  65. C++
    Core graph execution (C++)
    Python
    Ruby
    “The system includes front-ends for
    specifying TensorFlow computations in
    Python and C++, and we expect other front-
    ends to be added over time in response to
    the desires of both internal Google users
    and the broader open-source community.”
    SWIG

    View Slide

  66. View Slide

  67. tensorflow.rb

    View Slide

  68. View Slide

  69. Next steps…
    • Complete enough functionality to train basic
    neural networks
    • Gradient descent optimizer
    • Create easy install flow for dependencies
    • Add more test coverage
    • (Still far from production ready)

    View Slide

  70. To learn more…
    • Andrew Ng’s Machine Learning Coursera class
    • Google’s Udacity Tensorflow class
    • Contributors to tensorflow.rb welcome!:
    • github.com/somaticio/tensorflow.rb
    • I’ll tweet slides: @geoffreylitt

    View Slide

  71. Thanks!

    View Slide