Deep Learning - an Introduction for Ruby Developers

7a6310a990212e2b392d95a67855afc2?s=47 Geoffrey Litt
September 13, 2016

Deep Learning - an Introduction for Ruby Developers

Deep learning is exploding in popularity these days, powering everything from photo tagging to self driving cars – and you don’t need to hire a data science team or learn C++ to use machine learning models in your application. We'll learn some modern deep learning techniques, targeted at Ruby developers with no prior experience in the field. We’ll also cover the various tools available in the Ruby machine learning ecosystem.

7a6310a990212e2b392d95a67855afc2?s=128

Geoffrey Litt

September 13, 2016
Tweet

Transcript

  1. Deep Learning: An Introduction for Ruby Developers

  2. @geoffreylitt

  3. None
  4. None
  5. None
  6. Machine learning = learning functions from data Machine learning

  7. None
  8. None
  9. None
  10. None
  11. What we’ll cover today • Some fundamental intuition for how

    machine learning works • Recent developments in deep neural networks • Ruby tools for machine learning, including a new emerging project
  12. None
  13. Apartment  size  (sq ft) Monthly rent  ($) 500 1000 650

    1200 800 1650 1000 2200 1050 2250 1200 2350 1300 2600
  14. 0 500 1000 1500 2000 2500 3000 400 600 800

    1000 1200 1400 Monthly  rent Apartment  size
  15. None
  16. None
  17. 0 500 1000 1500 2000 2500 3000 400 600 800

    1000 1200 1400 Monthly  rent Apartment  size m    =    -­‐1 b      =    2500 Error = High
  18. 0 500 1000 1500 2000 2500 3000 400 600 800

    1000 1200 1400 Monthly  rent Apartment  size m    =    0 b      =    1750 Error = Lower
  19. 0 500 1000 1500 2000 2500 3000 400 600 800

    1000 1200 1400 Monthly  rent Apartment  size m    =      2.2 b =      -­‐200 Error = Low
  20. m b Cost (aka how far off are we?)

  21. None
  22. None
  23. None
  24. None
  25. Image from http://neuralnetworksanddeeplearning.com/

  26. None
  27. inputs[0] inputs[1] inputs[2] output output

  28. inputs[0] inputs[1] inputs[2] output output weights[0] weights[1] weights[2]

  29. inputs[0] inputs[1] inputs[2] output output weights[0] weights[1] weights[2]

  30. inputs[0] inputs[1] inputs[2] output output weights[0] weights[1] weights[2]

  31. inputs[0] inputs[1] inputs[2] output output weights[0] weights[1] weights[2]

  32. inputs[0] inputs[1] inputs[2] output output weights[0] weights[1] weights[2]

  33. Image from http://cs231n.github.io/ 0.1    0.8    0.9    0.1

    0.0    0.2    0.8    0.1 0.0    0.4    0.3    0.0 0.1    0.5    0.1    0.0 P(cat) P(dog) Learning Weights Weight updates flow backwards through the network Error = High
  34. Image from http://cs231n.github.io/ 0.1    0.5    0.1    0.0

    0.1    0.2    0.3    0.6 0.0    0.2    0.8    0.1 0.5    0.7    0.7    0.8 P(cat) P(dog) Learning Weights
  35. Image from http://cs231n.github.io/ 0.1    0.5    0.1    0.0

    0.1    0.2    0.3    0.6 0.0    0.2    0.8    0.1 0.5    0.7    0.7    0.8 P(cat) P(dog) Learning Weights Error = Lower
  36. Image from http://cs231n.github.io/ 0.0    0.3    0.5    0.1

    0.1    0.2    0.1    0.0 0.1    0.9    0.5    0.6 0.5    0.9    0.7    0.8 P(cat) P(dog) Learning Weights Error = Lower
  37. Image from http://cs231n.github.io/ P(cat) P(dog) Learning Weights Error = Low

    0.1    0.5    0.1    0.0 0.1    0.2    0.3    0.6 0.0    0.2    0.8    0.1 0.5    0.7    0.7    0.8
  38. “Deep Learning” = more layers = more data and compute

    needed to train = more parameters to train = things we have a ton of in 2016 neural nets with…
  39. Convolutional neural networks

  40. Convolutional neural networks 0% 5% 10% 15% 20% 25% 30%

    2011 2012 2013 2014 2015 ImageNet Error  rate Human  benchmark AlexNet,  2012 GoogLeNet,  2014
  41. Recurrent neural networks Vinyals et  al  (2014).  Show  and  Tell:

     A  Neural   Image  Caption  Generator. CoRR,  abs/1411.4555,  .
  42. Recurrent neural networks Karpathy (2015).  Deep  Visual-­‐Semantic  Alignments   for

     Generating  Image  Descriptions. CVPR
  43. Recurrent neural networks http://karpathy.github.io/2015/05/21/rnn-­‐effectiveness/

  44. Recurrent neural networks http://karpathy.github.io/2015/05/21/rnn-­‐effectiveness/

  45. Deep Learning = Less Feature Engineering Fancy audio feature engineering

    Fancy text feature engineering
  46. Deep Learning = Less Feature Engineering Early part of network

    learns features better than humans can create
  47. None
  48. Option 1: Roll your own

  49. Source: https://blog.intercom.io/machine-learning-way-easier-than-it-looks/

  50. - + • Avoid new dependencies • Flexibility • Understand

    your solution • Lots of work • Testing/debugging • Performance
  51. Option 2: Use a high-level library

  52. https://davidcel.is/recommendable/

  53. None
  54. - + • Quick results • No knowledge needed •

    Often basic algorithms • Only set use cases
  55. Option 3: Use a low-level library

  56. 0 20 40 60 80 100 120 140 160 180

    200 Number of machine learning libraries • scipy • numpy • NLTK • scikit-learn • keras • theano • caffe • tensorflow Source: https://github.com/josephmisiti/awesome-machine-learning
  57. None
  58. ruby-fann (thin Ruby wrapper) libfann (written in C)

  59. 0 2000 4000 6000 8000 10000 12000 14000 fann theano

    caffe Github Stars (9/5/16)
  60. 0 5000 10000 15000 20000 25000 30000 35000 fann theano

    caffe tensorflow Github Stars (9/5/16) ?
  61. None
  62. None
  63. None
  64. None
  65. C++ Core graph execution (C++) Python Ruby “The system includes

    front-ends for specifying TensorFlow computations in Python and C++, and we expect other front- ends to be added over time in response to the desires of both internal Google users and the broader open-source community.” SWIG
  66. None
  67. tensorflow.rb

  68. None
  69. Next steps… • Complete enough functionality to train basic neural

    networks • Gradient descent optimizer • Create easy install flow for dependencies • Add more test coverage • (Still far from production ready)
  70. To learn more… • Andrew Ng’s Machine Learning Coursera class

    • Google’s Udacity Tensorflow class • Contributors to tensorflow.rb welcome!: • github.com/somaticio/tensorflow.rb • I’ll tweet slides: @geoffreylitt
  71. Thanks!