Deep Learning - an Introduction for Ruby Developers

Deep learning is exploding in popularity these days, powering everything from photo tagging to self driving cars – and you don’t need to hire a data science team or learn C++ to use machine learning models in your application. We'll learn some modern deep learning techniques, targeted at Ruby developers with no prior experience in the field. We’ll also cover the various tools available in the Ruby machine learning ecosystem.

Transcript

1. Deep Learning:
   An Introduction for
   Ruby Developers
   

   View Slide

2. @geoffreylitt
   

   View Slide

3. View Slide

4. View Slide

5. View Slide

6. Machine learning = learning
   functions from data
   Machine learning
   

   View Slide

7. View Slide

8. View Slide

9. View Slide

10. View Slide

11. What we’ll cover today
    • Some fundamental intuition for how machine
    learning works
    • Recent developments in deep neural networks
    • Ruby tools for machine learning, including a
    new emerging project
    

    View Slide

12. View Slide

13. Apartment  size  (sq ft) Monthly rent  ($)
    500 1000
    650 1200
    800 1650
    1000 2200
    1050 2250
    1200 2350
    1300 2600
    

    View Slide

14. 0
    500
    1000
    1500
    2000
    2500
    3000
    400 600 800 1000 1200 1400
    Monthly  rent
    Apartment  size
    

    View Slide

15. View Slide

16. View Slide

17. 0
    500
    1000
    1500
    2000
    2500
    3000
    400 600 800 1000 1200 1400
    Monthly  rent
    Apartment  size
    m    =    -­‐1
    b      =    2500
    Error = High
    

    View Slide

18. 0
    500
    1000
    1500
    2000
    2500
    3000
    400 600 800 1000 1200 1400
    Monthly  rent
    Apartment  size
    m    =    0
    b      =    1750
    Error = Lower
    

    View Slide

19. 0
    500
    1000
    1500
    2000
    2500
    3000
    400 600 800 1000 1200 1400
    Monthly  rent
    Apartment  size
    m    =      2.2
    b =      -­‐200
    Error = Low
    

    View Slide

20. m
    b
    Cost
    (aka how far off are we?)
    

    View Slide

21. View Slide

22. View Slide

23. View Slide

24. View Slide

25. Image from http://neuralnetworksanddeeplearning.com/
    

    View Slide

26. View Slide

27. inputs[0]
    inputs[1]
    inputs[2]
    output
    output
    

    View Slide

28. inputs[0]
    inputs[1]
    inputs[2]
    output
    output
    weights[0]
    weights[1]
    weights[2]
    

    View Slide

29. inputs[0]
    inputs[1]
    inputs[2]
    output
    output
    weights[0]
    weights[1]
    weights[2]
    

    View Slide

30. inputs[0]
    inputs[1]
    inputs[2]
    output
    output
    weights[0]
    weights[1]
    weights[2]
    

    View Slide

31. inputs[0]
    inputs[1]
    inputs[2]
    output
    output
    weights[0]
    weights[1]
    weights[2]
    

    View Slide

32. inputs[0]
    inputs[1]
    inputs[2]
    output
    output
    weights[0]
    weights[1]
    weights[2]
    

    View Slide

33. Image from http://cs231n.github.io/
    0.1    0.8    0.9    0.1
    0.0    0.2    0.8    0.1
    0.0    0.4    0.3    0.0
    0.1    0.5    0.1    0.0
    P(cat)
    P(dog)
    Learning Weights Weight updates flow backwards
    through the network
    Error = High
    

    View Slide

34. Image from http://cs231n.github.io/
    0.1    0.5    0.1    0.0
    0.1    0.2    0.3    0.6
    0.0    0.2    0.8    0.1
    0.5    0.7    0.7    0.8
    P(cat)
    P(dog)
    Learning Weights
    

    View Slide

35. Image from http://cs231n.github.io/
    0.1    0.5    0.1    0.0
    0.1    0.2    0.3    0.6
    0.0    0.2    0.8    0.1
    0.5    0.7    0.7    0.8
    P(cat)
    P(dog)
    Learning Weights
    Error = Lower
    

    View Slide

36. Image from http://cs231n.github.io/
    0.0    0.3    0.5    0.1
    0.1    0.2    0.1    0.0
    0.1    0.9    0.5    0.6
    0.5    0.9    0.7    0.8
    P(cat)
    P(dog)
    Learning Weights
    Error = Lower
    

    View Slide

37. Image from http://cs231n.github.io/
    P(cat)
    P(dog)
    Learning Weights
    Error = Low
    0.1    0.5    0.1    0.0
    0.1    0.2    0.3    0.6
    0.0    0.2    0.8    0.1
    0.5    0.7    0.7    0.8
    

    View Slide

38. “Deep Learning” =
    more layers
    = more data and compute
    needed to train
    = more parameters to train
    = things we have a ton
    of in 2016
    neural nets with…
    

    View Slide

39. Convolutional neural networks
    

    View Slide

40. Convolutional neural networks
    0%
    5%
    10%
    15%
    20%
    25%
    30%
    2011 2012 2013 2014 2015
    ImageNet Error  rate
    Human  benchmark
    AlexNet,  2012
    GoogLeNet,  2014
    

    View Slide

41. Recurrent neural networks
    Vinyals et  al  (2014).  Show  and  Tell:  A  Neural  
    Image  Caption  Generator. CoRR,  abs/1411.4555,  .
    

    View Slide

42. Recurrent neural networks
    Karpathy (2015).  Deep  Visual-­‐Semantic  Alignments  
    for  Generating  Image  Descriptions. CVPR
    

    View Slide

43. Recurrent neural networks
    http://karpathy.github.io/2015/05/21/rnn-­‐effectiveness/
    

    View Slide

44. Recurrent neural networks
    http://karpathy.github.io/2015/05/21/rnn-­‐effectiveness/
    

    View Slide

45. Deep Learning =
    Less Feature Engineering
    Fancy audio
    feature
    engineering
    Fancy text
    feature
    engineering
    

    View Slide

46. Deep Learning =
    Less Feature Engineering
    Early part of network
    learns features better
    than humans can create
    

    View Slide

47. View Slide

48. Option 1:
    Roll your own
    

    View Slide

49. Source: https://blog.intercom.io/machine-learning-way-easier-than-it-looks/
    

    View Slide

50. -
    +
    • Avoid new dependencies
    • Flexibility
    • Understand your solution
    • Lots of work
    • Testing/debugging
    • Performance
    

    View Slide

51. Option 2:
    Use a high-level
    library
    

    View Slide

52. https://davidcel.is/recommendable/
    

    View Slide

53. View Slide

54. -
    +
    • Quick results
    • No knowledge needed
    • Often basic algorithms
    • Only set use cases
    

    View Slide

55. Option 3:
    Use a low-level
    library
    

    View Slide

56. 0
    20
    40
    60
    80
    100
    120
    140
    160
    180
    200
    Number of machine learning libraries
    • scipy
    • numpy
    • NLTK
    • scikit-learn
    • keras
    • theano
    • caffe
    • tensorflow
    Source: https://github.com/josephmisiti/awesome-machine-learning
    

    View Slide

57. View Slide

58. ruby-fann (thin Ruby wrapper)
    libfann (written in C)
    

    View Slide

59. 0
    2000
    4000
    6000
    8000
    10000
    12000
    14000
    fann theano caffe
    Github Stars (9/5/16)
    

    View Slide

60. 0
    5000
    10000
    15000
    20000
    25000
    30000
    35000
    fann theano caffe tensorflow
    Github Stars (9/5/16) ?
    

    View Slide

61. View Slide

62. View Slide

63. View Slide

64. View Slide

65. C++
    Core graph execution (C++)
    Python
    Ruby
    “The system includes front-ends for
    specifying TensorFlow computations in
    Python and C++, and we expect other front-
    ends to be added over time in response to
    the desires of both internal Google users
    and the broader open-source community.”
    SWIG
    

    View Slide

66. View Slide

67. tensorflow.rb
    

    View Slide

68. View Slide

69. Next steps…
    • Complete enough functionality to train basic
    neural networks
    • Gradient descent optimizer
    • Create easy install flow for dependencies
    • Add more test coverage
    • (Still far from production ready)
    

    View Slide

70. To learn more…
    • Andrew Ng’s Machine Learning Coursera class
    • Google’s Udacity Tensorflow class
    • Contributors to tensorflow.rb welcome!:
    • github.com/somaticio/tensorflow.rb
    • I’ll tweet slides: @geoffreylitt
    

    View Slide

71. Thanks!
    

    View Slide