Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Realtime Predictive Analytics with scikit-learn & RabbitMQ by Michael Becker

PyCon 2014
April 11, 2014
1.9k

Realtime Predictive Analytics with scikit-learn & RabbitMQ by Michael Becker

PyCon 2014

April 11, 2014
Tweet

More Decks by PyCon 2014

Transcript

  1. Realtime predictive
    analytics
    using scikit-learn & RabbitMQ
    Michael Becker
    @beckerfuffle
    #pycon2014
    1

    View Slide

  2. Who Is This Guy?
    Data guy @ AWeber
    @beckerfuffle
    beckerfuffle.com
    These slides and more @ github.com/mdbecker
    @beckerfuffle
    #pycon2014
    2

    View Slide

  3. What My Coworkers
    Think I Do
    @beckerfuffle
    #pycon2014
    3

    View Slide

  4. What I Actually Do
    @beckerfuffle
    #pycon2014
    4

    View Slide

  5. What I'll Cover
    • Scikit-learn overview
    @beckerfuffle
    #pycon2014
    5

    View Slide

  6. What I'll Cover
    • Scikit-learn overview
    • Model Distribution
    @beckerfuffle
    #pycon2014
    6

    View Slide

  7. What I'll Cover
    • Scikit-learn overview
    • Model Distribution
    • Data flow
    @beckerfuffle
    #pycon2014
    7

    View Slide

  8. What I'll Cover
    • Scikit-learn overview
    • Model Distribution
    • Data flow
    • RabbitMQ
    @beckerfuffle
    #pycon2014
    8

    View Slide

  9. What I'll Cover
    • Scikit-learn overview
    • Model Distribution
    • Data flow
    • RabbitMQ
    • Demo
    @beckerfuffle
    #pycon2014
    9

    View Slide

  10. What I'll Cover
    • Scikit-learn overview
    • Model Distribution
    • Data flow
    • RabbitMQ
    • Demo
    • Scalability
    @beckerfuffle
    #pycon2014
    10

    View Slide

  11. What I'll Cover
    • Scikit-learn overview
    • Model Distribution
    • Data flow
    • RabbitMQ
    • Demo
    • Scalability
    • Other considerations
    @beckerfuffle
    #pycon2014
    11

    View Slide

  12. Supervised Learning
    @beckerfuffle
    #pycon2014
    12

    View Slide

  13. Supervised Learning
    @beckerfuffle
    #pycon2014
    13

    View Slide

  14. Supervised Learning
    @beckerfuffle
    #pycon2014
    14

    View Slide

  15. Supervised Learning
    @beckerfuffle
    #pycon2014
    15

    View Slide

  16. Supervised Learning
    @beckerfuffle
    #pycon2014
    16

    View Slide

  17. Supervised Learning
    @beckerfuffle
    #pycon2014
    17

    View Slide

  18. Supervised Learning
    @beckerfuffle
    #pycon2014
    18

    View Slide

  19. Supervised Learning
    @beckerfuffle
    #pycon2014
    19

    View Slide

  20. 38 Top Wikipedias
    Arabic: !"#$%&ا
    Bulgarian: Български
    Catalan: Català
    Czech: Čeština
    Danish: Dansk
    German: Deutsch
    English: English
    Spanish: Español
    Estonian: Eesti
    Basque: Euskara
    Persian: ()ر+,
    Finnish: Suomi
    French: Français
    Hebrew: תירבע
    Hindi: !ह#दी
    Croatian: Hrvatski
    Hungarian: Magyar
    Indonesian: Bahasa Indonesia
    Italian: Italiano
    Japanese: ೔ຊޠ
    Kazakh: Қазақша
    Korean: ೠҴয
    Lithuanian: Lietuvių
    Malay Bahasa: Melayu
    Dutch: Nederlands
    Norwegian: Norsk (Bokmål)
    Polish: Polski
    Portuguese: Português
    Romanian: Română
    Russian: Русский
    Slovak: Slovenčina
    Slovenian: Slovenščina
    Serbian: Српски / Srpski
    Swedish: Svenska
    Turkish: Türkçe
    Ukrainian: Українська
    Vietnamese: Tiếng Việt
    Waray-Waray: Winaray
    @beckerfuffle
    #pycon2014
    20

    View Slide

  21. Vectorizing Text
    @beckerfuffle
    #pycon2014
    21

    View Slide

  22. @beckerfuffle
    #pycon2014
    22

    View Slide

  23. @beckerfuffle
    #pycon2014
    23

    View Slide

  24. @beckerfuffle
    #pycon2014
    24

    View Slide

  25. @beckerfuffle
    #pycon
    25

    View Slide

  26. The Model
    @beckerfuffle
    #pycon2014
    26

    View Slide

  27. Supervised Learning
    @beckerfuffle
    #pycon2014
    27

    View Slide

  28. X_train
    @beckerfuffle
    #pycon2014
    28

    View Slide

  29. Supervised Learning
    @beckerfuffle
    #pycon2014
    29

    View Slide

  30. y_train
    @beckerfuffle
    #pycon2014
    30

    View Slide

  31. Dimensionality
    Reduction
    @beckerfuffle
    #pycon2014
    31

    View Slide

  32. Dimensionality
    Reduction
    @beckerfuffle
    #pycon2014
    32

    View Slide

  33. Dimensionality
    Reduction
    @beckerfuffle
    #pycon2014
    33

    View Slide

  34. Dimensionality
    Reduction
    @beckerfuffle
    #pycon2014
    34

    View Slide

  35. Dimensionality
    Reduction
    @beckerfuffle
    #pycon2014
    35

    View Slide

  36. Don’t Do What I Just Did!
    @beckerfuffle
    #pycon2014
    36

    View Slide

  37. Tuning
    @beckerfuffle
    #pycon2014
    37

    View Slide

  38. RandomizedSearchCV
    @beckerfuffle
    #pycon2014
    38

    View Slide

  39. RandomizedSearchCV
    @beckerfuffle
    #pycon2014
    39

    View Slide

  40. Distributing The Model
    @beckerfuffle
    #pycon2014
    40

    View Slide

  41. Data Input
    @beckerfuffle
    #pycon2014
    41

    View Slide

  42. The Client
    @beckerfuffle
    #pycon2014
    42

    View Slide

  43. Message Loss
    @beckerfuffle
    #pycon2014
    43

    View Slide

  44. Enter RabbitMQ
    Reliability
    Flexible Routing
    Clustering
    Many clients
    HA Queues
    @beckerfuffle
    #pycon2014
    44

    View Slide

  45. Enter RabbitMQ
    Reliability
    Flexible Routing
    Clustering
    Many clients
    HA Queues
    @beckerfuffle
    #pycon2014
    45

    View Slide

  46. Pika
    @beckerfuffle
    #pycon2014
    46

    View Slide

  47. Pika
    @beckerfuffle
    #pycon2014
    47

    View Slide

  48. @beckerfuffle
    #pycon2014
    48

    View Slide

  49. All the things!
    @beckerfuffle
    #pycon2014
    49

    View Slide

  50. Data Processing
    @beckerfuffle
    #pycon2014
    50

    View Slide

  51. The Consumer
    @beckerfuffle
    #pycon2014
    51

    View Slide

  52. The Consumer
    @beckerfuffle
    #pycon2014
    52

    View Slide

  53. The Consumer
    @beckerfuffle
    #pycon2014
    53

    View Slide

  54. The Consumer
    @beckerfuffle
    #pycon2014
    54

    View Slide

  55. The Design
    
    
    
         
    
    
    
    
    
    
    
    
    @beckerfuffle
    #pycon2014
    55

    View Slide

  56. The Design
    
    
    
         
    
    
    
    
    
    
    
    
    @beckerfuffle
    #pycon2014
    56

    View Slide

  57. The Design
    
    
    
         
    
    
    
    
    
    
    
    
    @beckerfuffle
    #pycon2014
    57

    View Slide

  58. The Design
    
    
    
         
    
    
    
    
    
    
    
    
    @beckerfuffle
    #pycon2014
    58

    View Slide

  59. The Design
    
    
    
         
    
    
    
    
    
    
    
    
    @beckerfuffle
    #pycon2014
    59

    View Slide

  60. The Design
    
    
    
         
    
    
    
    
    
    
    
    
    @beckerfuffle
    #pycon2014
    60

    View Slide

  61. The Design
    
    
    
         
    
    
    
    
    
    
    
    
    @beckerfuffle
    #pycon2014
    61

    View Slide

  62. The Design
    
    
    
         
    
    
    
    
    
    
    
    
    @beckerfuffle
    #pycon2014
    62

    View Slide

  63. The Design
    
    
    
         
    
    
    
    
    
    
    
    
    @beckerfuffle
    #pycon2014
    63

    View Slide

  64. Demo Time!
    @beckerfuffle
    #pycon2014
    64

    View Slide

  65. Scaling
    @beckerfuffle
    #pycon2014
    65

    View Slide

  66. Realtime vs Batch
    @beckerfuffle
    #pycon2014
    66

    View Slide

  67. Monitoring
    @beckerfuffle
    #pycon2014
    67

    View Slide

  68. Load
    @beckerfuffle
    #pycon2014
    68

    View Slide

  69. Re-verify
    @beckerfuffle
    #pycon2014
    69

    View Slide

  70. Re-verify
    @beckerfuffle
    #pycon2014
    70

    View Slide

  71. Thank You!
    API & Consumer: Kelly O’Brien (linkedin.com/in/kellyobie)
    UI: Matt Parke (ordinaryrobot.com)
    Classifier: Michael Becker (github.com/mdbecker)
    Images: Wikipedia; flickr.com users: kohsa, spenceyc, statefarm, klara, rh2ox, nasahqphoto, frickfrack
    @beckerfuffle
    #pycon2014
    71

    View Slide

  72. My Info
    Twitter: @beckerfuffle
    Blog: beckerfuffle.com
    These slides and more @ github.com/mdbecker
    @beckerfuffle
    #pycon2014
    72

    View Slide

  73. Demo time!
    @beckerfuffle
    #pycon2014
    73

    View Slide

  74. Demo time!
    @beckerfuffle
    #pycon2014
    74

    View Slide

  75. @beckerfuffle
    #pycon2014
    Demo time!
    75

    View Slide