Realtime Predictive Analytics with scikit-learn & RabbitMQ by Michael Becker

D21717ea76044d31115c573d368e6ff4?s=47 PyCon 2014
April 11, 2014
1.8k

Realtime Predictive Analytics with scikit-learn & RabbitMQ by Michael Becker

D21717ea76044d31115c573d368e6ff4?s=128

PyCon 2014

April 11, 2014
Tweet

Transcript

  1. Realtime predictive analytics using scikit-learn & RabbitMQ Michael Becker @beckerfuffle

    #pycon2014 1
  2. Who Is This Guy? Data guy @ AWeber @beckerfuffle beckerfuffle.com

    These slides and more @ github.com/mdbecker @beckerfuffle #pycon2014 2
  3. What My Coworkers Think I Do @beckerfuffle #pycon2014 3

  4. What I Actually Do @beckerfuffle #pycon2014 4

  5. What I'll Cover • Scikit-learn overview @beckerfuffle #pycon2014 5

  6. What I'll Cover • Scikit-learn overview • Model Distribution @beckerfuffle

    #pycon2014 6
  7. What I'll Cover • Scikit-learn overview • Model Distribution •

    Data flow @beckerfuffle #pycon2014 7
  8. What I'll Cover • Scikit-learn overview • Model Distribution •

    Data flow • RabbitMQ @beckerfuffle #pycon2014 8
  9. What I'll Cover • Scikit-learn overview • Model Distribution •

    Data flow • RabbitMQ • Demo @beckerfuffle #pycon2014 9
  10. What I'll Cover • Scikit-learn overview • Model Distribution •

    Data flow • RabbitMQ • Demo • Scalability @beckerfuffle #pycon2014 10
  11. What I'll Cover • Scikit-learn overview • Model Distribution •

    Data flow • RabbitMQ • Demo • Scalability • Other considerations @beckerfuffle #pycon2014 11
  12. Supervised Learning @beckerfuffle #pycon2014 12

  13. Supervised Learning @beckerfuffle #pycon2014 13

  14. Supervised Learning @beckerfuffle #pycon2014 14

  15. Supervised Learning @beckerfuffle #pycon2014 15

  16. Supervised Learning @beckerfuffle #pycon2014 16

  17. Supervised Learning @beckerfuffle #pycon2014 17

  18. Supervised Learning @beckerfuffle #pycon2014 18

  19. Supervised Learning @beckerfuffle #pycon2014 19

  20. 38 Top Wikipedias Arabic: !"#$%&ا Bulgarian: Български Catalan: Català Czech:

    Čeština Danish: Dansk German: Deutsch English: English Spanish: Español Estonian: Eesti Basque: Euskara Persian: ()ر+, Finnish: Suomi French: Français Hebrew: תירבע Hindi: !ह#दी Croatian: Hrvatski Hungarian: Magyar Indonesian: Bahasa Indonesia Italian: Italiano Japanese: ೔ຊޠ Kazakh: Қазақша Korean: ೠҴয Lithuanian: Lietuvių Malay Bahasa: Melayu Dutch: Nederlands Norwegian: Norsk (Bokmål) Polish: Polski Portuguese: Português Romanian: Română Russian: Русский Slovak: Slovenčina Slovenian: Slovenščina Serbian: Српски / Srpski Swedish: Svenska Turkish: Türkçe Ukrainian: Українська Vietnamese: Tiếng Việt Waray-Waray: Winaray @beckerfuffle #pycon2014 20
  21. Vectorizing Text @beckerfuffle #pycon2014 21

  22. @beckerfuffle #pycon2014 22

  23. @beckerfuffle #pycon2014 23

  24. @beckerfuffle #pycon2014 24

  25. @beckerfuffle #pycon 25

  26. The Model @beckerfuffle #pycon2014 26

  27. Supervised Learning @beckerfuffle #pycon2014 27

  28. X_train @beckerfuffle #pycon2014 28

  29. Supervised Learning @beckerfuffle #pycon2014 29

  30. y_train @beckerfuffle #pycon2014 30

  31. Dimensionality Reduction @beckerfuffle #pycon2014 31

  32. Dimensionality Reduction @beckerfuffle #pycon2014 32

  33. Dimensionality Reduction @beckerfuffle #pycon2014 33

  34. Dimensionality Reduction @beckerfuffle #pycon2014 34

  35. Dimensionality Reduction @beckerfuffle #pycon2014 35

  36. Don’t Do What I Just Did! @beckerfuffle #pycon2014 36

  37. Tuning @beckerfuffle #pycon2014 37

  38. RandomizedSearchCV @beckerfuffle #pycon2014 38

  39. RandomizedSearchCV @beckerfuffle #pycon2014 39

  40. Distributing The Model @beckerfuffle #pycon2014 40

  41. Data Input @beckerfuffle #pycon2014 41

  42. The Client @beckerfuffle #pycon2014 42

  43. Message Loss @beckerfuffle #pycon2014 43

  44. Enter RabbitMQ Reliability Flexible Routing Clustering Many clients HA Queues

    @beckerfuffle #pycon2014 44
  45. Enter RabbitMQ Reliability Flexible Routing Clustering Many clients HA Queues

    @beckerfuffle #pycon2014 45
  46. Pika @beckerfuffle #pycon2014 46

  47. Pika @beckerfuffle #pycon2014 47

  48. @beckerfuffle #pycon2014 48

  49. All the things! @beckerfuffle #pycon2014 49

  50. Data Processing @beckerfuffle #pycon2014 50

  51. The Consumer @beckerfuffle #pycon2014 51

  52. The Consumer @beckerfuffle #pycon2014 52

  53. The Consumer @beckerfuffle #pycon2014 53

  54. The Consumer @beckerfuffle #pycon2014 54

  55. The Design        

             @beckerfuffle #pycon2014 55
  56. The Design        

             @beckerfuffle #pycon2014 56
  57. The Design        

             @beckerfuffle #pycon2014 57
  58. The Design        

             @beckerfuffle #pycon2014 58
  59. The Design        

             @beckerfuffle #pycon2014 59
  60. The Design        

             @beckerfuffle #pycon2014 60
  61. The Design        

             @beckerfuffle #pycon2014 61
  62. The Design        

             @beckerfuffle #pycon2014 62
  63. The Design        

             @beckerfuffle #pycon2014 63
  64. Demo Time! @beckerfuffle #pycon2014 64

  65. Scaling @beckerfuffle #pycon2014 65

  66. Realtime vs Batch @beckerfuffle #pycon2014 66

  67. Monitoring @beckerfuffle #pycon2014 67

  68. Load @beckerfuffle #pycon2014 68

  69. Re-verify @beckerfuffle #pycon2014 69

  70. Re-verify @beckerfuffle #pycon2014 70

  71. Thank You! API & Consumer: Kelly O’Brien (linkedin.com/in/kellyobie) UI: Matt

    Parke (ordinaryrobot.com) Classifier: Michael Becker (github.com/mdbecker) Images: Wikipedia; flickr.com users: kohsa, spenceyc, statefarm, klara, rh2ox, nasahqphoto, frickfrack @beckerfuffle #pycon2014 71
  72. My Info Twitter: @beckerfuffle Blog: beckerfuffle.com These slides and more

    @ github.com/mdbecker @beckerfuffle #pycon2014 72
  73. Demo time! @beckerfuffle #pycon2014 73

  74. Demo time! @beckerfuffle #pycon2014 74

  75. @beckerfuffle #pycon2014 Demo time! 75