Realtime Predictive Analytics with scikit-learn & RabbitMQ by Michael Becker

Transcript

1. Realtime predictive analytics using scikit-learn & RabbitMQ Michael Becker @beckerfuffle #pycon2014 1
2. Who Is This Guy? Data guy @ AWeber @beckerfuffle beckerfuffle.com These slides and more @ github.com/mdbecker @beckerfuffle #pycon2014 2
3. What My Coworkers Think I Do @beckerfuffle #pycon2014 3
4. What I Actually Do @beckerfuffle #pycon2014 4
5. What I'll Cover • Scikit-learn overview @beckerfuffle #pycon2014 5
6. What I'll Cover • Scikit-learn overview • Model Distribution @beckerfuffle #pycon2014 6
7. What I'll Cover • Scikit-learn overview • Model Distribution • Data flow @beckerfuffle #pycon2014 7
8. What I'll Cover • Scikit-learn overview • Model Distribution • Data flow • RabbitMQ @beckerfuffle #pycon2014 8
9. What I'll Cover • Scikit-learn overview • Model Distribution • Data flow • RabbitMQ • Demo @beckerfuffle #pycon2014 9
10. What I'll Cover • Scikit-learn overview • Model Distribution • Data flow • RabbitMQ • Demo • Scalability @beckerfuffle #pycon2014 10
11. What I'll Cover • Scikit-learn overview • Model Distribution • Data flow • RabbitMQ • Demo • Scalability • Other considerations @beckerfuffle #pycon2014 11
12. Supervised Learning @beckerfuffle #pycon2014 12
13. Supervised Learning @beckerfuffle #pycon2014 13
14. Supervised Learning @beckerfuffle #pycon2014 14
15. Supervised Learning @beckerfuffle #pycon2014 15
16. Supervised Learning @beckerfuffle #pycon2014 16
17. Supervised Learning @beckerfuffle #pycon2014 17
18. Supervised Learning @beckerfuffle #pycon2014 18
19. Supervised Learning @beckerfuffle #pycon2014 19
20. 38 Top Wikipedias Arabic: !"#$%&ا Bulgarian: Български Catalan: Català Czech: Čeština Danish: Dansk German: Deutsch English: English Spanish: Español Estonian: Eesti Basque: Euskara Persian: ()ر+, Finnish: Suomi French: Français Hebrew: תירבע Hindi: !ह#दी Croatian: Hrvatski Hungarian: Magyar Indonesian: Bahasa Indonesia Italian: Italiano Japanese: ೔ຊޠ Kazakh: Қазақша Korean: ೠҴয Lithuanian: Lietuvių Malay Bahasa: Melayu Dutch: Nederlands Norwegian: Norsk (Bokmål) Polish: Polski Portuguese: Português Romanian: Română Russian: Русский Slovak: Slovenčina Slovenian: Slovenščina Serbian: Српски / Srpski Swedish: Svenska Turkish: Türkçe Ukrainian: Українська Vietnamese: Tiếng Việt Waray-Waray: Winaray @beckerfuffle #pycon2014 20
21. Vectorizing Text @beckerfuffle #pycon2014 21
22. @beckerfuffle #pycon2014 22
23. @beckerfuffle #pycon2014 23
24. @beckerfuffle #pycon2014 24
25. @beckerfuffle #pycon 25
26. The Model @beckerfuffle #pycon2014 26
27. Supervised Learning @beckerfuffle #pycon2014 27
28. X_train @beckerfuffle #pycon2014 28
29. Supervised Learning @beckerfuffle #pycon2014 29
30. y_train @beckerfuffle #pycon2014 30
31. Dimensionality Reduction @beckerfuffle #pycon2014 31
32. Dimensionality Reduction @beckerfuffle #pycon2014 32
33. Dimensionality Reduction @beckerfuffle #pycon2014 33
34. Dimensionality Reduction @beckerfuffle #pycon2014 34
35. Dimensionality Reduction @beckerfuffle #pycon2014 35
36. Don’t Do What I Just Did! @beckerfuffle #pycon2014 36
37. Tuning @beckerfuffle #pycon2014 37
38. RandomizedSearchCV @beckerfuffle #pycon2014 38
39. RandomizedSearchCV @beckerfuffle #pycon2014 39
40. Distributing The Model @beckerfuffle #pycon2014 40
41. Data Input @beckerfuffle #pycon2014 41
42. The Client @beckerfuffle #pycon2014 42
43. Message Loss @beckerfuffle #pycon2014 43
44. Enter RabbitMQ Reliability Flexible Routing Clustering Many clients HA Queues @beckerfuffle #pycon2014 44
45. Enter RabbitMQ Reliability Flexible Routing Clustering Many clients HA Queues @beckerfuffle #pycon2014 45
46. Pika @beckerfuffle #pycon2014 46
47. Pika @beckerfuffle #pycon2014 47
48. @beckerfuffle #pycon2014 48
49. All the things! @beckerfuffle #pycon2014 49
50. Data Processing @beckerfuffle #pycon2014 50
51. The Consumer @beckerfuffle #pycon2014 51
52. The Consumer @beckerfuffle #pycon2014 52
53. The Consumer @beckerfuffle #pycon2014 53
54. The Consumer @beckerfuffle #pycon2014 54
55. The Design                  @beckerfuffle #pycon2014 55
56. The Design                  @beckerfuffle #pycon2014 56
57. The Design                  @beckerfuffle #pycon2014 57
58. The Design                  @beckerfuffle #pycon2014 58
59. The Design                  @beckerfuffle #pycon2014 59
60. The Design                  @beckerfuffle #pycon2014 60
61. The Design                  @beckerfuffle #pycon2014 61
62. The Design                  @beckerfuffle #pycon2014 62
63. The Design                  @beckerfuffle #pycon2014 63
64. Demo Time! @beckerfuffle #pycon2014 64
65. Scaling @beckerfuffle #pycon2014 65
66. Realtime vs Batch @beckerfuffle #pycon2014 66
67. Monitoring @beckerfuffle #pycon2014 67
68. Load @beckerfuffle #pycon2014 68
69. Re-verify @beckerfuffle #pycon2014 69
70. Re-verify @beckerfuffle #pycon2014 70
71. Thank You! API & Consumer: Kelly O’Brien (linkedin.com/in/kellyobie) UI: Matt Parke (ordinaryrobot.com) Classifier: Michael Becker (github.com/mdbecker) Images: Wikipedia; flickr.com users: kohsa, spenceyc, statefarm, klara, rh2ox, nasahqphoto, frickfrack @beckerfuffle #pycon2014 71
72. My Info Twitter: @beckerfuffle Blog: beckerfuffle.com These slides and more @ github.com/mdbecker @beckerfuffle #pycon2014 72
73. Demo time! @beckerfuffle #pycon2014 73
74. Demo time! @beckerfuffle #pycon2014 74
75. @beckerfuffle #pycon2014 Demo time! 75