Realtime Predictive Analytics with scikit-learn & RabbitMQ by Michael Becker

Transcript

1. Realtime predictive
   analytics
   using scikit-learn & RabbitMQ
   Michael Becker
   @beckerfuffle
   #pycon2014
   1
   

   View Slide

2. Who Is This Guy?
   Data guy @ AWeber
   @beckerfuffle
   beckerfuffle.com
   These slides and more @ github.com/mdbecker
   @beckerfuffle
   #pycon2014
   2
   

   View Slide

3. What My Coworkers
   Think I Do
   @beckerfuffle
   #pycon2014
   3
   

   View Slide

4. What I Actually Do
   @beckerfuffle
   #pycon2014
   4
   

   View Slide

5. What I'll Cover
   • Scikit-learn overview
   @beckerfuffle
   #pycon2014
   5
   

   View Slide

6. What I'll Cover
   • Scikit-learn overview
   • Model Distribution
   @beckerfuffle
   #pycon2014
   6
   

   View Slide

7. What I'll Cover
   • Scikit-learn overview
   • Model Distribution
   • Data flow
   @beckerfuffle
   #pycon2014
   7
   

   View Slide

8. What I'll Cover
   • Scikit-learn overview
   • Model Distribution
   • Data flow
   • RabbitMQ
   @beckerfuffle
   #pycon2014
   8
   

   View Slide

9. What I'll Cover
   • Scikit-learn overview
   • Model Distribution
   • Data flow
   • RabbitMQ
   • Demo
   @beckerfuffle
   #pycon2014
   9
   

   View Slide

10. What I'll Cover
    • Scikit-learn overview
    • Model Distribution
    • Data flow
    • RabbitMQ
    • Demo
    • Scalability
    @beckerfuffle
    #pycon2014
    10
    

    View Slide

11. What I'll Cover
    • Scikit-learn overview
    • Model Distribution
    • Data flow
    • RabbitMQ
    • Demo
    • Scalability
    • Other considerations
    @beckerfuffle
    #pycon2014
    11
    

    View Slide

12. Supervised Learning
    @beckerfuffle
    #pycon2014
    12
    

    View Slide

13. Supervised Learning
    @beckerfuffle
    #pycon2014
    13
    

    View Slide

14. Supervised Learning
    @beckerfuffle
    #pycon2014
    14
    

    View Slide

15. Supervised Learning
    @beckerfuffle
    #pycon2014
    15
    

    View Slide

16. Supervised Learning
    @beckerfuffle
    #pycon2014
    16
    

    View Slide

17. Supervised Learning
    @beckerfuffle
    #pycon2014
    17
    

    View Slide

18. Supervised Learning
    @beckerfuffle
    #pycon2014
    18
    

    View Slide

19. Supervised Learning
    @beckerfuffle
    #pycon2014
    19
    

    View Slide

20. 38 Top Wikipedias
    Arabic: !"#$%&ا
    Bulgarian: Български
    Catalan: Català
    Czech: Čeština
    Danish: Dansk
    German: Deutsch
    English: English
    Spanish: Español
    Estonian: Eesti
    Basque: Euskara
    Persian: ()ر+,
    Finnish: Suomi
    French: Français
    Hebrew: תירבע
    Hindi: !ह#दी
    Croatian: Hrvatski
    Hungarian: Magyar
    Indonesian: Bahasa Indonesia
    Italian: Italiano
    Japanese: ೔ຊޠ
    Kazakh: Қазақша
    Korean: ೠҴয
    Lithuanian: Lietuvių
    Malay Bahasa: Melayu
    Dutch: Nederlands
    Norwegian: Norsk (Bokmål)
    Polish: Polski
    Portuguese: Português
    Romanian: Română
    Russian: Русский
    Slovak: Slovenčina
    Slovenian: Slovenščina
    Serbian: Српски / Srpski
    Swedish: Svenska
    Turkish: Türkçe
    Ukrainian: Українська
    Vietnamese: Tiếng Việt
    Waray-Waray: Winaray
    @beckerfuffle
    #pycon2014
    20
    

    View Slide

21. Vectorizing Text
    @beckerfuffle
    #pycon2014
    21
    

    View Slide

22. @beckerfuffle
    #pycon2014
    22
    

    View Slide

23. @beckerfuffle
    #pycon2014
    23
    

    View Slide

24. @beckerfuffle
    #pycon2014
    24
    

    View Slide

25. @beckerfuffle
    #pycon
    25
    

    View Slide

26. The Model
    @beckerfuffle
    #pycon2014
    26
    

    View Slide

27. Supervised Learning
    @beckerfuffle
    #pycon2014
    27
    

    View Slide

28. X_train
    @beckerfuffle
    #pycon2014
    28
    

    View Slide

29. Supervised Learning
    @beckerfuffle
    #pycon2014
    29
    

    View Slide

30. y_train
    @beckerfuffle
    #pycon2014
    30
    

    View Slide

31. Dimensionality
    Reduction
    @beckerfuffle
    #pycon2014
    31
    

    View Slide

32. Dimensionality
    Reduction
    @beckerfuffle
    #pycon2014
    32
    

    View Slide

33. Dimensionality
    Reduction
    @beckerfuffle
    #pycon2014
    33
    

    View Slide

34. Dimensionality
    Reduction
    @beckerfuffle
    #pycon2014
    34
    

    View Slide

35. Dimensionality
    Reduction
    @beckerfuffle
    #pycon2014
    35
    

    View Slide

36. Don’t Do What I Just Did!
    @beckerfuffle
    #pycon2014
    36
    

    View Slide

37. Tuning
    @beckerfuffle
    #pycon2014
    37
    

    View Slide

38. RandomizedSearchCV
    @beckerfuffle
    #pycon2014
    38
    

    View Slide

39. RandomizedSearchCV
    @beckerfuffle
    #pycon2014
    39
    

    View Slide

40. Distributing The Model
    @beckerfuffle
    #pycon2014
    40
    

    View Slide

41. Data Input
    @beckerfuffle
    #pycon2014
    41
    

    View Slide

42. The Client
    @beckerfuffle
    #pycon2014
    42
    

    View Slide

43. Message Loss
    @beckerfuffle
    #pycon2014
    43
    

    View Slide

44. Enter RabbitMQ
    Reliability
    Flexible Routing
    Clustering
    Many clients
    HA Queues
    @beckerfuffle
    #pycon2014
    44
    

    View Slide

45. Enter RabbitMQ
    Reliability
    Flexible Routing
    Clustering
    Many clients
    HA Queues
    @beckerfuffle
    #pycon2014
    45
    

    View Slide

46. Pika
    @beckerfuffle
    #pycon2014
    46
    

    View Slide

47. Pika
    @beckerfuffle
    #pycon2014
    47
    

    View Slide

48. @beckerfuffle
    #pycon2014
    48
    

    View Slide

49. All the things!
    @beckerfuffle
    #pycon2014
    49
    

    View Slide

50. Data Processing
    @beckerfuffle
    #pycon2014
    50
    

    View Slide

51. The Consumer
    @beckerfuffle
    #pycon2014
    51
    

    View Slide

52. The Consumer
    @beckerfuffle
    #pycon2014
    52
    

    View Slide

53. The Consumer
    @beckerfuffle
    #pycon2014
    53
    

    View Slide

54. The Consumer
    @beckerfuffle
    #pycon2014
    54
    

    View Slide

55. The Design
    
    
    
         
    
    
    
    
    
    
    
    
    @beckerfuffle
    #pycon2014
    55
    

    View Slide

56. The Design
    
    
    
         
    
    
    
    
    
    
    
    
    @beckerfuffle
    #pycon2014
    56
    

    View Slide

57. The Design
    
    
    
         
    
    
    
    
    
    
    
    
    @beckerfuffle
    #pycon2014
    57
    

    View Slide

58. The Design
    
    
    
         
    
    
    
    
    
    
    
    
    @beckerfuffle
    #pycon2014
    58
    

    View Slide

59. The Design
    
    
    
         
    
    
    
    
    
    
    
    
    @beckerfuffle
    #pycon2014
    59
    

    View Slide

60. The Design
    
    
    
         
    
    
    
    
    
    
    
    
    @beckerfuffle
    #pycon2014
    60
    

    View Slide

61. The Design
    
    
    
         
    
    
    
    
    
    
    
    
    @beckerfuffle
    #pycon2014
    61
    

    View Slide

62. The Design
    
    
    
         
    
    
    
    
    
    
    
    
    @beckerfuffle
    #pycon2014
    62
    

    View Slide

63. The Design
    
    
    
         
    
    
    
    
    
    
    
    
    @beckerfuffle
    #pycon2014
    63
    

    View Slide

64. Demo Time!
    @beckerfuffle
    #pycon2014
    64
    

    View Slide

65. Scaling
    @beckerfuffle
    #pycon2014
    65
    

    View Slide

66. Realtime vs Batch
    @beckerfuffle
    #pycon2014
    66
    

    View Slide

67. Monitoring
    @beckerfuffle
    #pycon2014
    67
    

    View Slide

68. Load
    @beckerfuffle
    #pycon2014
    68
    

    View Slide

69. Re-verify
    @beckerfuffle
    #pycon2014
    69
    

    View Slide

70. Re-verify
    @beckerfuffle
    #pycon2014
    70
    

    View Slide

71. Thank You!
    API & Consumer: Kelly O’Brien (linkedin.com/in/kellyobie)
    UI: Matt Parke (ordinaryrobot.com)
    Classifier: Michael Becker (github.com/mdbecker)
    Images: Wikipedia; flickr.com users: kohsa, spenceyc, statefarm, klara, rh2ox, nasahqphoto, frickfrack
    @beckerfuffle
    #pycon2014
    71
    

    View Slide

72. My Info
    Twitter: @beckerfuffle
    Blog: beckerfuffle.com
    These slides and more @ github.com/mdbecker
    @beckerfuffle
    #pycon2014
    72
    

    View Slide

73. Demo time!
    @beckerfuffle
    #pycon2014
    73
    

    View Slide

74. Demo time!
    @beckerfuffle
    #pycon2014
    74
    

    View Slide

75. @beckerfuffle
    #pycon2014
    Demo time!
    75
    

    View Slide