Make Machine Learning Boring Again: Best Practices for Using Machine Learning in Businesses - LA Data Science Meetup - Playa Vista, August 2019

Transcript

1. 1.

   Make Machine Learning Boring Again: Best Practices for Using Machine

   Learning in Businesses Szilard Pafka, PhD Chief Scientist, Epoch LA Data Science Meetup Aug 2019
2. 2.

   None
3. 3.

   Disclaimer: I am not representing my employer (Epoch) in this

   talk I cannot confirm nor deny if Epoch is using any of the methods, tools, results etc. mentioned in this talk
4. 4.

   None
5. 5.

   None
6. 6.

   None
7. 7.

   None
8. 8.

   None
9. 9.

   y = f (x1, x2, ... , xn) Source: Hastie

   etal, ESL 2ed
10. 10.

    y = f (x1, x2, ... , xn)

11. 11.

    None
12. 12.

    None
13. 13.

    None
14. 14.

    None
15. 15.

    #1 Use the Right Algo

16. 16.

    Source: Andrew Ng

17. 17.

    None
18. 18.

    None
19. 19.

    None
20. 20.

    None
21. 21.

    None
22. 22.

    None
23. 23.

    None
24. 24.

    None
25. 25.

    None
26. 26.

    None
27. 27.

    None
28. 28.

    None
29. 29.

    None
30. 30.

    None
31. 31.

    None
32. 32.

    None
33. 33.

    None
34. 34.

    *

35. 35.

    #2 Use Open Source

36. 36.

    None
37. 37.

    None
38. 38.

    None
39. 39.

    None
40. 40.

    None
41. 41.

    in 2006 - cost was not a factor! - data.frame

    - [800] packages
42. 42.

    None
43. 43.

    None
44. 44.

    None
45. 45.

    None
46. 46.

    None
47. 47.

    None
48. 48.

    None
49. 49.

    #3 Simple > Complex

50. 50.

    None
51. 51.

    10x

52. 52.

    None
53. 53.

    None
54. 54.

    None
55. 55.

    None
56. 56.

    None
57. 57.

    None
58. 58.

    None
59. 59.

    None
60. 60.

    #4 Incorporate Domain Knowledge Do Feature Engineering (Still) Explore Your

    Data Clean Your Data
61. 61.

    None
62. 62.

    None
63. 63.

    None
64. 64.

    None
65. 65.

    None
66. 66.

    None
67. 67.

    None
68. 68.

    None
69. 69.

    None
70. 70.

    None
71. 71.

    None
72. 72.

    #5 Do Proper Validation Avoid: Overfitting, Data Leakage

73. 73.

    None
74. 74.

    None
75. 75.

    None
76. 76.

    None
77. 77.

    None
78. 78.

    None
79. 79.

    None
80. 80.

    None
81. 81.

    None
82. 82.

    None
83. 83.

    None
84. 84.

    None
85. 85.

    None
86. 86.

    None
87. 87.

    #6 Batch or Real-Time Scoring?

88. 88.

    None
89. 89.

    https://medium.com/@HarlanH/patterns-for-connecting-predictive-models-to-software-products-f9b6e923f02d

90. 90.

    https://medium.com/@dvelsner/deploying-a-simple-machine-learning-model-in-a-modern-web-application-flask-angular-docker-a657db075280 your app

91. 91.

    None
92. 92.

    None
93. 93.

    R/Python: - Slow(er) - Encoding of categ. variables

94. 94.

    #7 Do Online Validation as Well

95. 95.

    None
96. 96.

    https://www.oreilly.com/ideas/evaluating-machine-learning-models/page/2/orientation

97. 97.

    https://www.oreilly.com/ideas/evaluating-machine-learning-models/page/2/orientation

98. 98.

    https://www.oreilly.com/ideas/evaluating-machine-learning-models/page/2/orientation https://www.slideshare.net/FaisalZakariaSiddiqi/netflix-recommendations-feature-engineering-with-time-travel

99. 99.

    #8 Monitor Your Models

100. 100.

     None
101. 101.

     https://www.retentionscience.com/blog/automating-machine-learning-monitoring-rs-labs/

102. 102.

     https://www.retentionscience.com/blog/automating-machine-learning-monitoring-rs-labs/

103. 103.

     None
104. 104.

     20% 80% (my guess)

105. 105.

     20% 80% (my guess)

106. 106.

     #9 Business Value Seek / Measure / Sell

107. 107.

     None
108. 108.

     None
109. 109.

     None
110. 110.

     None
111. 111.

     None
112. 112.

     #10 Make it Reproducible

113. 113.

     None
114. 114.

     None
115. 115.

     None
116. 116.

     None
117. 117.

     None
118. 118.

     None
119. 119.

     None
120. 120.

     None
121. 121.

     None
122. 122.

     Cloud (servers)

123. 123.

     ML training: lots of CPU cores lots of RAM limited

     time
124. 124.

     ML training: lots of CPU cores lots of RAM limited

     time ML scoring: separated servers
125. 125.

     ML (cloud) services (MLaaS)

126. 126.

     None
127. 127.

     “people that know what they’re doing just use open source

     [...] the same open source tools that the MLaaS services offer” - Bradford Cross
128. 128.

     Kaggle

129. 129.

     None
130. 130.

     already pre-processed data less domain knowledge (or deliberately hidden) AUC

     0.0001 increases "relevant" no business metric no actual deployment models too complex no online evaluation no monitoring data leakage
131. 131.

     Tuning and Auto ML

132. 132.

     Ben Recht, Kevin Jamieson: http://www.argmin.net/2016/06/20/hypertuning/

133. 133.

     GPUs

134. 134.

     Aggregation 100M rows 1M groups Join 100M rows x 1M

     rows time [s] time [s]
135. 135.

     Aggregation 100M rows 1M groups Join 100M rows x 1M

     rows time [s] time [s] “Motherfucka!”
136. 136.

     None
137. 137.

     API and GUIs

138. 138.

     None
139. 139.

     None
140. 140.

     AI?

141. 141.

     None
142. 142.

     None
143. 143.

     None
144. 144.

     How to Start?

145. 145.

     None
146. 146.

     None