A Random Walk in Data Science and Machine Learning in Practice - Use Cases Seminar, MS Biz Analytics, CEU - Budapest, May 2019

Transcript

1. A Random Walk in Data Science and Machine Learning in Practice Szilard Pafka, PhD Chief Scientist, Epoch (USA) CEU, Business Analytics Masters Budapest, May 2019
3. Disclaimer: I am not representing my employer (Epoch) in this talk I cannot confirm nor deny if Epoch is using any of the methods, tools, results etc. mentioned in this talk
6. CRISP-DM, 1999
47. Best Practices for Using Machine Learning in Businesses in 2018 Szilárd Pafka, PhD Chief Scientist, Epoch (USA) Budapest BI Forum Conference November 2018
49. Disclaimer: I am not representing my employer (Epoch) in this talk I cannot confirm nor deny if Epoch is using any of the methods, tools, results etc. mentioned in this talk
50. https://twitter.com/baroquepasa/
56. y = f (x1, x2, ... , xn) Source: Hastie etal, ESL 2ed
57. y = f (x1, x2, ... , xn)
60. Source: Yann LeCun
63. 2018?
64. 2018?
65. #1 Use the Right Algo
66. Source: Andrew Ng
84. *
85. #2 Use Open Source
91. in 2006 - cost was not a factor! - data.frame - [800] packages
97. #3 Simple > Complex
99. 10x
108. #4 Incorporate Domain Knowledge Do Feature Engineering (Still) Explore Your Data Clean Your Data
120. #5 Do Proper Validation Avoid: Overfitting, Data Leakage
135. #6 Batch or Real-Time Scoring?
137. https://medium.com/@HarlanH/patterns-for-connecting-predictive-models-to-software-products-f9b6e923f02d
138. https://medium.com/@dvelsner/deploying-a-simple-machine-learning-model-in-a-modern-web-application-flask-angular-docker-a657db075280 your app
141. R/Python: - Slow(er) - Encoding of categ. variables
142. #7 Do Online Validation as Well
144. https://www.oreilly.com/ideas/evaluating-machine-learning-models/page/2/orientation
145. https://www.oreilly.com/ideas/evaluating-machine-learning-models/page/2/orientation
146. https://www.oreilly.com/ideas/evaluating-machine-learning-models/page/2/orientation https://www.slideshare.net/FaisalZakariaSiddiqi/netflix-recommendations-feature-engineering-with-time-travel
147. #8 Monitor Your Models
149. https://www.retentionscience.com/blog/automating-machine-learning-monitoring-rs-labs/
150. https://www.retentionscience.com/blog/automating-machine-learning-monitoring-rs-labs/
152. 20% 80% (my guess)
153. 20% 80% (my guess)
154. #9 Business Value Seek / Measure / Sell
160. #10 Make it Reproducible
170. Cloud (servers)
171. ML training: lots of CPU cores lots of RAM limited time
172. ML training: lots of CPU cores lots of RAM limited time ML scoring: separated servers
173. ML (cloud) services (MLaaS)
175. “people that know what they’re doing just use open source [...] the same open source tools that the MLaaS services offer” - Bradford Cross
176. Kaggle
178. already pre-processed data less domain knowledge (or deliberately hidden) AUC 0.0001 increases "relevant" no business metric no actual deployment models too complex no online evaluation no monitoring data leakage
179. Tuning and Auto ML
180. Ben Recht, Kevin Jamieson: http://www.argmin.net/2016/06/20/hypertuning/
181. GPUs
182. Aggregation 100M rows 1M groups Join 100M rows x 1M rows time [s] time [s]
183. Aggregation 100M rows 1M groups Join 100M rows x 1M rows time [s] time [s] “Motherfucka!”
185. API and GUIs
188. AI?
192. How to Start?
195. Better than Deep Learning: Gradient Boosting Machines (GBM) Szilard Pafka, PhD Chief Scientist, Epoch (USA) DataWorks Summit, Barcelona, Spain March 2019
197. Disclaimer: I am not representing my employer (Epoch) in this talk I cannot confirm nor deny if Epoch is using any of the methods, tools, results etc. mentioned in this talk
198. Source: Andrew Ng
199. Source: Andrew Ng
200. Source: Andrew Ng
205. Source: https://twitter.com/iamdevloper/
208. ...
211. http://www.cs.cornell.edu/~alexn/papers/empirical.icml06.pdf http://lowrank.net/nikos/pubs/empirical.pdf
227. Source: Hastie etal, ESL 2ed
228. Source: Hastie etal, ESL 2ed
229. Source: Hastie etal, ESL 2ed
230. Source: Hastie etal, ESL 2ed
232. I usually use other people’s code [...] I can find open source code for what I want to do, and my time is much better spent doing research and feature engineering -- Owen Zhang
239. 10x
242. 10x
260. http://www.jmlr.org/papers/volume13/bergstra12a/bergstra12a.pdf
261. http://www.argmin.net/2016/06/20/hypertuning/
268. no-one is using this crap
271. More:
273. Machine Learning Software in Practice: Quo Vadis? Szilárd Pafka, PhD Chief Scientist, Epoch KDD Conference - Applied Data Science Track Invited Talk August 2017, Halifax, Canada
274. Machine Learning Software in Practice: Quo Vadis? Szilárd Pafka, PhD Chief Scientist, Epoch KDD Conference - Applied Data Science Track Invited Talk August 2017, Halifax, Canada SOME OF
277. ML Tools Mismatch: - What practitioners wish for - What they truly need
278. ML Tools Mismatch: - What practitioners wish for - What they truly need - What’s available - What’s advertised - What developers/researchers focus on
279. This talk is mostly in the context of (binary) classification
280. Warning: This talk is a series or rants observations with the aim to provoke encourage thinking and constructive discussions about topics of impact on our industry.
281. Warning: This talk is a series or rants observations with the aim to provoke encourage thinking and constructive discussions about topics of impact on our industry. Rantometer:
282. Our tools are optimized for what use cases?
283. Is building this the best allocation of our developer resources?
284. Efficiency for users during usage?
287. Big Data
306. Machine Learning Tools Speed, Memory, Accuracy
308. I usually use other people’s code [...] I can find open source code for what I want to do, and my time is much better spent doing research and feature engineering -- Owen Zhang
309. binary classification, 10M records numeric & categorical features, non-sparse
310. http://www.cs.cornell.edu/~alexn/papers/empirical.icml06.pdf http://lowrank.net/nikos/pubs/empirical.pdf
311. http://www.cs.cornell.edu/~alexn/papers/empirical.icml06.pdf http://lowrank.net/nikos/pubs/empirical.pdf
318. EC2
319. n = 10K, 100K, 1M, 10M, 100M Training time RAM usage AUC CPU % by core read data, pre-process, score test data
320. n = 10K, 100K, 1M, 10M, 100M Training time RAM usage AUC CPU % by core read data, pre-process, score test data
328. 10x
334. http://datascience.la/benchmarking-random-forest-implementations/#comment-53599
337. Best linear: 71.1
340. learn_rate = 0.1, max_depth = 6, n_trees = 300 learn_rate = 0.01, max_depth = 16, n_trees = 1000
344. Deep Learning AI Oh my... OUT
345. Distributed ML OUT
346. Multicore ML
349. 1M: CPU cache effects
350. (lightgbm 10M)
351. 16 cores vs 1: 16 cores:
352. GPUs
354. Aggregation 100M rows 1M groups Join 100M rows x 1M rows time [s] time [s]
356. Benchmarks
359. Wishlist: - more datasets (10-100, structure, size) - automation: upgrading tools, re-running ($$)
360. Wishlist: - more datasets (10-100, structure, size) - automation: upgrading tools, re-running ($$) - more algos, more tools (OS/commercial?) - (even) more tuning of parameters
361. Wishlist: - more datasets (10-100, structure, size) - automation: upgrading tools, re-running ($$) - more algos, more tools (OS/commercial?) - (even) more tuning of parameters - BaaS? crowdsourcing (data, tools/tuning)? - other ML problems (recsys, NLP…)
362. so far we discussed performance + (some) system architecture but for training only
364. APIs (and GUIs) OUT
365. Cloud (MLaaS) OUT
366. Real-Time Scoring
368. R/Python: - Slow(er) - Encoding of categ. variables
369. Kaggle OUT
370. Tuning & AutoML OUT
371. Model Understanding, Accountability
372. Evaluation Metrics OUT
373. Machine Learning with H2O.ai Szilárd Pafka, PhD Chief Scientist, Epoch LA H2O Meetup @ AT&T January 2017
374. Machine Learning with H2O.ai Szilárd Pafka, PhD Chief Scientist, Epoch LA H2O Meetup @ AT&T January 2017 SOME OF
376. Supervised Learning y = f(x) train: “learn” f from data X (n*p), y (n) score: f(x’) algos: k-NN, LR, NB, RF, GBM, SVM, NN, DL… goal: max accuracy measure (on new data) f ∈ F(θ) min θ ( L(y, f(x,θ)) + R(θ) ) on train set evaluate on separate test set /cross validation
377. Structure/Hyperparameters λ min θ ( L(y, f(x,θ[,λ])) + R(θ,λ) ) often λ ~ capacity/complexity
378. Model selection: Need Vary λ and get model with best accuracy on validation set Evaluate final model on test set /cross validation
379. overfitting
380. http://datascience.la/meetup-summary-winning-data-science-competitions/
381. http://www.cs.cornell.edu/~alexn/papers/empirical.icml06.pdf http://lowrank.net/nikos/pubs/empirical.pdf
382. http://www.cs.cornell.edu/~alexn/papers/empirical.icml06.pdf http://lowrank.net/nikos/pubs/empirical.pdf
383. http://datascience.la/meetup-summary-winning-data-science-competitions/
385. data size [M] training time [s] 10x Gradient Boosting Machines
387. Disclaimer: I’m not affiliated with H2O.ai. It’s just that in my opinion H2O is a machine learning tool with several advantages. There are many other good tools (and many more awful ones).
388. - high-performance implementation of best algos (RF, GBM, NN etc.) - R, Python etc. interfaces, easy to use API
389. - high-performance implementation of best algos (RF, GBM, NN etc.) - R, Python etc. interfaces, easy to use API - open source - advisors: Hastie, Tibshirani
390. - high-performance implementation of best algos (RF, GBM, NN etc.) - R, Python etc. interfaces, easy to use API - open source - advisors: Hastie, Tibshirani - Java, but C-style memalloc, by Java gurus - distributed, “big data”
391. - high-performance implementation of best algos (RF, GBM, NN etc.) - R, Python etc. interfaces, easy to use API - open source - advisors: Hastie, Tibshirani - Java, but C-style memalloc, by Java gurus - distributed, “big data” - many knobs/tuning, model evaluation, cross validation, model selection (hyperparameter search)
392. - high-performance implementation of best algos (RF, GBM, NN etc.) - R, Python etc. interfaces, easy to use API - open source - advisors: Hastie, Tibshirani - Java, but C-style memalloc, by Java gurus - distributed, “big data” - many knobs/tuning, model evaluation, cross validation, model selection (hyperparameter search)
393. install.packages("h2o") http://www.h2o.ai/
394. https://gist.github.com/szilard/b87233bbf41a4b366c26eede7bb1a0f3 Laptop / 1 server / cluster
398. No need for manual 1-hot encoding of categorical variables
416. https://gist.github.com/szilard/b87233bbf41a4b366c26eede7bb1a0f3
418. Some Updates
424. A Few More Thoughts