Machine Learning Software in Practice: Quo Vadis? - Invited Talk, KDD Conference, Applied Data Science Track - August 2017, Halifax, Canada

Transcript

1. Machine Learning Software in Practice: Quo Vadis? Szilárd Pafka, PhD Chief Scientist, Epoch KDD Conference - Applied Data Science Track Invited Talk August 2017, Halifax, Canada
3. Disclaimer: I am not representing my employer (Epoch) in this talk I cannot confirm nor deny if Epoch is using any of the methods, tools, results etc. mentioned in this talk
6. ML Tools Mismatch: - What practitioners wish for - What they truly need
7. ML Tools Mismatch: - What practitioners wish for - What they truly need - What’s available - What’s advertised - What developers/researchers focus on
8. This talk is mostly in the context of (binary) classification
9. Warning: This talk is a series or rants observations with the aim to provoke encourage thinking and constructive discussions about topics of impact on our industry.
10. Warning: This talk is a series or rants observations with the aim to provoke encourage thinking and constructive discussions about topics of impact on our industry. Rantometer:
11. Our tools are optimized for what use cases?
12. Is building this the best allocation of our developer resources?
13. Efficiency for users during usage?
14. 10x
15. 100x
16. Big Data
35. Machine Learning Tools Speed, Memory, Accuracy
37. I usually use other people’s code [...] I can find open source code for what I want to do, and my time is much better spent doing research and feature engineering -- Owen Zhang http://blog.kaggle.com/2015/06/22/profiling-top-kagglers-owen-zhang-currently-1-in-the-world/
38. binary classification, 10M records numeric & categorical features, non-sparse
39. http://www.cs.cornell.edu/~alexn/papers/empirical.icml06.pdf http://lowrank.net/nikos/pubs/empirical.pdf
40. http://www.cs.cornell.edu/~alexn/papers/empirical.icml06.pdf http://lowrank.net/nikos/pubs/empirical.pdf
43. - R packages - Python scikit-learn - Vowpal Wabbit - H2O - xgboost - Spark MLlib - a few others
44. - R packages 30% - Python scikit-learn 40% - Vowpal Wabbit 8% - H2O 10% - xgboost 8% - Spark MLlib 6% - a few others
45. - R packages 30% - Python scikit-learn 40% - Vowpal Wabbit 8% - H2O 10% - xgboost 8% - Spark MLlib 6% - a few others
47. EC2
48. n = 10K, 100K, 1M, 10M, 100M Training time RAM usage AUC CPU % by core read data, pre-process, score test data
49. n = 10K, 100K, 1M, 10M, 100M Training time RAM usage AUC CPU % by core read data, pre-process, score test data
57. 10x
63. http://datascience.la/benchmarking-random-forest-implementations/#comment-53599
66. Best linear: 71.1
69. learn_rate = 0.1, max_depth = 6, n_trees = 300 learn_rate = 0.01, max_depth = 16, n_trees = 1000
73. Deep Learning AI Oh my...
74. Source: Andrew Ng
77. ...
80. Distributed ML
90. Multicore ML
93. 1M: CPU cache effects
94. (lightgbm 10M)
95. 16 cores vs 1: 16 cores:
96. GPUs
98. Aggregation 100M rows 1M groups Join 100M rows x 1M rows time [s] time [s]
100. Benchmarks
103. Wishlist: - more datasets (10-100, structure, size) - automation: upgrading tools, re-running ($$)
104. Wishlist: - more datasets (10-100, structure, size) - automation: upgrading tools, re-running ($$) - more algos, more tools (OS/commercial?) - (even) more tuning of parameters
105. Wishlist: - more datasets (10-100, structure, size) - automation: upgrading tools, re-running ($$) - more algos, more tools (OS/commercial?) - (even) more tuning of parameters - BaaS? crowdsourcing (data, tools/tuning)? - other ML problems (recsys, NLP…)
106. so far we discussed performance + (some) system architecture but for training only
108. APIs (and GUIs)
111. Cloud (MLaaS)
115. “people that know what they’re doing just use open source [...] the same open source tools that the MLaaS services offer” - Bradford Cross
116. Real-Time Scoring
117. R/Python: - Slow(er) - Encoding of categ. variables
118. Kaggle
119. already pre-processed data less domain knowledge (or deliberately hidden) AUC 0.0001 increases "relevant" no business metric no actual deployment models too complex no online evaluation no monitoring data leakage
120. Tuning & AutoML
121. Ben Recht, Kevin Jamieson: http://www.argmin.net/2016/06/20/hypertuning/
122. Model Understanding, Accountability
123. Evaluation Metrics
125. More... “Will We Ever Get Over the Mess?” KDD panel next