Machine Learning in Production with R or Python - Budapest Data Forum - June 2017

Transcript

1. Machine Learning in Production Szilárd Pafka, PhD Chief Scientist, Epoch Budapest Data Forum June 2017
2. Machine Learning in Production with R Szilárd Pafka, PhD Chief Scientist, Epoch Budapest Data Forum June 2017
3. Machine Learning in Production with R or Python Szilárd Pafka, PhD Chief Scientist, Epoch Budapest Data Forum June 2017
4. Machine Learning in Production with R or maybe Python Szilárd Pafka, PhD Chief Scientist, Epoch Budapest Data Forum June 2017
6. Disclaimer: I am not representing my employer (Epoch) in this talk I cannot confirm nor deny if Epoch is using any of the methods, tools, results etc. mentioned in this talk
8. http://datascience.la/meetup-machine-learning-in-production-with-szilard-pafka/
22. Aggregation 100M rows 1M groups Join 100M rows x 1M rows time [s] time [s]
23. Aggregation 100M rows 1M groups Join 100M rows x 1M rows time [s] time [s]
28. binary classification, 10M records numeric & categorical features, non-sparse
29. http://www.cs.cornell.edu/~alexn/papers/empirical.icml06.pdf http://lowrank.net/nikos/pubs/empirical.pdf
30. http://www.cs.cornell.edu/~alexn/papers/empirical.icml06.pdf http://lowrank.net/nikos/pubs/empirical.pdf
36. EC2
37. n = 10K, 100K, 1M, 10M, 100M Training time RAM usage AUC CPU % by core read data, pre-process, score test data
38. n = 10K, 100K, 1M, 10M, 100M Training time RAM usage AUC CPU % by core read data, pre-process, score test data
46. 10x
52. http://datascience.la/benchmarking-random-forest-implementations/#comment-53599
60. Best linear: 71.1
63. learn_rate = 0.1, max_depth = 6, n_trees = 300 learn_rate = 0.01, max_depth = 16, n_trees = 1000
78. ...