Size of Datasets for Analytics and Implications for R - useR! conference, Stanford University - June 2016

Transcript

1. Size of Datasets for Analytics and Implications for R Szilárd Pafka, PhD Chief Scientist, Epoch useR! conference, Stanford University June 2016
3. Disclaimer: I am not representing my employer (Epoch) in this talk I cannot confirm nor deny if Epoch is using any of the methods, tools, results etc. mentioned in this talk
16. (largest data analyzed)
17. (largest data analyzed)
21. https://deads.gitbooks.io/paratext-bench/content/teaser.html June 2016
22. Aggregation 100M rows 1M groups Join 100M rows x 1M rows time [s] time [s]
23. Aggregation 100M rows 1M groups Join 100M rows x 1M rows time [s] time [s] Speedup 5 nodes: Hive 1.5x Spark 2x
26. (largest data analyzed)
27. (largest data analyzed)
31. data size [M] training time [s] 10x Gradient Boosting Machines
34. linear tops off (data size) (accuracy)
35. linear tops off more data & better algo (data size) (accuracy)
36. linear tops off more data & better algo random forest on 1% of data beats linear on all data (data size) (accuracy)
37. linear tops off more data & better algo random forest on 1% of data beats linear on all data (data size) (accuracy)
45. Summary / Tips for analyzing “big” data: - Get lots of RAM (physical/ cloud) - Use R and high performance R packages (e.g. data. table, xgboost) - Do data reduction in database (analytical db/ big data system) - (Only) distribute embarrassingly parallel tasks (e.g. hyperparameter search for machine learning) - Let engineers (store and) ETL the data (“scalable”) - Use statistics/ domain knowledge/ thinking - Use “big data tools” only if the above tips not enough