Data Science: Methods and Tools Szilárd Pafka, PhD Chief Scientist, Epoch Organizer, LA Data Meetups Big Data Camp LA June 2014

About

Data Science Similar to KDD process (1996), CRISP-DM (1999) or SEMMA

https://blog.cloudera.com/blog/2013/02/big-datas-new-use-cases-transformation-active-archive-and-exploration/

No content

“the data preparation process prepares both the data and the modeler”

No content

Hastie, Tibshirani and Friedman: The Elements of Statistical Learning

Hastie, Tibshirani and Friedman: The Elements of Statistical Learning

http://icml2008.cs.helsinki.fi/papers/632.pdf

http://icml2008.cs.helsinki.fi/papers/632.pdf http://www.datanami.com/2014/03/26/forget_the_algorithms_and_start_cleaning_your_data/ http://www.nltk.org/book/ch06.html

No content

No content

https://www.youtube.com/watch?v=hVimVzgtD6w (2007)

Tools I Use ...

Tools Others Use (Survey) LA Data Science/ML meetup, Apr 2014, 200 people ● Data munging: R 60%, Python 50%, SQL 40%, Hadoop (mostly Hive) 30%, Unix shell 20%, Excel 10% + Perl, Matlab, SAS, Impala, Pig, Shark...

Tools Others Use (Survey) LA Data Science/ML meetup, Apr 2014, 200 people ● Data munging: R 60%, Python 50%, SQL 40%, Hadoop (mostly Hive) 30%, Unix shell 20%, Excel 10% + Perl, Matlab, SAS, Impala, Pig, Shark... ● Visualization: R 40%,Python 30%, Tableau 10%, Javascript 10% + Matlab, Excel...

Tools Others Use (Survey) LA Data Science/ML meetup, Apr 2014, 200 people ● Data munging: R 60%, Python 50%, SQL 40%, Hadoop (mostly Hive) 30%, Unix shell 20%, Excel 10% + Perl, Matlab, SAS, Impala, Pig, Shark... ● Visualization: R 40%,Python 30%, Tableau 10%, Javascript 10% + Matlab, Excel... ● Machine learning/modeling: R 30%, Python 30% + Vowpal Wabbit, Matlab, Mahout, SAS, SPSS... http://bit.ly/datasc-tools-survey many other surveys, but...

Is Data Science New? Data Science is a newly emerging field dedicated to analyzing and manipulating data to derive insights and build data products. https://www.kaggle.com/wiki/WhatIsDataScience The United States alone faces a shortage of 140,000 to 190,000 people with deep analytical skills. [2011] http://www.mckinsey.com/insights/business_technology/big_data_the_next_frontier_for_innovation

Four major influences act on data analysis today: ● The formal theory of statistics ● Revolutionary developments in computers and display devices ● The challenge, in many fields, of more and ever larger bodies of data ● The accelerating emphasis on quantification in an ever wider variety of disciplines

Four major influences act on data analysis today: ● The formal theory of statistics ● Revolutionary developments in computers and display devices ● The challenge, in many fields, of more and ever larger bodies of data ● The accelerating emphasis on quantification in an ever wider variety of disciplines Tukey & Wilk, 1965 Tukey, J.W., & Wilk, M.B. (1965). Data analysis and statistics: techniques and approaches Reprinted in The Collected Works of John W. Tukey, Vol. V, Graphics 1965- 1985, 1-22 (1988)

(meaning the entire data mining process)

Largest Impact on My Workflow: ● 2009: ggplot2 ● 2010: useR! conference ● 2011: Rstudio ● 2012: knitr ● 2013: Impala (Hadoop interactive SQL) ● 2014: see next...

Need Tools Like This

Got Tools Like This

● Grammar for data manipulation - filter, mutate, select, arrange, summarize, group_by, various joins (SQL is not that dumb after all), %>% ● Same API for several “backends” - data.frame, data.table, MySQL, PostgreSQL... ● Fast – Rcpp/C++ Demo...

LA Data Science Community

No content

No content

[ email removed ] @DataScienceLA www.linkedin.com/in/szilard