How to develop a data scientist – What business has requested v02

Presentation by Brendan Moran, Data Scientist @Greenplum EMC at Data Science London 21/03/12

Transcript

1. 1 © Copyright 2010 EMC Corporation. All rights reserved. Data Computing Division How to develop a data scientist What business has requested Big Data Meet Up 21 March 2012 Brendan Moran, EMC Data Scientist
2. 2 © Copyright 2010 EMC Corporation. All rights reserved. Data Computing Division Context •  McKinsey report –  Technology and techniques –  Mind the gap •  140-160k deep analytic talent •  1.5m data savvy managers –  UK Top 6 in producing talent •  EMC Global Survey •  Kaggle.com •  Our clients
3. 3 © Copyright 2010 EMC Corporation. All rights reserved. Data Computing Division Does it matter? Where’s the next generation coming from? Big data: The next frontier for innovation, competition, and productivity, http://tinyurl.com/74tdfdv
4. 4 © Copyright 2010 EMC Corporation. All rights reserved. Data Computing Division What about the UK? Big data: The next frontier for innovation, competition, and productivity, http://tinyurl.com/74tdfdv
5. 5 © Copyright 2010 EMC Corporation. All rights reserved. Data Computing Division What’s been trending – courtesy of Datasift http://tinyurl.com/6vek5ge •  # views –  1747072 What is a data scientist? | Datablog http://t.co/tFfVvstm –  1537330 I love Oxford-style debates. This one at #strataconf: the data science debate: domain expertise or machine learning? http://t.co/jKGhx8AY –  1536264 #strataconf is amazing. Data science is the new black. •  Most popular links: –  2812012-03-02 14:01-What is a data scientist? | Datablog | News | guardian.co.uk –  2082012-03-08 11:25-bitly's Hilary Mason on \"What is A Data Scientist?\" - Forbes –  1752012-03-02 15:24-A Data Scientist You’ve Never Heard of Is Now the Master of Your Domain
6. 6 © Copyright 2010 EMC Corporation. All rights reserved. Data Computing Division Who do we have in tonight? Show of hands…. •  Are you a data scientist? –  Beginner –  Proficient –  Expert?
7. 7 © Copyright 2010 EMC Corporation. All rights reserved. Data Computing Division Snapshot of you Tools in your toolbox •  Warehousing/Analytics (single response) –  SQL -> IBM -> Oracle •  How do you manipulate your data (multi-response) –  Excel -> SQL -> Python •  How do you analyse your data (multi-response) –  SAS -> STATA -> SPSS (R was last) •  How do you visualise your data (multi-response) –  MS BI tools -> Oracel -> IBM -> SAP -> Microstrategy
8. 8 © Copyright 2010 EMC Corporation. All rights reserved. Data Computing Division Snapshot of you Your traits •  You want a full set of data (53% ) •  Only 13% were comfortable working with complete data •  “I explore the data|report what it says” – evenly distributed •  “My findings drive decisions, or report what has happened” – evenly distributed
9. 9 © Copyright 2010 EMC Corporation. All rights reserved. Data Computing Division Why EMC? •  Award winner for enterprise development (TSIA 2011) •  Commitment to open source initiatives (chorus) •  Relationships with 700 universities around world – many already take our course content •  Carnegie Mellon •  Berkeley
10. 10 © Copyright 2010 EMC Corporation. All rights reserved. Data Computing Division How did we form the content? •  Experts •  Kaggle •  Universities •  Our enterprise clients
11. 11 © Copyright 2010 EMC Corporation. All rights reserved. Data Computing Division Guiding principles •  Open source – R, Rstudio, SQL, Python •  Vendor neutral •  No licensing implications (important for universities) –  Community editions of MPP DB, Hadoop •  Applied learning – lots of labs (~40% time) •  Foundation course
12. 12 © Copyright 2010 EMC Corporation. All rights reserved. Data Computing Division Where’s the bar? •  Solid understanding of statistics •  Experience with a scripting language (Jave, Perl, Python, R) •  Experience with SQL (or PSQL)
13. 13 © Copyright 2010 EMC Corporation. All rights reserved. Data Computing Division What’s on the course?
14. 14 © Copyright 2010 EMC Corporation. All rights reserved. Data Computing Division Is this for you? Practice, practice, practice •  “Tell me and I forget. Show me and I remember. Involve me and I understand” •  40% is hands on lab •  Take “dirty” data, tidy it up, start exploring data, basic statistics, simple plots, complex stats, beautiful graphs, build models, test models, present your findings
15. 15 © Copyright 2010 EMC Corporation. All rights reserved. Data Computing Division Show me : hypothesis testing Null & Alternative hypotheses •  is there a difference?
16. 16 © Copyright 2010 EMC Corporation. All rights reserved. Data Computing Division Show me : hypothesis testing How good is my model? •  Receiver Operating Characteristics (ROC) –  False positives –  True positives
17. 17 © Copyright 2010 EMC Corporation. All rights reserved. Data Computing Division Show me: visualising your data Study hours by education #Code #http://tinyurl.com/7rbx7qs library(arules) data ("AdultUCI") dframe = AdultUCI[, c ("education", "hours-per- week")] colnames(dframe) = c ("education", "hours_per_week") library(ggplot2) ggplot (dframe, aes(x=education, y=hours_per_week)) + geom_point(colour="lightblue", alpha=0.1, position="jitter") + geom_boxplot(outlier.size=0, alpha=0.2) + coord_flip()
18. 18 © Copyright 2010 EMC Corporation. All rights reserved. Data Computing Division Show me : what problem do I have How do I solve it Problem to solve Technique (e.g.) Need to group items by similarity Need to discover relationships between actions or items Want to determine relationship between outcome and input variables Want to assign (known) labels to items Want to analyse my text
19. 19 © Copyright 2010 EMC Corporation. All rights reserved. Data Computing Division Show me : what problem do I have How do I solve it Problem to solve Technique (e.g.) Need to group items by similarity Clustering (k-means) Need to discover relationships between actions or items Association rules (a priori) Want to determine relationship between outcome and input variables Regression (linear/logistic) Want to assign (known) labels to items Classification (Naïve Bayes, decision trees) Want to analyse my text Regular expressions, Bag of words
20. 20 © Copyright 2010 EMC Corporation. All rights reserved. Data Computing Division So what now? What about you? •  If you know all of this already – congratulations! •  If you’d like to know more – our course goes live 26 March (register at http://education.emc.com) •  If you couldn’t care less – you’re probably in the wrong room