Share
Embed
From Developer to Data Scientist (KCDC 2018)
Due to recent advances in technology, humanity is collecting vast amounts of data at an unprecedented rate, making the skills necessary to mine insights from this data increasingly valuable. So what does it take for a Developer to enter the world of data science?
Join me on a journey into the world of big data and machine learning where we will explore what the work actually looks like, identify which skills are most important, and design a road map for how you too can join this exciting and profitable industry.
Gaines Kergosien
July 12, 2018
Transcript
64 slides
-
@GainesK From Developer to Data Scientist - Gaines Kergosien @GainesK
From Developer to Data Scientist - Gaines Kergosien From Developer to Data Scientist A journey into the world of data analysis. -
@GainesK From Developer to Data Scientist - Gaines Kergosien “I
keep saying that the sexy job in the next 10 years will be statisticians. And I’m not kidding.” – Google’s Chief Economist The Demand -
@GainesK From Developer to Data Scientist - Gaines Kergosien Worldwide
spending on big data and analytics could top $210 billion by 2020 (40% growth from 2017). – IDC The Demand -
@GainesK From Developer to Data Scientist - Gaines Kergosien Worldwide
revenues for big data and business analytics (BDA) will grow from $130.1 billion in 2016 to more than $203 billion in 2020. - Worldwide Semiannual Big Data and Analytics Spending Guide The Demand -
@GainesK From Developer to Data Scientist - Gaines Kergosien “The
U.S. economy could be short as many as 250,000 data scientists by 2024.” – McKinsey Global The Demand -
@GainesK From Developer to Data Scientist - Gaines Kergosien 2017
2016 Infrastructure -
@GainesK From Developer to Data Scientist - Gaines Kergosien 2017
2016 Applications -
@GainesK From Developer to Data Scientist - Gaines Kergosien Big
Data Volume Variety • Records • Transactions • Tables & Files • Structured • Unstructured • Semi-structured -
@GainesK From Developer to Data Scientist - Gaines Kergosien Unstructured
Text • Books • Blog Posts • Comments • Tweets • Photos • Video • Audio The Variety -
@GainesK From Developer to Data Scientist - Gaines Kergosien Big
Data Volume Variety Velocity • Real Time • Near Time • Batch • Streams • Records • Transactions • Tables & Files • Structured • Unstructured • Semi-structured -
@GainesK From Developer to Data Scientist - Gaines Kergosien The
Velocity Twitter • 6,000 tweets per second • 500 million tweets/day Facebook • 300 million photos/day NY Stock Exchange • captures 1TB of trade information each session -
@GainesK From Developer to Data Scientist - Gaines Kergosien Big
Data Big Data Volume Variety Velocity • Real Time • Near Time • Batch • Streams • Records • Transactions • Tables & Files • Structured • Unstructured • Semi-structured -
@GainesK From Developer to Data Scientist - Gaines Kergosien The
Skills Subject Matter Expertise Statistics • Choose Procedures • Diagnose Problems • Develop Procedures Hacking Expertise • Technical Skills • Creativity • Values • Goals • Constraints Machine Learning Traditional Research Traditional Software Data Science -
@GainesK From Developer to Data Scientist - Gaines Kergosien The
Skills Subject Matter Expertise Hacking Expertise Social Sciences Statistics Machine Learning Traditional Software Data Science Traditional Research Traditional Research Holistic Research Socially Unaware Domain Unaware Holistic Software -
@GainesK From Developer to Data Scientist - Gaines Kergosien The
Overlap Data Science Big Data Big Data Science Big Data Volume Variety Velocity -
@GainesK From Developer to Data Scientist - Gaines Kergosien •
Data Lake • Data Mining • Unstructured Data • Dark Data • Fast Data • Edge Analytics • Predictive Analytics • Data Visualization The Buzzwords -
@GainesK From Developer to Data Scientist - Gaines Kergosien Data
• Define • Collect • Store • Explore The Phrase “Data Science” Science • Hypothesis • Plan Approach • Analysis • Report Results -
@GainesK From Developer to Data Scientist - Gaines Kergosien Data
scientists apply sophisticated quantitative and computer science skills to both structure and analyze massive stores or continuous streams of unstructured data, with the intent to derive insights and prescribe action. – Burtchworks The Data Scientist -
@GainesK From Developer to Data Scientist - Gaines Kergosien The
Data Scientist – Simple Definition -
@GainesK From Developer to Data Scientist - Gaines Kergosien •
Data Acquisition • Data Cleaning/Transformation • Analytics • Prescribing Actions • Programming/Automation The Job – Core Skills -
@GainesK From Developer to Data Scientist - Gaines Kergosien •
Educate the business • Design big data architecture • Look for problems to solve • Research new techniques • Collate data for analysis (ETL)* • Implement algorithms • Present insights The Job – Daily Activities -
@GainesK From Developer to Data Scientist - Gaines Kergosien The
Maturity – Years of Experience https://www.burtchworks.com/files/2014/07/Burtch-Works-Study_DS-2017-final.pdf -
@GainesK From Developer to Data Scientist - Gaines Kergosien The
Education https://www.burtchworks.com/files/2014/07/Burtch-Works-Study_DS-2017-final.pdf -
@GainesK From Developer to Data Scientist - Gaines Kergosien The
US Distribution https://www.burtchworks.com/files/2014/07/Burtch-Works-Study_DS-2017-final.pdf -
@GainesK From Developer to Data Scientist - Gaines Kergosien The
Areas of Study https://www.burtchworks.com/files/2014/07/Burtch-Works-Study_DS-2017-final.pdf -
@GainesK From Developer to Data Scientist - Gaines Kergosien The
Industries 2016 2017 https://www.burtchworks.com/files/2014/07/Burtch-Works-Study_DS-2017-final.pdf -
@GainesK From Developer to Data Scientist - Gaines Kergosien •
Classification – Is this A or B? • Anomaly Detection – Is this weird? • Regression – How much -or- how many? • Clustering – How is this organized? • Reinforcement Learning – What should I do next? The Five Questions -
@GainesK From Developer to Data Scientist - Gaines Kergosien The
Tool Trends Python KNIME RapidMiner R SPSS SAS Hadoop -
@GainesK From Developer to Data Scientist - Gaines Kergosien •
SQL • Excel • Python • R • MySQL The Top Tools -
@GainesK From Developer to Data Scientist - Gaines Kergosien •
SPSS • Matlab • Julia • Kafka/Storm • R • Python • Java/Scala • Stata • SAS The Languages http://www.kdnuggets.com/2015/05/r-vs-python-data-science.html -
@GainesK From Developer to Data Scientist - Gaines Kergosien The
Languages – SAS, Phython or R? http://www.burtchworks.com/2016/07/13/sas-r-python-survey-2016-tool-analytics-pros-prefer/ -
@GainesK From Developer to Data Scientist - Gaines Kergosien The
Languages – Trends http://www.burtchworks.com/2016/07/13/sas-r-python-survey-2016-tool-analytics-pros-prefer/ -
@GainesK From Developer to Data Scientist - Gaines Kergosien The
Languages – Industries http://www.burtchworks.com/2016/07/13/sas-r-python-survey-2016-tool-analytics-pros-prefer/ -
@GainesK From Developer to Data Scientist - Gaines Kergosien The
Languages – Education -
@GainesK From Developer to Data Scientist - Gaines Kergosien The
Languages – Trends -
@GainesK From Developer to Data Scientist - Gaines Kergosien The
Languages – The Future -
@GainesK From Developer to Data Scientist - Gaines Kergosien •
R Statistical Programming Language • Based on the S programming language • R Development Environment • Statistical and Visual Analysis • Cross-Platform • Free Open Source • Active User Community • Over 9,000 Extension Packages The R -
@GainesK From Developer to Data Scientist - Gaines Kergosien •
Created in 1991 to emphasize productivity and code readability • Easier learning curve than R • Free Open Source • Active User Community The Python -
@GainesK From Developer to Data Scientist - Gaines Kergosien •
Pig • Hive • Hbase • Storm • Spark • etc. The Hadoop Collective -
@GainesK From Developer to Data Scientist - Gaines Kergosien Random
• Simple Random • Stratified • Systematic/Sequential • Cluster Nonrandom • Judgement • Convenience • Snowball The Wrangling – Sampling Stratified Sampling •Sequential Sampling -
@GainesK From Developer to Data Scientist - Gaines Kergosien •
Discard • Infer The Wrangling – Data Reconciliation -
@GainesK From Developer to Data Scientist - Gaines Kergosien Normalize
Numeric Values • Standard Unit of Measure • Subtract Average (Mean = 0) • Divide by Standard Deviation Reduce Dimensionality • Irrelevant Input Variables • Redundant Input Variables The Wrangling Add Derivative Values • Generalize Attributes • Discretize Attributes to Categories • Binarize Categorical Attributes Design Training Data • Select • Combine • Aggregate Power and Log transformation • Approximate Normal Distribution -
@GainesK From Developer to Data Scientist - Gaines Kergosien •
basic statistics (ie. p-value) • statistical modeling • statistical tests • experiment design • distributions • maximum likelihood estimators • probability theory • linear algebra • multivariable calculus The Math -
@GainesK From Developer to Data Scientist - Gaines Kergosien •
Tableau (enterprise visualization products) - www.tableau.com • ggvis (R visualization package) - ggvis.rstudio.com • ggplot (plotting system) - ggplot.yhathq.com • D3.js (declarative DOM manipulation) - d3js.org • Vega (visualization grammar)- trifacta.github.com/vega • Rickshaw (charting library - code.shutterstock.com/rickshaw • modest maps (map library) - modestmaps.com • Chart.js (plotting library) - www.chartjs.org The Visualization Tools -
@GainesK From Developer to Data Scientist - Gaines Kergosien “Machine
Learning is the science of getting computers to learn and act like humans do, and improve their learning over time in autonomous fashion, by feeding them data and information in the form of observations and real-world interactions.” • Representation • Evaluation • Optimization Machine Learning -
@GainesK From Developer to Data Scientist - Gaines Kergosien Machine
Learning - Supervised -
@GainesK From Developer to Data Scientist - Gaines Kergosien Machine
Learning – Unsupervised -
@GainesK From Developer to Data Scientist - Gaines Kergosien Generative
Adversarial Networks (GAN) -
@GainesK From Developer to Data Scientist - Gaines Kergosien •
Narrow AI • General AI Artificial Intelligence -
@GainesK From Developer to Data Scientist - Gaines Kergosien •
IBM Watson • Microsoft Azure Machine Learning API • Google Prediction API • Amazon Machine Learning API • BigML The Cloud - MLaaS & AIaaS -
@GainesK From Developer to Data Scientist - Gaines Kergosien Concepts
• k-nearest neighbors • random forests • ensemble methods • …use Python libraries! Tools • Weka - www.cs.waikato.ac.nz/ml/weka/ The Machine Learning -
@GainesK From Developer to Data Scientist - Gaines Kergosien •
Report • Presentation • Demo • Prototype • Component The Results -
@GainesK From Developer to Data Scientist - Gaines Kergosien •
Data Analyst (A) • Data Engineer (B) • Academic (Ab) • Generalist (AB) The Skills -
@GainesK From Developer to Data Scientist - Gaines Kergosien •
Coursera - www.coursera.org • EdX- www.edx.org • Udacity - www.udacity.com • Kaggle - www.kaggle.com • Youtube - projects.iq.harvard.edu/stat110/youtube • Boot Camps The Training -
@GainesK From Developer to Data Scientist - Gaines Kergosien 1.
Fundamentals 2. Statistics 3. Programming 4. ML 5. Text Mining 6. Visualization 7. Big Data 8. Data Munging 9. Toolbox The Path -
@GainesK From Developer to Data Scientist - Gaines Kergosien Q
& A Slides at DotNetDude.net Subject Matter Expertise Hacking Expertise Social Sciences Statistics Machine Learning Traditional Software Data Science Traditional Research Traditional Research Holistic Research Socially Unaware Domain Unaware Holistic Software Big Data Volume Variety Velocity Data Science Big Data Big Data Science