#datacareer - Analyst? Engineer? Scientist? Roles in industry and startups with senior data engineer Ellen König

“The problem for AI in Europe is not the money, it is finding the talent” (Leading European AI practitioner)

Data and Artificial Intelligence constitute the fastest-growing job market for the highly qualified. This workshop offers you the following:

- How to find the right role for you among the emerging specialized roles in e.g. data engineering, data analytics, data science, machine learning, and deep learning.
- Pragmatic advice on handling your CV and skills profile for your next role.
- Orientation on the labour market, what employers miss most, and which #aiusecase are winning.

Transcript

1. #DATACAREER DATA ROLES IN THE INDUSTRY AND STARTUPS SPECIAL EDITION WITH SENIOR DATA ENGINEER ELLEN KÖNIG
2. #DATACAREER “No matter who you are, self-improvement is one of the most important and most overlooked attributes of young AI talent. It only takes four years of experience to become a senior AI researcher, or five years of experience to lead an entire institute. The determination and discipline to improve both the hard and soft skills continually will be the deciding factor in an AI researcher’s career.” Jean-François Gagné
4. Dânia Meira Founding member, AI Guild ML models for predictive analytics Former bootcamp teacher #datacareer since 2012 LinkedIn
5. Ellen König • Senior Data Engineer, ThoughtWorks • 3 years in Data Engineering, 3 in Data Science, 5 in Software Engineering • Worked in industry (consulting, startups) and non-profits • MSc. in Computer Science Twitter and Website
6. Chris Armbruster Founding member, AI Guild 10,000 Data Scientists for Europe Former bootcamp director #datacareer coaching since 2017 LinkedIn
7. #DATACAREER WORKSHOP OUTLINE AI Guild career coaching #dataroles specialization #dataroles skills #datacareer orientation
9. AI GUILD CAREER COACHING Running for junior and for senior practitioners since early 2019 Runs monthly for AI Guild members Coaching capacity per year: 240 participants
10. INSIGHTS FROM CAREER COACHING Search for the 1st as well as the 2nd role may take >6 months Upgrading inside a company may be easier Job advertisements may be misleading and confusing The role ‘in real life’ may not match the talents expectations
11. OBSERVING THE MARKET Specialization and differentiation of roles Rising value of domain expertise Experimental phase with PoC plays ending Increasing focus on deployment
12. #DATACAREER WORKSHOP OUTLINE #dataroles specialization #dataroles skills #datacareer orientation
13. PRODUCTIONIZING MACHINE LEARNING ML Models Data Collection Data Quality Infrastructure Process Management Tools Machine Resource Management Monitoring Configuration Feature Extraction Analysis Data Preprocessing Parameter Configuration Offline Validation Business Logic A/B Testing Data Engineer Data Scientist Data Analyst ML Engineer AI Researcher #dataroles See also: “Hidden Technical Debt in Machine Learning System” by Sculley et al, Google inc, 2015
14. #DATAROLES Task Understand business case, build features to train predictive models to address such use cases Skill Statistics, SQL, programming (e.g. python, R), ML & DL techniques. Data Scientist Task Business and data under- standing to report on what happens Skill Descriptive analytics, SQL, statistics, dashboarding and visualization tools Data Analyst Data Engineer Task Build and maintain infra- structure and pipeline to collect, clean and pre-process data Skill Distributed systems, databases, software engineering Task Optimize, deploy and maintain machine learning models in production Skill Software engineering, devops and systems architecture Machine Learning Engineer Task Build new machine learning algorithms, find custom scientific solutions Skill Research, presenting at conferences, writing publications AI Researcher
15. ‚COOKING‘ DATA: EXPLAINING SPECIALIZATION ML Models Data Collection Data Quality Infrastructure Process Management Tools Machine Resource Management Monitoring Configuration Feature Extraction Analysis Data Preprocessing Parameter Configuration Offline Validation Business Logic A/B Testing See also: Understanding a Machine Learning Workflow Through Food by Daniel Godoy Sowing Harvesting Choose recipe Prepare ingredients Customers tasting Kitchen Tasting Use utensils Try combinations of appliances and recipes Kitchen space and available appliances
16. UNDERSTANDING #DATAROLES Build Kitchen Appliances Create and use recipes to cook Check quality of ingredients and recipes Process ingredients at scale Turn a recipe into many dishes served efficiently Data Engineer Data Scientist Data Analyst ML Engineer AI Researcher
17. ELLEN’S VIEW: HOW MUCH DOES EARLY SPECIALIZATION MATTER IN DATA CAREERS?
19. WHY CONSIDER DATA ENGINEERING? You enjoy coding more than modeling or analysis You like to think about the quality of your code Complex technology is more intriguing to you than complex maths You prefer collaborating intensely with other people over strong autonomy You prefer being closer to the technology than the product and business You love buildings things that work and figuring out why things don’t work
20. #DATACAREER WORKSHOP OUTLINE #dataroles skills #datacareer orientation
21. D e e p Data Engineer Data Scientist Data Analyst ML Engineer AI Researcher Data Roles Cross-discipline ML Algorithms Visualization Domain Expertise Programming SW Engineering Communication Tools Platforms Statistics SKILLS SETS FOR DATA ROLES
22. Data Engineer Hadoop: Hive, Pig, Spark Databases Git, Docker, Airflow, Jenkins SQL, Bash, Java, Scala, Python Cross-discipline ML Algorithms Visualization Domain Expertise Programming SW Engineering Communication Tools Platforms Data pipelines Data structures Statistics Linux, AWS, Google Cloud Platform, Microsoft Azure
23. ELLEN ON GETTING STARTED IN DATA ENGINEERING Consists of four technical skill areas (in my order of importance!) 1. Software engineering foundations 2. Data modeling and analysis 3. Cloud infrastructure and operations 4. „Big“ data technologies/distributed data systems (Spark, Flink, Hadoop, Kafka etc.)
24. BUILDING SKILLS Start with small projects or projects-based classes/tutorials Check out this thread on Twitter https://twitter.com/ellen_koenig/status/ 1261571177354592256
25. CRUCIAL SKILLS AND CONCEPTS OF SOFTWARE ENGINEERING ¡ Testing ¡ Version Control ¡ Writing clean code (understandable and maintainable) ¡ Deployment ¡ Monitoring and logging ¡ Software design ¡ Software architecture
26. STANDARDS AND QUALITY Data engineers build production software! We are held to the same standards as other production software engineers
27. Cross-discipline ML Algorithms Visualization Domain Expertise Programming SW Engineering Communication Tools Platforms Data Analyst SQL R, Python Descriptive statistics Hypothesis testing Probability distributions Regression Statistics Excel, Tableau Data interpretation Logical approach Marketing Healthcare E-commerce Mobility Manufacturing ...
28. Cross-discipline ML Algorithms Visualization Domain Expertise Programming SW Engineering Communication Tools Platforms ML Engineer Statistics Hadoop: Hive, Pig, Spark Git, Docker, Airflow, Jenkins SQL, Bash, Java, Scala, Python sk-learn MLlib Linux, AWS, Google Cloud Platform, Microsoft Azure Microservices Infrastructure
29. Cross-discipline ML Algorithms Visualization Domain Expertise Programming SW Engineering Communication Tools Platforms Data Scientist Statistics Jupyter Lab, RStudio Git, Docker, Airflow, Jenkins SQL, Bash, R, Scala, Python Python: numpy, scipy, pandas, matplotlib, scikit-learn, keras, Prophet, NLTK, gensim R: dplyr, sqldf, tidyr, magrittr, lubridate, shiny, ggplot2, forecast, MLR, ranger, xgboost, Prophet MLlib Linux, AWS, Google Cloud Platform, Microsoft Azure Python: matplotlib, seaborn R: shiny, ggplot2 Marketing Healthcare E-commerce Mobility Manufacturing ...
30. #DATACAREER WORKSHOP OUTLINE #datacareer orientation
31. MARKET ORIENTATION
32. WINNING BIG AND SMALL
33. KEY INDUSTRY CHALLENGES* ¡ Data volume, accessibility, and quality ¡ Trust of customers, stakeholders, and employees, including governance, compliance, and reputation ¡ Competence of employees, management, and company *Based on the 2019 PWC report “Künstliche Intelligenz in Unternehmen”, p. 12
34. SOME STARTUP CHALLENGES • Data volume, accessibility, and quality • Company funding and runway • Expertise levels and team size
35. STARTUP MARKET MAP
36. EMPLOYER SKILLS GAPS
37. …AMONG EARLY AI ADOPTERS IN THE UNITED STATES
38. …IN CORPORATE EUROPE
39. …IN GERMAN INDUSTRY
40. …AMONG AI PLAYERS IN GERMANY
41. PROMISING #AIUSECASES
42. …AMONG EARLY ADOPTERS IN THE USA
43. ….AMONG AI PLAYERS IN GERMANY
44. ….PROSPECTIVE USE CASES IN GERMANY
45. WRAPPING UP Keep observing the market Look for matches between employers’ needs and your skills profile Scan the industry and startups for the most promising #aiusecase
46. THANK YOU Join at theguild.ai/community