Data Engineering behind Data Science World

Presented in Data Science and Data Engineering class for computer engineering student at Chulalongkorn University.

13 February 2024

Transcript

1. Data Engineering Behind Data Science World กมลพรรณ ลิ้วประเสริฐ (ฝน) PhD Student @ Chula Google Developer Expert - Cloud Women Techmakers Ambassador CP39
2. Warm up 🔥
3. Andrej Karpathy, “Building the Software 2.0 Stack,” Spark+AI Summit 2018, video, 17:54, https://oreil.ly/Z21Oz. Chip Huyen, “Designing Machine Learning Systems”
4. Data มาจากไหน ?
6. Why we need a data engineer?
7. Data Science Hierarchy of Needs Data Engineers Data Analysts Data Scientists
8. Application Analytics
9. Application Workload ✅ For application and backend services ✅ Transactional data ✅ Read / Write / Update (ACID) ✅ E.g. Database ✅ For query, dashboards and analytics ✅ Historical data ✅ Write once, read many ✅ E.g. Data Warehouse, Data Lake, Data Mart Analytics Workload OLTP OLAP
10. - เมื่อไร “Data Engineer” มี Technology ใหม ๆ เกิดขึ้นมากมาย แตอาจจะยังไมไดรับความ นิยมในตอนนี้ Brief History of Data Engineer ‘06 ‘10 ยุค Big Data (20xx) 1983 ยุค Database ‘15 1979 1989 ‘14 ‘13 ‘12 ‘11 เริ่มมี Job Data Engineer MapReduce paper ‘04 ‘18 ‘19 10
11. การสรางและดูแล Data Infrastructure • Database • Data Lake • Data Warehouse • Cloud / on-premise Infrastructure • Data Pipeline Azure Blob Storage Amazon S3 Google Cloud Storage Amazon Redshift Google BigQuery Azure Synapse Snowflake Apache Hive Database Data Lake Data Warehouse Cloud / on-premise 11
12. DE ในแต่ละบริษัทเป็นอย่างไร
13. https://towardsdatascience.com/how-to-measure-your-organizations-data-maturity-2352cbaf1896
14. ยกตัวอย่าง Tech stack Data Engineering
15. Techstack ของบริษัทดัง หาได้จาก blog / newsletter
16. ตัวอย่าง Techstack ของบริษัทในไทย
17. ตัวอย่าง Techstack ของบริษัท (2)
18. แหลงเรียน Data Engineer ฟรี [FREE WIKI] Data Engineering Wiki https://dataengineering.wiki/ [FREE PDF] Data Engineering Cookbook https://cookbook.learndataengineering.com/ [ZOOM COURSE] Data Engineering Zoomcamp https://github.com/DataTalksClub/data-engineerin g-zoomcamp [NEWSLETTER] Start Data Engineering https://www.startdataengineering.com/ Youtube.com (ฟรี) รวม Talk จาก Meetup และ Conference ตาง ๆ Channel ที่แนะนำ: - DataTalksClub - Data Council - PyData - Andreas Kretz - Seattle Data Guy - The Cloud Girl - Google Cloud Tech 18
19. Next Action for Data Engineer https://www.startdataengineering.com/ Learn Do 1. Go through the Quickstart 2. Understand its fundamentals 3. Know their objectives Bash Python Java / Scala (optional) SQL Programming Languages Frameworks Platforms
20. DE Trend • Data Mesh / Data Fabric ( datamesh-architecture.com ) • Data Observability • Data Reliability Engineering • Self-service Analytic System - Analytics Engineering • Reverse ETL
21. MARCH 2023 HOW IT’S GOING (2023)
22. Bonus: ML Engineering
23. นิยาม AI System แบบง่าย https://youtu.be/VfcY0edoSLU AI System = * Software Engineering DevOps Data Engineering Data Science ML/AI Research ML Engineering Software + Model + DATA
24. High-level of ML Systems 1. Data ▸ Data Engineering Pipelines ▸ Data Quality is a key 🔑 2. Model ▸ Machine Learning Pipelines ▸ Train, Evaluate, Test 󰙤 3. Software ▸ Model Serving & Predictions ▸ Deployment strategies & Infra 🐳☁
25. Machine Learning Engineers are technically proficient programmers who research, build, and design self-running software to automate predictive models. An ML Engineer builds artificial intelligence (AI) systems that leverage huge data sets to generate and develop algorithms capable of learning and eventually making predictions. - Brainstation.io ML Engineer Machine Learning Engineer คือ นักพัฒนาโปรแกรมที่วิจัย สราง และออกแบบ ซอฟตแวรที่ สามารถรันโมเดลทํานายผลได และสรางระบบ AI โดยใชประโยชนจากขอมูลจํานวนมาก
26. ML Engineer MLE / DS / DE แตกต่างกันอย่างไร Data Engineer Data Scientist DATA MODEL MODEL In PROD E.g. Data pipeline Stats, Accuracy, Business value Performance, Throughput, Automation
27. ML Engineer MLE / DS / DE แตกต่างกันอย่างไร (2) Data Engineer Data Scientist Automate ʻem all!
28. มารู้จัก ML System ML System Users Business Requirements ML System Developers ML System Deployment, monitoring, updating of logics Feature engineering ML algorithms Chip Huyen, “Designing Machine Learning Systems” Evaluation Data Infrastructure Data Scientist Feature engineering ML algorithms Evaluation
29. Rule of Machine Learning Martin Zinkevich https://developers.google.com/machine-learning/guides/rules-of-ml
30. To make great products: do machine learning like the great engineer you are, not like the great machine learning expert you aren’t. Martin Zinkevich https://developers.google.com/machine-learning/guides/rules-of-ml - Rule of Machine Learning -
31. Thank you! กมลพรรณ ลิ้วประเสริฐ (ฝน) PhD Student @ Chula Google Developer Expert - Cloud Women Techmakers Ambassador CP39
32. Ads
33. Image slides facebook.com/wtmbkk developers.google.com/womentechmakers IG: @wtmbkk
34. Upcoming: International Women’s Day 2024 9 March 2024 16 March 2024 @wtmbkk