Data Engineer who doesn't build only pipeline

Being a special guest for ThursTech by Prodigy9
in the topic of "Data Engineer who doesn't build only pipeline"

recording:
https://www.facebook.com/prodigy9co/videos/1302258193929483

Transcript

1. Data Engineer ที่ไม่ได้ทําแค่ pipeline Mils Burasakorn
2. Burasakorn Sabyeying (Mils, มิลส์, มิล, มิว) Data Engineer @ CJ Express (TILDI team) Website: mesodiar.com FB page: Mesodiar
3. Data Engineer Data engineers set up and operate the organization’s data infrastructure preparing it for further analysis by data analysts and scientists
4. Data Pipeline ingest transform serve
5. Problems handle more data, evolve their data pipelines What data do we have now? being the last to find out about data problems Has data arrived? Taking time to understand data journey Data Engineer Data Analyst / Data Scientist ? ? ‘garbage in’ is ‘garbage out’ problem Data Downtime - data is partial, erroneous, missing, inaccurate Is it updated? I have no trust on this table..
6. Data Monitoring Data Validation/ Data Quality Happy Data Engineer Data Observability Data Discovery / Catalog Data Lineage Traditionally, data governance is defined as the process of maintaining the availability, usability, provenance, and security of data.
7. Disclaimer: Tools != solution
8. Data Monitoring
9. Data Observability https://www.montecarlodata.com/blog-what-is-data-observability/
10. Data Observability Volume: completeness of your data tables and it offers insights on the health of the data sources - Metadata collecting - Pipeline stats - Show stats on Grafana
12. Data Discovery / Data Catalog
13. Tags: Informal controlled for search & discovery. Help co-workers to find assets more quickly. Glossary Terms: describe core business concepts and/or measurements. Label dataset with sensitivity info Domains: top-level folders aligned to business units/teams commonly used in Data Mesh to organize entities by department (i.e., Finance, Marketing, Data Platform engineering)
14. Data Quality
15. Data Quality
16. Data Lineage
17. Data Monitoring Data Validation/ Data Quality reliable Data Observability Data Discovery/ Catalog Data Lineage Confident transparency trust confident proud Data Analyst / Data Scientist Data Engineer
18. Role: Data Reliability Engineer (DRE) borrows heavily from observability and other concepts of site reliability engineering (SRE) - Set Standards: Is the data of good quality or not? - Data SLA definition and documentation - Data observability strategy and implementation Role: Analytics Engineer
19. ต้องการคนช่วยทํา (เยอะๆ) หะ อะไรนะ ทั้งหมดแค่เพียงส่วนหนึ่ง
20. mesodiar.com Fb: Mesodiar Thank you https://medium.com/cj-express-tech-tildi