Back To The Future: Emerging Trends in Data Engineering

Data Engineering Trends 2021 from www.dataengineeringweekly.com

Transcript

1. Emerging Trends in Data Engineering
2. Principal Data Engineer @ Zendesk www.dataengineeringweekly.com - weekly data engineering newsletter @ananthdurai
3. Data Practitioners life Infinite Loop of Sadness
4. #1: Data Discovery & Metadata Management
5. Open Source Data Discovery Tools ➔ Amundsen - https://www.amundsen.io/ ➔ Marquez - https://marquezproject.github.io/ marquez/ ➔ DataHub - https://github.com/linkedin/datah ub
6. #2 Data Mesh & Domain Ownership
8. Catalog The Mes(h)s
9. #3 Data Observability
10. Data Observability ➔ DBT ➔ Great Expectations ➔ Deeque ➔ Airflow ➔ Dagster ➔ Prefect
11. #4 Data LakeHouse
12. Data LakeHouse ➔ Apache Iceberg ➔ Delta Lake ➔ Apache Hudi
13. #5 Modern Data Stack
14. Modern Data Stack ➔ Extraction & Load: AirByte, FiveTran, RudderStack etc., ➔ Data Transformation: DBT, Dataform ➔ Data Warehouse: BigQuery, Redshift, Snowflake etc., ➔ Data Governance: Acryl data, Stemma, Atlan etc., ➔ BI: Looker, Mode, Metabase etc.,
15. #6 Industrialized ML
16. Industrialized ML ➔ Tensorflow ➔ PyTorch ➔ Transformer Neural Network & Trunk Model ➔ TPU
17. #7 Diversity, Privacy, AI Ethics
18. Diversity, Privacy & AI Ethics ➔ Explainable AI ➔ Privacy preserve modeling ➔ AI model bias.
19. Emerging trends in Data Engineering 1. Data Discovery & Metadata Management 2. Data Mesh & Domain Ownership 3. Data Observability 4. Data LakeHouse 5. Modern Data Stack 6. Industrialized ML 7. Diversity, Privacy & AI Ethics
20. Thank You!