Back To The Future: Emerging Trends in Data Engineering

Transcript

1. Emerging Trends in Data Engineering

2. Principal Data Engineer @ Zendesk www.dataengineeringweekly.com - weekly data engineering

   newsletter @ananthdurai

3. Data Practitioners life Infinite Loop of Sadness

4. #1: Data Discovery & Metadata Management

5. Open Source Data Discovery Tools ➔ Amundsen - https://www.amundsen.io/ ➔

   Marquez - https://marquezproject.github.io/ marquez/ ➔ DataHub - https://github.com/linkedin/datah ub

6. #2 Data Mesh & Domain Ownership

7. None

8. Catalog The Mes(h)s

9. #3 Data Observability

10. Data Observability ➔ DBT ➔ Great Expectations ➔ Deeque ➔

    Airflow ➔ Dagster ➔ Prefect

11. #4 Data LakeHouse

12. Data LakeHouse ➔ Apache Iceberg ➔ Delta Lake ➔ Apache

    Hudi

13. #5 Modern Data Stack

14. Modern Data Stack ➔ Extraction & Load: AirByte, FiveTran, RudderStack

    etc., ➔ Data Transformation: DBT, Dataform ➔ Data Warehouse: BigQuery, Redshift, Snowflake etc., ➔ Data Governance: Acryl data, Stemma, Atlan etc., ➔ BI: Looker, Mode, Metabase etc.,

15. #6 Industrialized ML

16. Industrialized ML ➔ Tensorflow ➔ PyTorch ➔ Transformer Neural Network

    & Trunk Model ➔ TPU

17. #7 Diversity, Privacy, AI Ethics

18. Diversity, Privacy & AI Ethics ➔ Explainable AI ➔ Privacy

    preserve modeling ➔ AI model bias.

19. Emerging trends in Data Engineering 1. Data Discovery & Metadata

    Management 2. Data Mesh & Domain Ownership 3. Data Observability 4. Data LakeHouse 5. Modern Data Stack 6. Industrialized ML 7. Diversity, Privacy & AI Ethics

20. Thank You!