Back To The Future: Emerging Trends in Data Engineering

Data Engineering Trends 2021 from www.dataengineeringweekly.com

Transcript

1. Emerging Trends in
   Data Engineering
   

   View Slide

2. Principal Data Engineer @ Zendesk
   www.dataengineeringweekly.com - weekly data
   engineering newsletter
   @ananthdurai
   

   View Slide

3. Data Practitioners life
   Infinite Loop of
   Sadness
   

   View Slide

4. #1: Data Discovery &
   Metadata
   Management
   

   View Slide

5. Open Source Data Discovery Tools
   ➔ Amundsen -
   https://www.amundsen.io/
   ➔ Marquez -
   https://marquezproject.github.io/
   marquez/
   ➔ DataHub -
   https://github.com/linkedin/datah
   ub
   

   View Slide

6. #2 Data Mesh &
   Domain Ownership
   

   View Slide

7. View Slide

8. Catalog The Mes(h)s
   

   View Slide

9. #3 Data Observability
   

   View Slide

10. Data Observability
    ➔ DBT
    ➔ Great Expectations
    ➔ Deeque
    ➔ Airflow
    ➔ Dagster
    ➔ Prefect
    

    View Slide

11. #4 Data LakeHouse
    

    View Slide

12. Data LakeHouse
    ➔ Apache Iceberg
    ➔ Delta Lake
    ➔ Apache Hudi
    

    View Slide

13. #5 Modern Data Stack
    

    View Slide

14. Modern Data Stack
    ➔ Extraction & Load:
    AirByte, FiveTran, RudderStack etc.,
    ➔ Data Transformation:
    DBT, Dataform
    ➔ Data Warehouse:
    BigQuery, Redshift, Snowflake etc.,
    ➔ Data Governance:
    Acryl data, Stemma, Atlan etc.,
    ➔ BI:
    Looker, Mode, Metabase etc.,
    

    View Slide

15. #6 Industrialized ML
    

    View Slide

16. Industrialized ML
    ➔ Tensorflow
    ➔ PyTorch
    ➔ Transformer Neural Network &
    Trunk Model
    ➔ TPU
    

    View Slide

17. #7 Diversity, Privacy, AI
    Ethics
    

    View Slide

18. Diversity, Privacy & AI Ethics
    ➔ Explainable AI
    ➔ Privacy preserve modeling
    ➔ AI model bias.
    

    View Slide

19. Emerging trends in Data Engineering
    1. Data Discovery & Metadata
    Management
    2. Data Mesh & Domain Ownership
    3. Data Observability
    4. Data LakeHouse
    5. Modern Data Stack
    6. Industrialized ML
    7. Diversity, Privacy & AI Ethics
    

    View Slide

20. Thank You!
    

    View Slide