Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Back To The Future: Emerging Trends in Data Engineering

Back To The Future: Emerging Trends in Data Engineering

Data Engineering Trends 2021 from www.dataengineeringweekly.com

Ananth Packkildurai

October 08, 2021
Tweet

More Decks by Ananth Packkildurai

Other Decks in Technology

Transcript

  1. Emerging Trends in
    Data Engineering

    View Slide

  2. Principal Data Engineer @ Zendesk
    www.dataengineeringweekly.com - weekly data
    engineering newsletter
    @ananthdurai

    View Slide

  3. Data Practitioners life
    Infinite Loop of
    Sadness

    View Slide

  4. #1: Data Discovery &
    Metadata
    Management

    View Slide

  5. Open Source Data Discovery Tools
    ➔ Amundsen -
    https://www.amundsen.io/
    ➔ Marquez -
    https://marquezproject.github.io/
    marquez/
    ➔ DataHub -
    https://github.com/linkedin/datah
    ub

    View Slide

  6. #2 Data Mesh &
    Domain Ownership

    View Slide

  7. View Slide

  8. Catalog The Mes(h)s

    View Slide

  9. #3 Data Observability

    View Slide

  10. Data Observability
    ➔ DBT
    ➔ Great Expectations
    ➔ Deeque
    ➔ Airflow
    ➔ Dagster
    ➔ Prefect

    View Slide

  11. #4 Data LakeHouse

    View Slide

  12. Data LakeHouse
    ➔ Apache Iceberg
    ➔ Delta Lake
    ➔ Apache Hudi

    View Slide

  13. #5 Modern Data Stack

    View Slide

  14. Modern Data Stack
    ➔ Extraction & Load:
    AirByte, FiveTran, RudderStack etc.,
    ➔ Data Transformation:
    DBT, Dataform
    ➔ Data Warehouse:
    BigQuery, Redshift, Snowflake etc.,
    ➔ Data Governance:
    Acryl data, Stemma, Atlan etc.,
    ➔ BI:
    Looker, Mode, Metabase etc.,

    View Slide

  15. #6 Industrialized ML

    View Slide

  16. Industrialized ML
    ➔ Tensorflow
    ➔ PyTorch
    ➔ Transformer Neural Network &
    Trunk Model
    ➔ TPU

    View Slide

  17. #7 Diversity, Privacy, AI
    Ethics

    View Slide

  18. Diversity, Privacy & AI Ethics
    ➔ Explainable AI
    ➔ Privacy preserve modeling
    ➔ AI model bias.

    View Slide

  19. Emerging trends in Data Engineering
    1. Data Discovery & Metadata
    Management
    2. Data Mesh & Domain Ownership
    3. Data Observability
    4. Data LakeHouse
    5. Modern Data Stack
    6. Industrialized ML
    7. Diversity, Privacy & AI Ethics

    View Slide

  20. Thank You!

    View Slide