Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Back To The Future: Emerging Trends in Data Engineering

Back To The Future: Emerging Trends in Data Engineering

Data Engineering Trends 2021 from www.dataengineeringweekly.com

2c4b23630d3e6ee69efb4db16186d266?s=128

Ananth Packkildurai

October 08, 2021
Tweet

Transcript

  1. Emerging Trends in Data Engineering

  2. Principal Data Engineer @ Zendesk www.dataengineeringweekly.com - weekly data engineering

    newsletter @ananthdurai
  3. Data Practitioners life Infinite Loop of Sadness

  4. #1: Data Discovery & Metadata Management

  5. Open Source Data Discovery Tools ➔ Amundsen - https://www.amundsen.io/ ➔

    Marquez - https://marquezproject.github.io/ marquez/ ➔ DataHub - https://github.com/linkedin/datah ub
  6. #2 Data Mesh & Domain Ownership

  7. None
  8. Catalog The Mes(h)s

  9. #3 Data Observability

  10. Data Observability ➔ DBT ➔ Great Expectations ➔ Deeque ➔

    Airflow ➔ Dagster ➔ Prefect
  11. #4 Data LakeHouse

  12. Data LakeHouse ➔ Apache Iceberg ➔ Delta Lake ➔ Apache

    Hudi
  13. #5 Modern Data Stack

  14. Modern Data Stack ➔ Extraction & Load: AirByte, FiveTran, RudderStack

    etc., ➔ Data Transformation: DBT, Dataform ➔ Data Warehouse: BigQuery, Redshift, Snowflake etc., ➔ Data Governance: Acryl data, Stemma, Atlan etc., ➔ BI: Looker, Mode, Metabase etc.,
  15. #6 Industrialized ML

  16. Industrialized ML ➔ Tensorflow ➔ PyTorch ➔ Transformer Neural Network

    & Trunk Model ➔ TPU
  17. #7 Diversity, Privacy, AI Ethics

  18. Diversity, Privacy & AI Ethics ➔ Explainable AI ➔ Privacy

    preserve modeling ➔ AI model bias.
  19. Emerging trends in Data Engineering 1. Data Discovery & Metadata

    Management 2. Data Mesh & Domain Ownership 3. Data Observability 4. Data LakeHouse 5. Modern Data Stack 6. Industrialized ML 7. Diversity, Privacy & AI Ethics
  20. Thank You!