Data Engineering Trends 2021 from www.dataengineeringweekly.com
Emerging Trends inData Engineering
View Slide
Principal Data Engineer @ Zendeskwww.dataengineeringweekly.com - weekly dataengineering newsletter@ananthdurai
Data Practitioners lifeInfinite Loop ofSadness
#1: Data Discovery &MetadataManagement
Open Source Data Discovery Tools➔ Amundsen -https://www.amundsen.io/➔ Marquez -https://marquezproject.github.io/marquez/➔ DataHub -https://github.com/linkedin/datahub
#2 Data Mesh &Domain Ownership
Catalog The Mes(h)s
#3 Data Observability
Data Observability➔ DBT➔ Great Expectations➔ Deeque➔ Airflow➔ Dagster➔ Prefect
#4 Data LakeHouse
Data LakeHouse➔ Apache Iceberg➔ Delta Lake➔ Apache Hudi
#5 Modern Data Stack
Modern Data Stack➔ Extraction & Load:AirByte, FiveTran, RudderStack etc.,➔ Data Transformation:DBT, Dataform➔ Data Warehouse:BigQuery, Redshift, Snowflake etc.,➔ Data Governance:Acryl data, Stemma, Atlan etc.,➔ BI:Looker, Mode, Metabase etc.,
#6 Industrialized ML
Industrialized ML➔ Tensorflow➔ PyTorch➔ Transformer Neural Network &Trunk Model➔ TPU
#7 Diversity, Privacy, AIEthics
Diversity, Privacy & AI Ethics➔ Explainable AI➔ Privacy preserve modeling➔ AI model bias.
Emerging trends in Data Engineering1. Data Discovery & MetadataManagement2. Data Mesh & Domain Ownership3. Data Observability4. Data LakeHouse5. Modern Data Stack6. Industrialized ML7. Diversity, Privacy & AI Ethics
Thank You!