Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Back To The Future: Emerging Trends in Data Eng...

Back To The Future: Emerging Trends in Data Engineering

Data Engineering Trends 2021 from www.dataengineeringweekly.com

Ananth Packkildurai

October 08, 2021
Tweet

More Decks by Ananth Packkildurai

Other Decks in Technology

Transcript

  1. Open Source Data Discovery Tools ➔ Amundsen - https://www.amundsen.io/ ➔

    Marquez - https://marquezproject.github.io/ marquez/ ➔ DataHub - https://github.com/linkedin/datah ub
  2. Modern Data Stack ➔ Extraction & Load: AirByte, FiveTran, RudderStack

    etc., ➔ Data Transformation: DBT, Dataform ➔ Data Warehouse: BigQuery, Redshift, Snowflake etc., ➔ Data Governance: Acryl data, Stemma, Atlan etc., ➔ BI: Looker, Mode, Metabase etc.,
  3. Emerging trends in Data Engineering 1. Data Discovery & Metadata

    Management 2. Data Mesh & Domain Ownership 3. Data Observability 4. Data LakeHouse 5. Modern Data Stack 6. Industrialized ML 7. Diversity, Privacy & AI Ethics