Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Features
Speaker Deck
PRO
Sign in
Sign up for free
Search
Search
Back To The Future: Emerging Trends in Data Eng...
Search
Ananth Packkildurai
October 08, 2021
Technology
0
1.2k
Back To The Future: Emerging Trends in Data Engineering
Data Engineering Trends 2021 from
www.dataengineeringweekly.com
Ananth Packkildurai
October 08, 2021
Tweet
Share
More Decks by Ananth Packkildurai
See All by Ananth Packkildurai
Data Contracts & Domain Ownership
vananth22
0
110
Data Catalogs - Rebuild the Broken Promise
vananth22
0
84
Functional Data Engineering - A Blueprint for adopting functional principles in data pipeline
vananth22
0
540
Murron: A Reliable Monitoring Pipeline
vananth22
0
390
The_journey_towards_Pinot.pdf
vananth22
0
220
Reliable_Event_Pipeline___scale.pdf
vananth22
0
200
Operating Data Pipeline with Airflow @ Slack
vananth22
1
2.5k
Streaming data pipelines @ Slack
vananth22
2
2.4k
measuring api performance using druid
vananth22
0
1.7k
Other Decks in Technology
See All in Technology
IPA&AWSダブル全冠が明かす、人生を変えた勉強法のすべて
iwamot
PRO
2
150
KubeCon + CloudNativeCon Japan 2025 Recap by CA
ponkio_o
PRO
0
300
AWS認定を取る中で感じたこと
siromi
1
190
ネットワーク保護はどう変わるのか?re:Inforce 2025最新アップデート解説
tokushun
0
210
SaaS型なのに自由度の高い本格CMSでサイト構築と運用のコスパ&タイパUP! MovableType.net の便利機能とユーザー事例のご紹介
masakah
0
110
マネジメントって難しい、けどおもしろい / Management is tough, but fun! #em_findy
ar_tama
7
1.1k
開発生産性を測る前にやるべきこと - 組織改善の実践 / Before Measuring Dev Productivity
kaonavi
10
4.7k
OPENLOGI Company Profile
hr01
0
67k
AI専用のリンターを作る #yumemi_patch
bengo4com
5
4.3k
Geminiとv0による高速プロトタイピング
shinya337
1
270
Backlog ユーザー棚卸しRTA、多分これが一番早いと思います
__allllllllez__
1
150
KubeCon + CloudNativeCon Japan 2025 Recap Opening & Choose Your Own Adventureシリーズまとめ
mmmatsuda
0
280
Featured
See All Featured
Save Time (by Creating Custom Rails Generators)
garrettdimon
PRO
31
1.3k
Measuring & Analyzing Core Web Vitals
bluesmoon
7
510
Performance Is Good for Brains [We Love Speed 2024]
tammyeverts
10
950
The Pragmatic Product Professional
lauravandoore
35
6.7k
Six Lessons from altMBA
skipperchong
28
3.9k
A designer walks into a library…
pauljervisheath
207
24k
KATA
mclloyd
30
14k
Why Our Code Smells
bkeepers
PRO
336
57k
A better future with KSS
kneath
238
17k
Scaling GitHub
holman
460
140k
The Art of Programming - Codeland 2020
erikaheidi
54
13k
How to Ace a Technical Interview
jacobian
278
23k
Transcript
Emerging Trends in Data Engineering
Principal Data Engineer @ Zendesk www.dataengineeringweekly.com - weekly data engineering
newsletter @ananthdurai
Data Practitioners life Infinite Loop of Sadness
#1: Data Discovery & Metadata Management
Open Source Data Discovery Tools ➔ Amundsen - https://www.amundsen.io/ ➔
Marquez - https://marquezproject.github.io/ marquez/ ➔ DataHub - https://github.com/linkedin/datah ub
#2 Data Mesh & Domain Ownership
None
Catalog The Mes(h)s
#3 Data Observability
Data Observability ➔ DBT ➔ Great Expectations ➔ Deeque ➔
Airflow ➔ Dagster ➔ Prefect
#4 Data LakeHouse
Data LakeHouse ➔ Apache Iceberg ➔ Delta Lake ➔ Apache
Hudi
#5 Modern Data Stack
Modern Data Stack ➔ Extraction & Load: AirByte, FiveTran, RudderStack
etc., ➔ Data Transformation: DBT, Dataform ➔ Data Warehouse: BigQuery, Redshift, Snowflake etc., ➔ Data Governance: Acryl data, Stemma, Atlan etc., ➔ BI: Looker, Mode, Metabase etc.,
#6 Industrialized ML
Industrialized ML ➔ Tensorflow ➔ PyTorch ➔ Transformer Neural Network
& Trunk Model ➔ TPU
#7 Diversity, Privacy, AI Ethics
Diversity, Privacy & AI Ethics ➔ Explainable AI ➔ Privacy
preserve modeling ➔ AI model bias.
Emerging trends in Data Engineering 1. Data Discovery & Metadata
Management 2. Data Mesh & Domain Ownership 3. Data Observability 4. Data LakeHouse 5. Modern Data Stack 6. Industrialized ML 7. Diversity, Privacy & AI Ethics
Thank You!