Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Features
Speaker Deck
PRO
Sign in
Sign up for free
Search
Search
Back To The Future: Emerging Trends in Data Eng...
Search
Ananth Packkildurai
October 08, 2021
Technology
0
1.2k
Back To The Future: Emerging Trends in Data Engineering
Data Engineering Trends 2021 from
www.dataengineeringweekly.com
Ananth Packkildurai
October 08, 2021
Tweet
Share
More Decks by Ananth Packkildurai
See All by Ananth Packkildurai
Data Contracts & Domain Ownership
vananth22
0
110
Data Catalogs - Rebuild the Broken Promise
vananth22
0
84
Functional Data Engineering - A Blueprint for adopting functional principles in data pipeline
vananth22
0
540
Murron: A Reliable Monitoring Pipeline
vananth22
0
390
The_journey_towards_Pinot.pdf
vananth22
0
220
Reliable_Event_Pipeline___scale.pdf
vananth22
0
200
Operating Data Pipeline with Airflow @ Slack
vananth22
1
2.5k
Streaming data pipelines @ Slack
vananth22
2
2.4k
measuring api performance using druid
vananth22
0
1.7k
Other Decks in Technology
See All in Technology
Glacierだからってコストあきらめてない? / JAWS Meet Glacier Cost
taishin
1
160
Tokyo_reInforce_2025_recap_iam_access_analyzer
hiashisan
0
190
Lazy application authentication with Tailscale
bluehatbrit
0
210
AWS Organizations 新機能!マルチパーティ承認の紹介
yhana
1
280
2025-07-06 QGIS初級ハンズオン「はじめてのQGIS」
kou_kita
0
170
Core Audio tapを使ったリアルタイム音声処理のお話
yuta0306
0
190
american airlines®️ USA Contact Numbers: Complete 2025 Support Guide
supportflight
1
110
LangChain Interrupt & LangChain Ambassadors meetingレポート
os1ma
2
310
「クラウドコスト絶対削減」を支える技術—FinOpsを超えた徹底的なクラウドコスト削減の実践論
delta_tech
4
170
スタートアップに選択肢を 〜生成AIを活用したセカンダリー事業への挑戦〜
nstock
0
200
データグループにおけるフロントエンド開発
lycorptech_jp
PRO
1
100
American airlines ®️ USA Contact Numbers: Complete 2025 Support Guide
airhelpsupport
0
380
Featured
See All Featured
ピンチをチャンスに:未来をつくるプロダクトロードマップ #pmconf2020
aki_iinuma
126
53k
Easily Structure & Communicate Ideas using Wireframe
afnizarnur
194
16k
A designer walks into a library…
pauljervisheath
207
24k
Mobile First: as difficult as doing things right
swwweet
223
9.7k
Bash Introduction
62gerente
613
210k
We Have a Design System, Now What?
morganepeng
53
7.7k
How to Think Like a Performance Engineer
csswizardry
25
1.7k
Into the Great Unknown - MozCon
thekraken
40
1.9k
A Modern Web Designer's Workflow
chriscoyier
695
190k
Performance Is Good for Brains [We Love Speed 2024]
tammyeverts
10
950
Designing Dashboards & Data Visualisations in Web Apps
destraynor
231
53k
Agile that works and the tools we love
rasmusluckow
329
21k
Transcript
Emerging Trends in Data Engineering
Principal Data Engineer @ Zendesk www.dataengineeringweekly.com - weekly data engineering
newsletter @ananthdurai
Data Practitioners life Infinite Loop of Sadness
#1: Data Discovery & Metadata Management
Open Source Data Discovery Tools ➔ Amundsen - https://www.amundsen.io/ ➔
Marquez - https://marquezproject.github.io/ marquez/ ➔ DataHub - https://github.com/linkedin/datah ub
#2 Data Mesh & Domain Ownership
None
Catalog The Mes(h)s
#3 Data Observability
Data Observability ➔ DBT ➔ Great Expectations ➔ Deeque ➔
Airflow ➔ Dagster ➔ Prefect
#4 Data LakeHouse
Data LakeHouse ➔ Apache Iceberg ➔ Delta Lake ➔ Apache
Hudi
#5 Modern Data Stack
Modern Data Stack ➔ Extraction & Load: AirByte, FiveTran, RudderStack
etc., ➔ Data Transformation: DBT, Dataform ➔ Data Warehouse: BigQuery, Redshift, Snowflake etc., ➔ Data Governance: Acryl data, Stemma, Atlan etc., ➔ BI: Looker, Mode, Metabase etc.,
#6 Industrialized ML
Industrialized ML ➔ Tensorflow ➔ PyTorch ➔ Transformer Neural Network
& Trunk Model ➔ TPU
#7 Diversity, Privacy, AI Ethics
Diversity, Privacy & AI Ethics ➔ Explainable AI ➔ Privacy
preserve modeling ➔ AI model bias.
Emerging trends in Data Engineering 1. Data Discovery & Metadata
Management 2. Data Mesh & Domain Ownership 3. Data Observability 4. Data LakeHouse 5. Modern Data Stack 6. Industrialized ML 7. Diversity, Privacy & AI Ethics
Thank You!