Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Features
Speaker Deck
PRO
Sign in
Sign up for free
Search
Search
Back To The Future: Emerging Trends in Data Eng...
Search
Ananth Packkildurai
October 08, 2021
Technology
0
1.2k
Back To The Future: Emerging Trends in Data Engineering
Data Engineering Trends 2021 from
www.dataengineeringweekly.com
Ananth Packkildurai
October 08, 2021
Tweet
Share
More Decks by Ananth Packkildurai
See All by Ananth Packkildurai
Data Contracts & Domain Ownership
vananth22
0
110
Data Catalogs - Rebuild the Broken Promise
vananth22
0
84
Functional Data Engineering - A Blueprint for adopting functional principles in data pipeline
vananth22
0
550
Murron: A Reliable Monitoring Pipeline
vananth22
0
390
The_journey_towards_Pinot.pdf
vananth22
0
220
Reliable_Event_Pipeline___scale.pdf
vananth22
0
200
Operating Data Pipeline with Airflow @ Slack
vananth22
1
2.5k
Streaming data pipelines @ Slack
vananth22
2
2.4k
measuring api performance using druid
vananth22
0
1.7k
Other Decks in Technology
See All in Technology
VLMサービスを用いた請求書データ化検証 / SaaSxML_Session_1
sansan_randd
0
210
Unson OS|48時間で「売れるか」を判定する AI 市場検証プラットフォーム
unson
0
170
robocopy の怖い話/scary-story-about-robocopy
emiki
0
460
データエンジニアがクラシルでやりたいことの現在地
gappy50
3
850
Lambda management with ecspresso and Terraform
ijin
2
120
隙間時間で爆速開発! Claude Code × Vibe Coding で作るマニュアル自動生成サービス
akitomonam
3
250
生成AI導入の効果を最大化する データ活用戦略
ham0215
0
110
【2025 Japan AWS Jr. Champions Ignition】点から線、線から面へ〜僕たちが起こすコラボレーション・ムーブメント〜
amixedcolor
1
120
Mambaで物体検出 完全に理解した
shirarei24
2
210
「育てる」サーバーレス 〜チーム開発研修で学んだ、小さく始めて大きく拡張するAWS設計〜
yu_kod
1
250
Oracle Cloud Infrastructure:2025年7月度サービス・アップデート
oracle4engineer
PRO
1
110
風が吹けばWHOISが使えなくなる~なぜWHOIS・RDAPはサーバー証明書のメール認証に使えなくなったのか~
orangemorishita
15
5.4k
Featured
See All Featured
How to Think Like a Performance Engineer
csswizardry
25
1.8k
Build The Right Thing And Hit Your Dates
maggiecrowley
37
2.8k
What's in a price? How to price your products and services
michaelherold
246
12k
How to Ace a Technical Interview
jacobian
278
23k
Testing 201, or: Great Expectations
jmmastey
45
7.6k
Put a Button on it: Removing Barriers to Going Fast.
kastner
60
3.9k
Documentation Writing (for coders)
carmenintech
73
5k
Done Done
chrislema
185
16k
The MySQL Ecosystem @ GitHub 2015
samlambert
251
13k
Sharpening the Axe: The Primacy of Toolmaking
bcantrill
44
2.4k
Docker and Python
trallard
45
3.5k
The Myth of the Modular Monolith - Day 2 Keynote - Rails World 2024
eileencodes
26
3k
Transcript
Emerging Trends in Data Engineering
Principal Data Engineer @ Zendesk www.dataengineeringweekly.com - weekly data engineering
newsletter @ananthdurai
Data Practitioners life Infinite Loop of Sadness
#1: Data Discovery & Metadata Management
Open Source Data Discovery Tools ➔ Amundsen - https://www.amundsen.io/ ➔
Marquez - https://marquezproject.github.io/ marquez/ ➔ DataHub - https://github.com/linkedin/datah ub
#2 Data Mesh & Domain Ownership
None
Catalog The Mes(h)s
#3 Data Observability
Data Observability ➔ DBT ➔ Great Expectations ➔ Deeque ➔
Airflow ➔ Dagster ➔ Prefect
#4 Data LakeHouse
Data LakeHouse ➔ Apache Iceberg ➔ Delta Lake ➔ Apache
Hudi
#5 Modern Data Stack
Modern Data Stack ➔ Extraction & Load: AirByte, FiveTran, RudderStack
etc., ➔ Data Transformation: DBT, Dataform ➔ Data Warehouse: BigQuery, Redshift, Snowflake etc., ➔ Data Governance: Acryl data, Stemma, Atlan etc., ➔ BI: Looker, Mode, Metabase etc.,
#6 Industrialized ML
Industrialized ML ➔ Tensorflow ➔ PyTorch ➔ Transformer Neural Network
& Trunk Model ➔ TPU
#7 Diversity, Privacy, AI Ethics
Diversity, Privacy & AI Ethics ➔ Explainable AI ➔ Privacy
preserve modeling ➔ AI model bias.
Emerging trends in Data Engineering 1. Data Discovery & Metadata
Management 2. Data Mesh & Domain Ownership 3. Data Observability 4. Data LakeHouse 5. Modern Data Stack 6. Industrialized ML 7. Diversity, Privacy & AI Ethics
Thank You!