Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Features
Speaker Deck
PRO
Sign in
Sign up for free
Search
Search
Back To The Future: Emerging Trends in Data Eng...
Search
Ananth Packkildurai
October 08, 2021
Technology
0
1.2k
Back To The Future: Emerging Trends in Data Engineering
Data Engineering Trends 2021 from
www.dataengineeringweekly.com
Ananth Packkildurai
October 08, 2021
Tweet
Share
More Decks by Ananth Packkildurai
See All by Ananth Packkildurai
Data Contracts & Domain Ownership
vananth22
0
110
Data Catalogs - Rebuild the Broken Promise
vananth22
0
84
Functional Data Engineering - A Blueprint for adopting functional principles in data pipeline
vananth22
0
550
Murron: A Reliable Monitoring Pipeline
vananth22
0
400
The_journey_towards_Pinot.pdf
vananth22
0
230
Reliable_Event_Pipeline___scale.pdf
vananth22
0
210
Operating Data Pipeline with Airflow @ Slack
vananth22
1
2.5k
Streaming data pipelines @ Slack
vananth22
2
2.4k
measuring api performance using druid
vananth22
0
1.7k
Other Decks in Technology
See All in Technology
生成AI利用プログラミング:誰でもプログラムが書けると 世の中どうなる?/opencampus202508
okana2ki
0
190
Figma + Storybook + PlaywrightのMCPを使ったフロントエンド開発
yug1224
9
2.7k
LLM時代の検索とコンテキストエンジニアリング
shibuiwilliam
2
1.1k
TypeScript入門
recruitengineers
PRO
20
6.7k
Preferred Networks (PFN) とLLM Post-Training チームの紹介 / 第4回 関東Kaggler会 スポンサーセッション
pfn
PRO
1
230
AIエージェントの開発に必須な「コンテキスト・エンジニアリング」とは何か──プロンプト・エンジニアリングとの違いを手がかりに考える
masayamoriofficial
0
400
マイクロモビリティシェアサービスを支える プラットフォームアーキテクチャ
grimoh
1
240
コスト削減の基本の「キ」~ コスト消費3大リソースへの対策 ~
smt7174
2
160
ECS モニタリング手法大整理
yendoooo
1
120
つくって納得、つかって実感! 大規模言語モデルことはじめ
recruitengineers
PRO
24
6.2k
Claude Code x Androidアプリ 開発
kgmyshin
1
590
アジャイルテストで高品質のスプリントレビューを
takesection
0
120
Featured
See All Featured
Documentation Writing (for coders)
carmenintech
73
5k
The Art of Delivering Value - GDevCon NA Keynote
reverentgeek
15
1.6k
A better future with KSS
kneath
239
17k
The Invisible Side of Design
smashingmag
301
51k
Six Lessons from altMBA
skipperchong
28
4k
A Tale of Four Properties
chriscoyier
160
23k
Building Applications with DynamoDB
mza
96
6.6k
Navigating Team Friction
lara
189
15k
No one is an island. Learnings from fostering a developers community.
thoeni
21
3.4k
Making Projects Easy
brettharned
117
6.3k
Producing Creativity
orderedlist
PRO
347
40k
Put a Button on it: Removing Barriers to Going Fast.
kastner
60
4k
Transcript
Emerging Trends in Data Engineering
Principal Data Engineer @ Zendesk www.dataengineeringweekly.com - weekly data engineering
newsletter @ananthdurai
Data Practitioners life Infinite Loop of Sadness
#1: Data Discovery & Metadata Management
Open Source Data Discovery Tools ➔ Amundsen - https://www.amundsen.io/ ➔
Marquez - https://marquezproject.github.io/ marquez/ ➔ DataHub - https://github.com/linkedin/datah ub
#2 Data Mesh & Domain Ownership
None
Catalog The Mes(h)s
#3 Data Observability
Data Observability ➔ DBT ➔ Great Expectations ➔ Deeque ➔
Airflow ➔ Dagster ➔ Prefect
#4 Data LakeHouse
Data LakeHouse ➔ Apache Iceberg ➔ Delta Lake ➔ Apache
Hudi
#5 Modern Data Stack
Modern Data Stack ➔ Extraction & Load: AirByte, FiveTran, RudderStack
etc., ➔ Data Transformation: DBT, Dataform ➔ Data Warehouse: BigQuery, Redshift, Snowflake etc., ➔ Data Governance: Acryl data, Stemma, Atlan etc., ➔ BI: Looker, Mode, Metabase etc.,
#6 Industrialized ML
Industrialized ML ➔ Tensorflow ➔ PyTorch ➔ Transformer Neural Network
& Trunk Model ➔ TPU
#7 Diversity, Privacy, AI Ethics
Diversity, Privacy & AI Ethics ➔ Explainable AI ➔ Privacy
preserve modeling ➔ AI model bias.
Emerging trends in Data Engineering 1. Data Discovery & Metadata
Management 2. Data Mesh & Domain Ownership 3. Data Observability 4. Data LakeHouse 5. Modern Data Stack 6. Industrialized ML 7. Diversity, Privacy & AI Ethics
Thank You!