Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Features
Speaker Deck
PRO
Sign in
Sign up for free
Search
Search
Back To The Future: Emerging Trends in Data Eng...
Search
Ananth Packkildurai
October 08, 2021
Technology
0
1.2k
Back To The Future: Emerging Trends in Data Engineering
Data Engineering Trends 2021 from
www.dataengineeringweekly.com
Ananth Packkildurai
October 08, 2021
Tweet
Share
More Decks by Ananth Packkildurai
See All by Ananth Packkildurai
Data Contracts & Domain Ownership
vananth22
0
110
Data Catalogs - Rebuild the Broken Promise
vananth22
0
84
Functional Data Engineering - A Blueprint for adopting functional principles in data pipeline
vananth22
0
540
Murron: A Reliable Monitoring Pipeline
vananth22
0
390
The_journey_towards_Pinot.pdf
vananth22
0
220
Reliable_Event_Pipeline___scale.pdf
vananth22
0
200
Operating Data Pipeline with Airflow @ Slack
vananth22
1
2.5k
Streaming data pipelines @ Slack
vananth22
2
2.4k
measuring api performance using druid
vananth22
0
1.7k
Other Decks in Technology
See All in Technology
スタックチャン家庭用アシスタントへの道
kanekoh
0
120
AIでテストプロセス自動化に挑戦する
sakatakazunori
1
530
【あのMCPって、どんな処理してるの?】 AWS CDKでの開発で便利なAWS MCP Servers特集
yoshimi0227
6
950
Rethinking Incident Response: Context-Aware AI in Practice
rrreeeyyy
2
940
cdk initで生成されるあのファイル達は何なのか/cdk-init-generated-files
tomoki10
1
670
振り返りTransit Gateway ~VPCをいい感じでつなげるために~
masakiokuda
3
210
組織内、組織間の資産保護に必要なアイデンティティ基盤と関連技術の最新動向
fujie
0
270
対話型音声AIアプリケーションの信頼性向上の取り組み
ivry_presentationmaterials
3
1.1k
TLSから見るSREの未来
atpons
2
310
20250708オープンエンドな探索と知識発見
sakana_ai
PRO
4
1k
IPA&AWSダブル全冠が明かす、人生を変えた勉強法のすべて
iwamot
PRO
2
230
AIエージェントが書くのなら直接CloudFormationを書かせればいいじゃないですか何故AWS CDKを使う必要があるのさ
watany
18
7.6k
Featured
See All Featured
Sharpening the Axe: The Primacy of Toolmaking
bcantrill
44
2.4k
Optimizing for Happiness
mojombo
379
70k
The Myth of the Modular Monolith - Day 2 Keynote - Rails World 2024
eileencodes
26
2.9k
Building Adaptive Systems
keathley
43
2.7k
ReactJS: Keep Simple. Everything can be a component!
pedronauck
667
120k
Unsuck your backbone
ammeep
671
58k
Code Reviewing Like a Champion
maltzj
524
40k
Producing Creativity
orderedlist
PRO
346
40k
Improving Core Web Vitals using Speculation Rules API
sergeychernyshev
18
990
The Art of Delivering Value - GDevCon NA Keynote
reverentgeek
15
1.6k
[RailsConf 2023] Rails as a piece of cake
palkan
55
5.7k
Raft: Consensus for Rubyists
vanstee
140
7k
Transcript
Emerging Trends in Data Engineering
Principal Data Engineer @ Zendesk www.dataengineeringweekly.com - weekly data engineering
newsletter @ananthdurai
Data Practitioners life Infinite Loop of Sadness
#1: Data Discovery & Metadata Management
Open Source Data Discovery Tools ➔ Amundsen - https://www.amundsen.io/ ➔
Marquez - https://marquezproject.github.io/ marquez/ ➔ DataHub - https://github.com/linkedin/datah ub
#2 Data Mesh & Domain Ownership
None
Catalog The Mes(h)s
#3 Data Observability
Data Observability ➔ DBT ➔ Great Expectations ➔ Deeque ➔
Airflow ➔ Dagster ➔ Prefect
#4 Data LakeHouse
Data LakeHouse ➔ Apache Iceberg ➔ Delta Lake ➔ Apache
Hudi
#5 Modern Data Stack
Modern Data Stack ➔ Extraction & Load: AirByte, FiveTran, RudderStack
etc., ➔ Data Transformation: DBT, Dataform ➔ Data Warehouse: BigQuery, Redshift, Snowflake etc., ➔ Data Governance: Acryl data, Stemma, Atlan etc., ➔ BI: Looker, Mode, Metabase etc.,
#6 Industrialized ML
Industrialized ML ➔ Tensorflow ➔ PyTorch ➔ Transformer Neural Network
& Trunk Model ➔ TPU
#7 Diversity, Privacy, AI Ethics
Diversity, Privacy & AI Ethics ➔ Explainable AI ➔ Privacy
preserve modeling ➔ AI model bias.
Emerging trends in Data Engineering 1. Data Discovery & Metadata
Management 2. Data Mesh & Domain Ownership 3. Data Observability 4. Data LakeHouse 5. Modern Data Stack 6. Industrialized ML 7. Diversity, Privacy & AI Ethics
Thank You!