Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Speaker Deck
PRO
Sign in
Sign up for free
Back To The Future: Emerging Trends in Data Engineering
Ananth Packkildurai
October 08, 2021
Technology
0
1k
Back To The Future: Emerging Trends in Data Engineering
Data Engineering Trends 2021 from
www.dataengineeringweekly.com
Ananth Packkildurai
October 08, 2021
Tweet
Share
More Decks by Ananth Packkildurai
See All by Ananth Packkildurai
Murron: A Reliable Monitoring Pipeline
vananth22
0
240
The_journey_towards_Pinot.pdf
vananth22
0
120
Reliable_Event_Pipeline___scale.pdf
vananth22
0
81
Operating Data Pipeline with Airflow @ Slack
vananth22
1
1.8k
Streaming data pipelines @ Slack
vananth22
2
1.7k
measuring api performance using druid
vananth22
0
1.3k
Search Infrastructure using Lambda Architecture
vananth22
1
240
Other Decks in Technology
See All in Technology
Modern Android dependency injection
hugovisser
1
120
Custom GitHub Actions by Java
kazamori
0
280
Azure Arc Virtual MachineとAzure Arc Resource Bridge / VM provisioning through Azure portal on Azure Stack HCI (preview)
sashizaki
0
120
Design for Humans: How to make better modernization decisions
indualagarsamy
2
120
ソフトウェアライセンス 2022 / Software License 2022
cybozuinsideout
PRO
1
1k
データをモデリングしていたら、組織をモデリングし始めた話 / engineers-in-carta-vol3-data-engineer
pei0804
4
3.3k
JDK Flight Recorder入門
chiroito
1
500
複数のスクラムチームをサポートするエンジニアリングマネジメントの話
okeicalm
0
1.1k
WACATE 2022 夏 ワークショップの目的
imtnd
0
120
The Fractal Geometry of Software Design
vladikk
0
570
The role of the data organization as a business progresses
line_developers
PRO
3
840
The application of formal methods in Kafka reliability engineering
line_developers
PRO
0
150
Featured
See All Featured
CoffeeScript is Beautiful & I Never Want to Write Plain JavaScript Again
sstephenson
151
13k
Web development in the modern age
philhawksworth
197
9.3k
Design and Strategy: How to Deal with People Who Don’t "Get" Design
morganepeng
104
16k
Refactoring Trust on Your Teams (GOTO; Chicago 2020)
rmw
19
1.4k
Six Lessons from altMBA
skipperchong
14
1.4k
What's new in Ruby 2.0
geeforr
336
30k
StorybookのUI Testing Handbookを読んだ
zakiyama
5
2.2k
Building Your Own Lightsaber
phodgson
94
4.6k
jQuery: Nuts, Bolts and Bling
dougneiner
56
6.4k
Keith and Marios Guide to Fast Websites
keithpitt
404
21k
How GitHub Uses GitHub to Build GitHub
holman
465
280k
Side Projects
sachag
450
37k
Transcript
Emerging Trends in Data Engineering
Principal Data Engineer @ Zendesk www.dataengineeringweekly.com - weekly data engineering
newsletter @ananthdurai
Data Practitioners life Infinite Loop of Sadness
#1: Data Discovery & Metadata Management
Open Source Data Discovery Tools ➔ Amundsen - https://www.amundsen.io/ ➔
Marquez - https://marquezproject.github.io/ marquez/ ➔ DataHub - https://github.com/linkedin/datah ub
#2 Data Mesh & Domain Ownership
None
Catalog The Mes(h)s
#3 Data Observability
Data Observability ➔ DBT ➔ Great Expectations ➔ Deeque ➔
Airflow ➔ Dagster ➔ Prefect
#4 Data LakeHouse
Data LakeHouse ➔ Apache Iceberg ➔ Delta Lake ➔ Apache
Hudi
#5 Modern Data Stack
Modern Data Stack ➔ Extraction & Load: AirByte, FiveTran, RudderStack
etc., ➔ Data Transformation: DBT, Dataform ➔ Data Warehouse: BigQuery, Redshift, Snowflake etc., ➔ Data Governance: Acryl data, Stemma, Atlan etc., ➔ BI: Looker, Mode, Metabase etc.,
#6 Industrialized ML
Industrialized ML ➔ Tensorflow ➔ PyTorch ➔ Transformer Neural Network
& Trunk Model ➔ TPU
#7 Diversity, Privacy, AI Ethics
Diversity, Privacy & AI Ethics ➔ Explainable AI ➔ Privacy
preserve modeling ➔ AI model bias.
Emerging trends in Data Engineering 1. Data Discovery & Metadata
Management 2. Data Mesh & Domain Ownership 3. Data Observability 4. Data LakeHouse 5. Modern Data Stack 6. Industrialized ML 7. Diversity, Privacy & AI Ethics
Thank You!