Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Features
Speaker Deck
PRO
Sign in
Sign up for free
Search
Search
Back To The Future: Emerging Trends in Data Eng...
Search
Ananth Packkildurai
October 08, 2021
Technology
0
1.2k
Back To The Future: Emerging Trends in Data Engineering
Data Engineering Trends 2021 from
www.dataengineeringweekly.com
Ananth Packkildurai
October 08, 2021
Tweet
Share
More Decks by Ananth Packkildurai
See All by Ananth Packkildurai
Data Contracts & Domain Ownership
vananth22
0
87
Data Catalogs - Rebuild the Broken Promise
vananth22
0
79
Functional Data Engineering - A Blueprint for adopting functional principles in data pipeline
vananth22
0
480
Murron: A Reliable Monitoring Pipeline
vananth22
0
360
The_journey_towards_Pinot.pdf
vananth22
0
210
Reliable_Event_Pipeline___scale.pdf
vananth22
0
160
Operating Data Pipeline with Airflow @ Slack
vananth22
1
2.4k
Streaming data pipelines @ Slack
vananth22
2
2.2k
measuring api performance using druid
vananth22
0
1.6k
Other Decks in Technology
See All in Technology
強すぎるIAMをCloudTrailを使って適正化した話
yjszk
0
200
最新のAIツールは何を解決しようとしているのか - Python エンジニアが取り組むべき "越境" とは - / Python-Engineer-in-AI-Era
iktakahiro
1
110
サーバーレス SaaS における運用監視の負荷軽減のためのアプローチ
ririru0325
0
100
MobileActOsaka_241018.pdf
akaitadaaki
0
120
Reality is not an End-to-End Prediction Problem: Applied NLP in the Age of Generative AI
inesmontani
PRO
0
230
管理画面とユーザー機能の調和を取り戻す!~クエリパフォーマンス改善の成功物語~ / Restore harmony between administrative and user functions!
minisera
1
360
生成AI入門
shukob
0
170
新入社員 オンボーディング改善プロジェクト - シンプルな仕組みで変革のきっかけを
enpipi
0
520
0x5F3759DF
ykozw
0
370
Delta Commit…の最近...
akuwano
2
140
生成AI、LLMの いまさら聞けないキホンのキ!/ Generative AI and LLM 101
gakumura
1
170
俺とVSCode Python Debugger Extension
sat
PRO
1
150
Featured
See All Featured
Docker and Python
trallard
40
3k
Evolution of real-time – Irina Nazarova, EuRuKo, 2024
irinanazarova
4
260
Building Flexible Design Systems
yeseniaperezcruz
327
38k
GraphQLとの向き合い方2022年版
quramy
43
13k
Intergalactic Javascript Robots from Outer Space
tanoku
268
27k
Principles of Awesome APIs and How to Build Them.
keavy
126
17k
Happy Clients
brianwarren
97
6.7k
RailsConf 2023
tenderlove
28
870
Six Lessons from altMBA
skipperchong
26
3.4k
The Cost Of JavaScript in 2023
addyosmani
44
5.9k
Designing Experiences People Love
moore
138
23k
I Don’t Have Time: Getting Over the Fear to Launch Your Podcast
jcasabona
27
1.9k
Transcript
Emerging Trends in Data Engineering
Principal Data Engineer @ Zendesk www.dataengineeringweekly.com - weekly data engineering
newsletter @ananthdurai
Data Practitioners life Infinite Loop of Sadness
#1: Data Discovery & Metadata Management
Open Source Data Discovery Tools ➔ Amundsen - https://www.amundsen.io/ ➔
Marquez - https://marquezproject.github.io/ marquez/ ➔ DataHub - https://github.com/linkedin/datah ub
#2 Data Mesh & Domain Ownership
None
Catalog The Mes(h)s
#3 Data Observability
Data Observability ➔ DBT ➔ Great Expectations ➔ Deeque ➔
Airflow ➔ Dagster ➔ Prefect
#4 Data LakeHouse
Data LakeHouse ➔ Apache Iceberg ➔ Delta Lake ➔ Apache
Hudi
#5 Modern Data Stack
Modern Data Stack ➔ Extraction & Load: AirByte, FiveTran, RudderStack
etc., ➔ Data Transformation: DBT, Dataform ➔ Data Warehouse: BigQuery, Redshift, Snowflake etc., ➔ Data Governance: Acryl data, Stemma, Atlan etc., ➔ BI: Looker, Mode, Metabase etc.,
#6 Industrialized ML
Industrialized ML ➔ Tensorflow ➔ PyTorch ➔ Transformer Neural Network
& Trunk Model ➔ TPU
#7 Diversity, Privacy, AI Ethics
Diversity, Privacy & AI Ethics ➔ Explainable AI ➔ Privacy
preserve modeling ➔ AI model bias.
Emerging trends in Data Engineering 1. Data Discovery & Metadata
Management 2. Data Mesh & Domain Ownership 3. Data Observability 4. Data LakeHouse 5. Modern Data Stack 6. Industrialized ML 7. Diversity, Privacy & AI Ethics
Thank You!