Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Features
Speaker Deck
PRO
Sign in
Sign up for free
Search
Search
Back To The Future: Emerging Trends in Data Engineering
Search
Ananth Packkildurai
October 08, 2021
Technology
0
1.2k
Back To The Future: Emerging Trends in Data Engineering
Data Engineering Trends 2021 from
www.dataengineeringweekly.com
Ananth Packkildurai
October 08, 2021
Tweet
Share
More Decks by Ananth Packkildurai
See All by Ananth Packkildurai
Data Contracts & Domain Ownership
vananth22
0
83
Data Catalogs - Rebuild the Broken Promise
vananth22
0
73
Functional Data Engineering - A Blueprint for adopting functional principles in data pipeline
vananth22
0
460
Murron: A Reliable Monitoring Pipeline
vananth22
0
360
The_journey_towards_Pinot.pdf
vananth22
0
210
Reliable_Event_Pipeline___scale.pdf
vananth22
0
140
Operating Data Pipeline with Airflow @ Slack
vananth22
1
2.3k
Streaming data pipelines @ Slack
vananth22
2
2.2k
measuring api performance using druid
vananth22
0
1.6k
Other Decks in Technology
See All in Technology
20240717_イケコパ代表Copilot_in_Teams会社でこう使ってます
ponponmikankan
2
430
コミュニティサービスに「あなたへ」フィードを リリースするまでの試行錯誤
takapy
1
150
AWS IAMのアンチパターン/AWSが考える最低権限実現へのアプローチ概略(JAWS-UG朝会#59資料改修20分版)
htan
0
330
CTOから見た事業開発とプロダクト開発 / My Perspective on Business and Product Development as CTO
keisuke69
4
960
[NIKKEI Tech Talk] KDDI/KAG Scrum & Community for Engineering Training
curanosuke
2
220
LLMアプリケーションの評価の実践と課題 ~PharmaXにおける今後の展望~
pharma_x_tech
2
160
「単なる OAuth 2.0 を認証に使うと、車が通れるほどのどでかいセキュリティー・ホールができる」のか検証してみた
terara
0
380
What is DRE? - Road to SRE NEXT@広島
chanyou0311
3
630
フルリモートワークはエンジニアの夢を叶えたか? #cm_odyssey
mamohacy
2
600
サービスの持続的な成長と技術負債について
siva_official
PRO
10
4.4k
AI研修【MIXI 24新卒技術研修】
mixi_engineers
PRO
0
130
大規模ドラレコデータ収集・機械学習基盤を支える AWS CDK 〜導入・運用事例紹介〜
pemugi
0
110
Featured
See All Featured
Templates, Plugins, & Blocks: Oh My! Creating the theme that thinks of everything
marktimemedia
23
1.9k
Put a Button on it: Removing Barriers to Going Fast.
kastner
58
3.3k
CSS Pre-Processors: Stylus, Less & Sass
bermonpainter
353
29k
The World Runs on Bad Software
bkeepers
PRO
63
11k
Clear Off the Table
cherdarchuk
89
320k
Java REST API Framework Comparison - PWX 2021
mraible
PRO
20
7.2k
Producing Creativity
orderedlist
PRO
340
39k
Building Applications with DynamoDB
mza
89
5.8k
Building Better People: How to give real-time feedback that sticks.
wjessup
357
18k
Distributed Sagas: A Protocol for Coordinating Microservices
caitiem20
325
21k
Testing 201, or: Great Expectations
jmmastey
33
6.9k
Fashionably flexible responsive web design (full day workshop)
malarkey
399
65k
Transcript
Emerging Trends in Data Engineering
Principal Data Engineer @ Zendesk www.dataengineeringweekly.com - weekly data engineering
newsletter @ananthdurai
Data Practitioners life Infinite Loop of Sadness
#1: Data Discovery & Metadata Management
Open Source Data Discovery Tools ➔ Amundsen - https://www.amundsen.io/ ➔
Marquez - https://marquezproject.github.io/ marquez/ ➔ DataHub - https://github.com/linkedin/datah ub
#2 Data Mesh & Domain Ownership
None
Catalog The Mes(h)s
#3 Data Observability
Data Observability ➔ DBT ➔ Great Expectations ➔ Deeque ➔
Airflow ➔ Dagster ➔ Prefect
#4 Data LakeHouse
Data LakeHouse ➔ Apache Iceberg ➔ Delta Lake ➔ Apache
Hudi
#5 Modern Data Stack
Modern Data Stack ➔ Extraction & Load: AirByte, FiveTran, RudderStack
etc., ➔ Data Transformation: DBT, Dataform ➔ Data Warehouse: BigQuery, Redshift, Snowflake etc., ➔ Data Governance: Acryl data, Stemma, Atlan etc., ➔ BI: Looker, Mode, Metabase etc.,
#6 Industrialized ML
Industrialized ML ➔ Tensorflow ➔ PyTorch ➔ Transformer Neural Network
& Trunk Model ➔ TPU
#7 Diversity, Privacy, AI Ethics
Diversity, Privacy & AI Ethics ➔ Explainable AI ➔ Privacy
preserve modeling ➔ AI model bias.
Emerging trends in Data Engineering 1. Data Discovery & Metadata
Management 2. Data Mesh & Domain Ownership 3. Data Observability 4. Data LakeHouse 5. Modern Data Stack 6. Industrialized ML 7. Diversity, Privacy & AI Ethics
Thank You!