Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Features
Speaker Deck
PRO
Sign in
Sign up for free
Search
Search
What is Data Engineering?
Search
Kan Ouivirach
February 27, 2019
Technology
1
110
What is Data Engineering?
Data Engineering helps make the world/organization a better place with "Open Data".
Kan Ouivirach
February 27, 2019
Tweet
Share
More Decks by Kan Ouivirach
See All by Kan Ouivirach
Getting to Know the Data Council
zkan
1
72
Other Decks in Technology
See All in Technology
2024-10-30-reInventStandby_StudyGroup_Intro
shinichirokawano
1
610
プロダクト成長に対応するプラットフォーム戦略:Authleteによる共通認証基盤の移行事例 / Building an authentication platform using Authlete and AWS
kakehashi
1
150
フルカイテン株式会社 採用資料
fullkaiten
0
36k
来年もre:Invent2024 に行きたいあなたへ - “集中”と“つながり”で楽しむ -
ny7760
0
450
生成AIの強みと弱みを理解して、生成AIがもたらすパワーをプロダクトの価値へ繋げるために実践したこと / advance-ai-generating
cyberagentdevelopers
PRO
1
180
VPC間の接続方法を整理してみた #自治体クラウド勉強会
non97
1
800
【若手エンジニア応援LT会】AWSで繋がり、共に成長! ~コミュニティ活動と新人教育への挑戦~
kazushi_ohata
0
170
「最高のチューニング」をしないために / hack@delta 24.10
fujiwara3
21
3.4k
Figma Dev Modeで進化するデザインとエンジニアリングの協働 / figma-with-engineering
cyberagentdevelopers
PRO
1
430
LeSSに潜む「隠れWF病」とその処方箋
lycorptech_jp
PRO
2
120
日経電子版におけるリアルタイムレコメンドシステム開発の事例紹介/nikkei-realtime-recommender-system
yng87
1
490
わたしとトラックポイント / TrackPoint tips
masahirokawahara
1
240
Featured
See All Featured
Building Your Own Lightsaber
phodgson
102
6k
Designing for Performance
lara
604
68k
The Language of Interfaces
destraynor
154
24k
Adopting Sorbet at Scale
ufuk
73
9k
Practical Tips for Bootstrapping Information Extraction Pipelines
honnibal
PRO
9
680
Rails Girls Zürich Keynote
gr2m
93
13k
How to Think Like a Performance Engineer
csswizardry
19
1.1k
The Success of Rails: Ensuring Growth for the Next 100 Years
eileencodes
43
6.6k
Documentation Writing (for coders)
carmenintech
65
4.4k
No one is an island. Learnings from fostering a developers community.
thoeni
19
3k
Building Better People: How to give real-time feedback that sticks.
wjessup
363
19k
[RailsConf 2023 Opening Keynote] The Magic of Rails
eileencodes
28
9.1k
Transcript
K A N O U I V I R A
C H W H AT I S D ATA E N G I N E E R I N G ?
Kan Ouivirach Lead Software Architect
Photo by Frank Vex on Unsplash Companies use DATA to
answer these business questions: • What is our churn rate? • Are we doing better? How can we improve our services? • Any recommendation to generate more revenue from repeated customers?
J O I N T H E C L U
B . .
M A K I N G D ATA S C
I E N C E B E T T E R
M A K I N G D ATA S C
I E N C E B E T T E R
M A K I N G D ATA S C
I E N C E B E T T E R Data Engineering helps make the world/organization a better place with “Open Data”.
None
https://medium.freecodecamp.com/the-rise-of-the-data-engineer-91be18f1e603 T H E R I S E O F
T H E D ATA E N G I N E E R
Use programming languages to ensure clean, reliable, and performative access
to data and databases D ATA E N G I N E E R https://medium.freecodecamp.com/the-rise-of-the-data-engineer-91be18f1e603 The Rise of the Data Engineer
https://www.oreilly.com/ideas/data-engineers-vs-data-scientists
D ATA E N G I N E E R
V S . D ATA S C I E N T I S T https://www.datacamp.com/community/blog/data-scientist-vs-data-engineer
D ATA E N G I N E E R
V S . D ATA S C I E N T I S T Updated Feb 25, 2019
H O W M U C H D O E
S A D ATA E N G I N E E R / S C I E N T I S T M A K E ? https://www.glassdoor.com/Salaries/index.htm
H O W M U C H D O E
S A S E N I O R D ATA E N G I N E E R / S C I E N T I S T M A K E ? https://www.glassdoor.com/Salaries/index.htm
“In Data Engineering, if an expensive data scientist can’t work,
it is an emergency.” https://blog.datasyndrome.com/rules-for-crash-course-data-engineering-d19bed5f277
• Sanitize while processing data • Decouple components • Access
data at multiple stages • Promote transparency • Automate! http://prismalytics.io/datapipeline/ Data Pipeline
Data scientists doing data engineering?
http://mattturck.com/bigdata2018/
Data engineers doing data science?
https://analyticks.wordpress.com/2016/12/23/machine-learning-cheat-sheet-2/
Data Engineers Data Scientists Photo by rawpixel on Unsplash