Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Features
Speaker Deck
PRO
Sign in
Sign up for free
Search
Search
Evolving Sustainable Data Pipelines
Search
Hakka Labs
February 13, 2015
Programming
0
3.5k
Evolving Sustainable Data Pipelines
Full post here:
Hakka Labs
February 13, 2015
Tweet
Share
More Decks by Hakka Labs
See All by Hakka Labs
New Workflows for Building Data Pipelines
hakka_labs
0
2.9k
Collaborative Topic Models for Users and Texts
hakka_labs
0
2.8k
Groupcache with Evan Owen
hakka_labs
2
5.3k
Testing Android at Spotify
hakka_labs
1
4.5k
It's Not a Bug, It's a Feature!
hakka_labs
0
3.2k
K-means Clustering to Understand Your Users
hakka_labs
0
2k
Building Amy: The Email-based Virtual Assistant by x.ai
hakka_labs
0
5k
Deep Learning and NLP Applications
hakka_labs
3
13k
Go and the Gophers
hakka_labs
2
11k
Other Decks in Programming
See All in Programming
読まないコードリーディング術
hisaju
0
110
Formの複雑さに立ち向かう
bmthd
1
940
技術を改善し続ける
gumioji
0
160
Introduction to kotlinx.rpc
arawn
0
770
ML.NETで始める機械学習
ymd65536
0
240
iOSでQRコード生成奮闘記
ktcryomm
2
110
JAWS Days 2025のインフラ
komakichi
1
130
PHPカンファレンス名古屋2025 タスク分解の試行錯誤〜レビュー負荷を下げるために〜
soichi
1
730
Better Code Design in PHP
afilina
0
180
たのしいSocketのしくみ / Socket Under a Microscope
coe401_
8
1.4k
Honoをフロントエンドで使う 3つのやり方
yusukebe
7
3.6k
お前もAI鬼にならないか?👹Bolt & Cursor & Supabase & Vercelで人間をやめるぞ、ジョジョー!👺
taishiyade
7
4.2k
Featured
See All Featured
Thoughts on Productivity
jonyablonski
69
4.5k
Why You Should Never Use an ORM
jnunemaker
PRO
55
9.2k
How to Ace a Technical Interview
jacobian
276
23k
Building Adaptive Systems
keathley
40
2.4k
Refactoring Trust on Your Teams (GOTO; Chicago 2020)
rmw
33
2.8k
Exploring the Power of Turbo Streams & Action Cable | RailsConf2023
kevinliebholz
30
4.6k
The Straight Up "How To Draw Better" Workshop
denniskardys
232
140k
For a Future-Friendly Web
brad_frost
176
9.6k
Responsive Adventures: Dirty Tricks From The Dark Corners of Front-End
smashingmag
251
21k
Building Better People: How to give real-time feedback that sticks.
wjessup
367
19k
RailsConf & Balkan Ruby 2019: The Past, Present, and Future of Rails at GitHub
eileencodes
134
33k
What’s in a name? Adding method to the madness
productmarketing
PRO
22
3.3k
Transcript
None
ESP Evolving Sustainable {data} Pipelines Anna Smith - @OMGannaks 29
January 2015
A bit about RTR A bit about our analytics Some
evolution stuff A bit about Luigi The future bits
A bit about RTR A bit about our analytics Some
evolution stuff A bit about Luigi The future bits
None
1. 2. 3.
WHERE THE MAGIC HAPPENS
A bit about RTR A bit about our analytics Some
evolution stuff A bit about Luigi The future bits
DEMAND website user interaction transactions SUPPLY warehouse inventory allocation
DEMAND website user interaction transactions SUPPLY warehouse inventory allocation INTERACTION
reservation calendar
None
None
None
A bit about RTR A bit about our analytics Some
evolution stuff A bit about Luigi The future bits
PHASE 0
None
YUP
KEEP IT TOGETHER
PHASE 1 primordial soup
STABILITY
PHASE 2 process
orderwarehouse.job
$ ./runjob.py orderwarehouse.job $ ./runjob.py orderwarehouse.job --show $ ./runjob.py orderwarehouse.job
--only 2
runjob.py
ADAPTING ensuring data quality
PHASE 3 exposing weaknesses
A bit about RTR A bit about our analytics Some
evolution stuff A bit about Luigi The future bits
dependency manager
None
None
A bit about RTR A bit about our analytics Some
evolution stuff A bit about Luigi The future bits
PHASE 4 ownership
RELIABILITY
COMMUNICATION
THE FUTURE @OMGannaks
[email protected]