Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Features
Speaker Deck
PRO
Sign in
Sign up for free
Search
Search
Evolving Sustainable Data Pipelines
Search
Hakka Labs
February 13, 2015
Programming
0
3.5k
Evolving Sustainable Data Pipelines
Full post here:
Hakka Labs
February 13, 2015
Tweet
Share
More Decks by Hakka Labs
See All by Hakka Labs
New Workflows for Building Data Pipelines
hakka_labs
0
2.9k
Collaborative Topic Models for Users and Texts
hakka_labs
0
2.8k
Groupcache with Evan Owen
hakka_labs
2
5.4k
Testing Android at Spotify
hakka_labs
1
4.5k
It's Not a Bug, It's a Feature!
hakka_labs
0
3.2k
K-means Clustering to Understand Your Users
hakka_labs
0
2k
Building Amy: The Email-based Virtual Assistant by x.ai
hakka_labs
0
5k
Deep Learning and NLP Applications
hakka_labs
3
13k
Go and the Gophers
hakka_labs
2
11k
Other Decks in Programming
See All in Programming
git worktree × Claude Code × MCP ~生成AI時代の並列開発フロー~
hisuzuya
1
590
Startups on Rails in Past, Present and Future–Irina Nazarova, RailsConf 2025
irinanazarova
0
140
生成AI時代のコンポーネントライブラリの作り方
touyou
1
250
NPOでのDevinの活用
codeforeveryone
0
860
Quand Symfony, ApiPlatform, OpenAI et LangChain s'allient pour exploiter vos PDF : de la théorie à la production…
ahmedbhs123
0
210
効率的な開発手段として VRTを活用する
ishkawa
0
150
「テストは愚直&&網羅的に書くほどよい」という誤解 / Test Smarter, Not Harder
munetoshi
0
180
Webの外へ飛び出せ NativePHPが切り拓くPHPの未来
takuyakatsusa
2
560
20250704_教育事業におけるアジャイルなデータ基盤構築
hanon52_
5
850
ニーリーにおけるプロダクトエンジニア
nealle
0
870
Python型ヒント完全ガイド 初心者でも分かる、現代的で実践的な使い方
mickey_kubo
1
140
型で語るカタ
irof
0
200
Featured
See All Featured
Being A Developer After 40
akosma
90
590k
Faster Mobile Websites
deanohume
307
31k
Raft: Consensus for Rubyists
vanstee
140
7k
The Art of Programming - Codeland 2020
erikaheidi
54
13k
JavaScript: Past, Present, and Future - NDC Porto 2020
reverentgeek
50
5.5k
Chrome DevTools: State of the Union 2024 - Debugging React & Beyond
addyosmani
7
740
Building Applications with DynamoDB
mza
95
6.5k
Statistics for Hackers
jakevdp
799
220k
For a Future-Friendly Web
brad_frost
179
9.8k
RailsConf 2023
tenderlove
30
1.1k
The Language of Interfaces
destraynor
158
25k
The World Runs on Bad Software
bkeepers
PRO
69
11k
Transcript
None
ESP Evolving Sustainable {data} Pipelines Anna Smith - @OMGannaks 29
January 2015
A bit about RTR A bit about our analytics Some
evolution stuff A bit about Luigi The future bits
A bit about RTR A bit about our analytics Some
evolution stuff A bit about Luigi The future bits
None
1. 2. 3.
WHERE THE MAGIC HAPPENS
A bit about RTR A bit about our analytics Some
evolution stuff A bit about Luigi The future bits
DEMAND website user interaction transactions SUPPLY warehouse inventory allocation
DEMAND website user interaction transactions SUPPLY warehouse inventory allocation INTERACTION
reservation calendar
None
None
None
A bit about RTR A bit about our analytics Some
evolution stuff A bit about Luigi The future bits
PHASE 0
None
YUP
KEEP IT TOGETHER
PHASE 1 primordial soup
STABILITY
PHASE 2 process
orderwarehouse.job
$ ./runjob.py orderwarehouse.job $ ./runjob.py orderwarehouse.job --show $ ./runjob.py orderwarehouse.job
--only 2
runjob.py
ADAPTING ensuring data quality
PHASE 3 exposing weaknesses
A bit about RTR A bit about our analytics Some
evolution stuff A bit about Luigi The future bits
dependency manager
None
None
A bit about RTR A bit about our analytics Some
evolution stuff A bit about Luigi The future bits
PHASE 4 ownership
RELIABILITY
COMMUNICATION
THE FUTURE @OMGannaks
[email protected]