Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Features
Speaker Deck
PRO
Sign in
Sign up for free
Search
Search
Evolving Sustainable Data Pipelines
Search
Hakka Labs
February 13, 2015
Programming
0
3.5k
Evolving Sustainable Data Pipelines
Full post here:
Hakka Labs
February 13, 2015
Tweet
Share
More Decks by Hakka Labs
See All by Hakka Labs
New Workflows for Building Data Pipelines
hakka_labs
0
2.9k
Collaborative Topic Models for Users and Texts
hakka_labs
0
2.8k
Groupcache with Evan Owen
hakka_labs
2
5.4k
Testing Android at Spotify
hakka_labs
1
4.5k
It's Not a Bug, It's a Feature!
hakka_labs
0
3.2k
K-means Clustering to Understand Your Users
hakka_labs
0
2k
Building Amy: The Email-based Virtual Assistant by x.ai
hakka_labs
0
5k
Deep Learning and NLP Applications
hakka_labs
3
13k
Go and the Gophers
hakka_labs
2
11k
Other Decks in Programming
See All in Programming
オンコール⼊⾨〜ページャーが鳴る前に、あなたが備えられること〜 / Before The Pager Rings
yktakaha4
2
1.1k
Reactの歴史を振り返る
tutinoko
1
140
可変性を制する設計: 構造と振る舞いから考える概念モデリングとその実装
a_suenami
7
1k
JetBrainsのAI機能の紹介 #jjug
yusuke
0
100
0から始めるモジュラーモノリス-クリーンなモノリスを目指して
sushi0120
0
170
リッチエディターを安全に開発・運用するために
unachang113
1
270
SwiftでMCPサーバーを作ろう!
giginet
PRO
2
210
テスターからテストエンジニアへ ~新米テストエンジニアが歩んだ9ヶ月振り返り~
non0113
2
240
20250708_JAWS_opscdk
takuyay0ne
2
150
AIのメモリー
watany
11
1k
Jakarta EE Meets AI
ivargrimstad
0
370
AI Agent 時代のソフトウェア開発を支える AWS Cloud Development Kit (CDK)
konokenj
6
1k
Featured
See All Featured
The Web Performance Landscape in 2024 [PerfNow 2024]
tammyeverts
8
720
Building Applications with DynamoDB
mza
95
6.5k
個人開発の失敗を避けるイケてる考え方 / tips for indie hackers
panda_program
110
19k
Building a Scalable Design System with Sketch
lauravandoore
462
33k
Site-Speed That Sticks
csswizardry
10
720
Navigating Team Friction
lara
187
15k
Designing for Performance
lara
610
69k
Writing Fast Ruby
sferik
628
62k
Practical Orchestrator
shlominoach
189
11k
Building Adaptive Systems
keathley
43
2.7k
The Language of Interfaces
destraynor
158
25k
Documentation Writing (for coders)
carmenintech
72
4.9k
Transcript
None
ESP Evolving Sustainable {data} Pipelines Anna Smith - @OMGannaks 29
January 2015
A bit about RTR A bit about our analytics Some
evolution stuff A bit about Luigi The future bits
A bit about RTR A bit about our analytics Some
evolution stuff A bit about Luigi The future bits
None
1. 2. 3.
WHERE THE MAGIC HAPPENS
A bit about RTR A bit about our analytics Some
evolution stuff A bit about Luigi The future bits
DEMAND website user interaction transactions SUPPLY warehouse inventory allocation
DEMAND website user interaction transactions SUPPLY warehouse inventory allocation INTERACTION
reservation calendar
None
None
None
A bit about RTR A bit about our analytics Some
evolution stuff A bit about Luigi The future bits
PHASE 0
None
YUP
KEEP IT TOGETHER
PHASE 1 primordial soup
STABILITY
PHASE 2 process
orderwarehouse.job
$ ./runjob.py orderwarehouse.job $ ./runjob.py orderwarehouse.job --show $ ./runjob.py orderwarehouse.job
--only 2
runjob.py
ADAPTING ensuring data quality
PHASE 3 exposing weaknesses
A bit about RTR A bit about our analytics Some
evolution stuff A bit about Luigi The future bits
dependency manager
None
None
A bit about RTR A bit about our analytics Some
evolution stuff A bit about Luigi The future bits
PHASE 4 ownership
RELIABILITY
COMMUNICATION
THE FUTURE @OMGannaks asmith@renttherunway.com