Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Evolving Sustainable Data Pipelines

Avatar for Hakka Labs Hakka Labs
February 13, 2015
Programming
0
3.5k

Evolving Sustainable Data Pipelines

Full post here:

Avatar for Hakka Labs

Hakka Labs

February 13, 2015
Tweet

More Decks by Hakka Labs

See All by Hakka Labs
New Workflows for Building Data Pipelines
Avatar for Hakka Labs hakka_labs
0
2.9k
Collaborative Topic Models for Users and Texts
Avatar for Hakka Labs hakka_labs
0
2.8k
Groupcache with Evan Owen
Avatar for Hakka Labs hakka_labs
2
5.4k
Testing Android at Spotify
Avatar for Hakka Labs hakka_labs
1
4.5k
It's Not a Bug, It's a Feature!
Avatar for Hakka Labs hakka_labs
0
3.2k
K-means Clustering to Understand Your Users
Avatar for Hakka Labs hakka_labs
0
2k
Building Amy: The Email-based Virtual Assistant by x.ai
Avatar for Hakka Labs hakka_labs
0
5k
Deep Learning and NLP Applications
Avatar for Hakka Labs hakka_labs
3
13k
Go and the Gophers
Avatar for Hakka Labs hakka_labs
2
11k

Other Decks in Programming

See All in Programming
AIに安心して任せるためにTypeScriptで一意な型を作ろう
Avatar for Yuki Yata arfes0e2b3c
0
370
대규모 트래픽을 처리하는 프론트 개발자의 전략
Avatar for Seungmin 마량 maryang
0
120
Constant integer division faster than compiler-generated code
Avatar for herumi herumi
2
670
decksh - a little language for decks
Avatar for Anthony Starks ajstarks
4
21k
Reactの歴史を振り返る
Avatar for つちのこ tutinoko
1
180
Google I/O recap web編 大分Web祭り2025
Avatar for kponda kponda
0
2.9k
実践！App Intents対応
Avatar for 野瀬田　裕樹 yuukiw00w
1
280
画像コンペでのベースラインモデルの育て方
Avatar for tattaka tattaka
3
1.7k
それ CLI フレームワークがなくてもできるよ / Building CLI Tools Without Frameworks
Avatar for Kuniwak orgachem
PRO
17
3.9k
The State of Fluid (2025)
Avatar for Simon Praetorius s2b
0
170
サイトを作ったらNFCタグキーホルダーを爆速で作れ！
Avatar for Yuuki Shimizu yuukis
0
360
コーディングは技術者(エンジニア)の嗜みでして / Learning the System Development Mindset from Rock Lady
Avatar for mackey0225 mackey0225
2
500

Featured

See All Featured
Design and Strategy: How to Deal with People Who Don’t "Get" Design
Avatar for Morgane Peng morganepeng
131
19k
How STYLIGHT went responsive
Avatar for nonsquared nonsquared
100
5.7k
Templates, Plugins, & Blocks: Oh My! Creating the theme that thinks of everything
Avatar for Michelle Schulp Hunt marktimemedia
31
2.5k
Thoughts on Productivity
Avatar for Jon Yablonski jonyablonski
69
4.8k
The Psychology of Web Performance [Beyond Tellerrand 2023]
Avatar for Tammy Everts tammyeverts
49
3k
Side Projects
Avatar for Sacha Greif sachag
455
43k
KATA
Avatar for Megan Lloyd mclloyd
32
14k
Designing Experiences People Love
Avatar for Jonathan Moore moore
142
24k
Adopting Sorbet at Scale
Avatar for Ufuk Kayserilioglu ufuk
77
9.5k
Making the Leap to Tech Lead
Avatar for Ryan Cromwell cromwellryan
134
9.5k
JavaScript: Past, Present, and Future - NDC Porto 2020
Avatar for David Neal reverentgeek
50
5.5k
jQuery: Nuts, Bolts and Bling
Avatar for Doug Neiner dougneiner
64
7.9k

Transcript

  1. None

  2. ESP Evolving Sustainable {data} Pipelines Anna Smith - @OMGannaks 29

    January 2015

  3. A bit about RTR A bit about our analytics Some

    evolution stuff A bit about Luigi The future bits

  4. A bit about RTR A bit about our analytics Some

    evolution stuff A bit about Luigi The future bits
  5. None

  6. 1. 2. 3.

  7. WHERE THE MAGIC HAPPENS

  8. A bit about RTR A bit about our analytics Some

    evolution stuff A bit about Luigi The future bits

  9. DEMAND website user interaction transactions SUPPLY warehouse inventory allocation

  10. DEMAND website user interaction transactions SUPPLY warehouse inventory allocation INTERACTION

    reservation calendar
  11. None
  12. None
  13. None

  14. A bit about RTR A bit about our analytics Some

    evolution stuff A bit about Luigi The future bits

  15. PHASE 0

  16. None

  17. YUP

  18. KEEP IT TOGETHER

  19. PHASE 1 primordial soup

  20. STABILITY

  21. PHASE 2 process

  22. orderwarehouse.job

  23. $ ./runjob.py orderwarehouse.job $ ./runjob.py orderwarehouse.job --show $ ./runjob.py orderwarehouse.job

    --only 2

  24. runjob.py

  25. ADAPTING ensuring data quality

  26. PHASE 3 exposing weaknesses

  27. A bit about RTR A bit about our analytics Some

    evolution stuff A bit about Luigi The future bits

  28. dependency manager

  29. None
  30. None

  31. A bit about RTR A bit about our analytics Some

    evolution stuff A bit about Luigi The future bits

  32. PHASE 4 ownership

  33. RELIABILITY

  34. COMMUNICATION

  35. THE FUTURE @OMGannaks [email protected]