Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Evolving Sustainable Data Pipelines

Hakka Labs
February 13, 2015
Programming
0
3.5k

Evolving Sustainable Data Pipelines

Full post here:

Hakka Labs

February 13, 2015
Tweet

More Decks by Hakka Labs

See All by Hakka Labs
New Workflows for Building Data Pipelines
hakka_labs
0
2.9k
Collaborative Topic Models for Users and Texts
hakka_labs
0
2.8k
Groupcache with Evan Owen
hakka_labs
2
5.3k
Testing Android at Spotify
hakka_labs
1
4.5k
It's Not a Bug, It's a Feature!
hakka_labs
0
3.2k
K-means Clustering to Understand Your Users
hakka_labs
0
2k
Building Amy: The Email-based Virtual Assistant by x.ai
hakka_labs
0
5k
Deep Learning and NLP Applications
hakka_labs
3
13k
Go and the Gophers
hakka_labs
2
11k

Other Decks in Programming

See All in Programming
【re:Growth 2024】 Aurora DSQL をちゃんと話します！
maroon1st
0
780
Exploring: Partial and Independent Composables
blackbracken
0
100
rails statsで大解剖 🔍 “B/43流” のRailsの育て方を歴史とともに振り返ります
shoheimitani
2
940
Spatial Rendering for Apple Vision Pro
warrenm
0
110
fs2-io を試してたらバグを見つけて直した話
chencmd
0
240
tidymodelsによるtidyな生存時間解析 / Japan.R2024
dropout009
1
790
Stackless и stackful? Корутины и асинхронность в Go
lamodatech
0
830
開発者とQAの越境で自動テストが増える開発プロセスを実現する
92thunder
1
190
menu基盤チームによるGoogle Cloudの活用事例~Application Integration, Cloud Tasks編~
yoshifumi_ishikura
0
110
CSC305 Lecture 26
javiergs
PRO
0
140
Go の GC の不得意な 部分を克服したい
taiyow
3
800
103 Early Hints
sugi_0000
1
230

Featured

See All Featured
The Language of Interfaces
destraynor
154
24k
The Web Performance Landscape in 2024 [PerfNow 2024]
tammyeverts
2
290
The Illustrated Children's Guide to Kubernetes
chrisshort
48
48k
CoffeeScript is Beautiful & I Never Want to Write Plain JavaScript Again
sstephenson
159
15k
VelocityConf: Rendering Performance Case Studies
addyosmani
326
24k
Product Roadmaps are Hard
iamctodd
PRO
49
11k
Easily Structure & Communicate Ideas using Wireframe
afnizarnur
191
16k
How to Think Like a Performance Engineer
csswizardry
22
1.2k
GitHub's CSS Performance
jonrohan
1030
460k
Adopting Sorbet at Scale
ufuk
73
9.1k
For a Future-Friendly Web
brad_frost
175
9.4k
[Rails World 2023 - Day 1 Closing Keynote] - The Magic of Rails
eileencodes
33
2k

Transcript

  1. None

  2. ESP Evolving Sustainable {data} Pipelines Anna Smith - @OMGannaks 29

    January 2015

  3. A bit about RTR A bit about our analytics Some

    evolution stuff A bit about Luigi The future bits

  4. A bit about RTR A bit about our analytics Some

    evolution stuff A bit about Luigi The future bits
  5. None

  6. 1. 2. 3.

  7. WHERE THE MAGIC HAPPENS

  8. A bit about RTR A bit about our analytics Some

    evolution stuff A bit about Luigi The future bits

  9. DEMAND website user interaction transactions SUPPLY warehouse inventory allocation

  10. DEMAND website user interaction transactions SUPPLY warehouse inventory allocation INTERACTION

    reservation calendar
  11. None
  12. None
  13. None

  14. A bit about RTR A bit about our analytics Some

    evolution stuff A bit about Luigi The future bits

  15. PHASE 0

  16. None

  17. YUP

  18. KEEP IT TOGETHER

  19. PHASE 1 primordial soup

  20. STABILITY

  21. PHASE 2 process

  22. orderwarehouse.job

  23. $ ./runjob.py orderwarehouse.job $ ./runjob.py orderwarehouse.job --show $ ./runjob.py orderwarehouse.job

    --only 2

  24. runjob.py

  25. ADAPTING ensuring data quality

  26. PHASE 3 exposing weaknesses

  27. A bit about RTR A bit about our analytics Some

    evolution stuff A bit about Luigi The future bits

  28. dependency manager

  29. None
  30. None

  31. A bit about RTR A bit about our analytics Some

    evolution stuff A bit about Luigi The future bits

  32. PHASE 4 ownership

  33. RELIABILITY

  34. COMMUNICATION

  35. THE FUTURE @OMGannaks [email protected]