Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Evolving Sustainable Data Pipelines

Hakka Labs
February 13, 2015
Programming
0
3.5k

Evolving Sustainable Data Pipelines

Full post here:

Hakka Labs

February 13, 2015
Tweet

More Decks by Hakka Labs

See All by Hakka Labs
New Workflows for Building Data Pipelines
hakka_labs
0
2.9k
Collaborative Topic Models for Users and Texts
hakka_labs
0
2.8k
Groupcache with Evan Owen
hakka_labs
2
5.3k
Testing Android at Spotify
hakka_labs
1
4.5k
It's Not a Bug, It's a Feature!
hakka_labs
0
3.2k
K-means Clustering to Understand Your Users
hakka_labs
0
2k
Building Amy: The Email-based Virtual Assistant by x.ai
hakka_labs
0
5k
Deep Learning and NLP Applications
hakka_labs
3
13k
Go and the Gophers
hakka_labs
2
11k

Other Decks in Programming

See All in Programming
CloudRun, Spanner に対する負荷試験の反省と オブザーバビリティによるアプローチ
oyasumipants
1
170
Visual StudioのGitHub Copilotでいろいろやってみる
tomokusaba
1
220
Duke on CRaC with Jakarta EE
ivargrimstad
0
160
Ça bouge du côté des animations CSS !
goetter
2
160
Drawing Heighway’s Dragon - Recursive Function Rewrite - From Imperative Style in Pascal 64 To Functional Style in Scala 3
philipschwarz
PRO
0
150
AIプログラミング雑キャッチアップ
yuheinakasaka
20
5.2k
ABEMA iOS 大規模プロジェクトにおける段階的な技術刷新 / ABEMA iOS Technology Upgrade
akkyie
1
250
Swift Testingのモチベを上げたい
stoticdev
2
200
もう僕は OpenAPI を書きたくない
sgash708
6
1.9k
Domain-Driven Design (Tutorial)
hschwentner
13
22k
LINE messaging APIを使ってGoogleカレンダーと連携した予約ツールを作ってみた
takumakoike
0
130
Djangoにおける複数ユーザー種別認証の設計アプローチ@DjangoCongress JP 2025
delhi09
PRO
4
510

Featured

See All Featured
Evolution of real-time – Irina Nazarova, EuRuKo, 2024
irinanazarova
6
580
It's Worth the Effort
3n
184
28k
How GitHub (no longer) Works
holman
314
140k
Producing Creativity
orderedlist
PRO
344
40k
Dealing with People You Can't Stand - Big Design 2015
cassininazir
366
25k
Visualization
eitanlees
146
15k
Cheating the UX When There Is Nothing More to Optimize - PixelPioneers
stephaniewalter
280
13k
Intergalactic Javascript Robots from Outer Space
tanoku
270
27k
Designing Experiences People Love
moore
140
23k
Documentation Writing (for coders)
carmenintech
68
4.6k
Embracing the Ebb and Flow
colly
84
4.6k
Bash Introduction
62gerente
611
210k

Transcript

  1. None

  2. ESP Evolving Sustainable {data} Pipelines Anna Smith - @OMGannaks 29

    January 2015

  3. A bit about RTR A bit about our analytics Some

    evolution stuff A bit about Luigi The future bits

  4. A bit about RTR A bit about our analytics Some

    evolution stuff A bit about Luigi The future bits
  5. None

  6. 1. 2. 3.

  7. WHERE THE MAGIC HAPPENS

  8. A bit about RTR A bit about our analytics Some

    evolution stuff A bit about Luigi The future bits

  9. DEMAND website user interaction transactions SUPPLY warehouse inventory allocation

  10. DEMAND website user interaction transactions SUPPLY warehouse inventory allocation INTERACTION

    reservation calendar
  11. None
  12. None
  13. None

  14. A bit about RTR A bit about our analytics Some

    evolution stuff A bit about Luigi The future bits

  15. PHASE 0

  16. None

  17. YUP

  18. KEEP IT TOGETHER

  19. PHASE 1 primordial soup

  20. STABILITY

  21. PHASE 2 process

  22. orderwarehouse.job

  23. $ ./runjob.py orderwarehouse.job $ ./runjob.py orderwarehouse.job --show $ ./runjob.py orderwarehouse.job

    --only 2

  24. runjob.py

  25. ADAPTING ensuring data quality

  26. PHASE 3 exposing weaknesses

  27. A bit about RTR A bit about our analytics Some

    evolution stuff A bit about Luigi The future bits

  28. dependency manager

  29. None
  30. None

  31. A bit about RTR A bit about our analytics Some

    evolution stuff A bit about Luigi The future bits

  32. PHASE 4 ownership

  33. RELIABILITY

  34. COMMUNICATION

  35. THE FUTURE @OMGannaks asmith@renttherunway.com