Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Evolving Sustainable Data Pipelines

Sponsored · SiteGround - Reliable hosting with speed, security, and support you can count on. →
Avatar for Hakka Labs Hakka Labs
February 13, 2015
Programming
0
3.5k

Evolving Sustainable Data Pipelines

Full post here:

Avatar for Hakka Labs

Hakka Labs

February 13, 2015
Tweet

More Decks by Hakka Labs

See All by Hakka Labs
New Workflows for Building Data Pipelines
Avatar for Hakka Labs hakka_labs
0
2.9k
Collaborative Topic Models for Users and Texts
Avatar for Hakka Labs hakka_labs
0
2.8k
Groupcache with Evan Owen
Avatar for Hakka Labs hakka_labs
2
5.4k
Testing Android at Spotify
Avatar for Hakka Labs hakka_labs
1
4.5k
It's Not a Bug, It's a Feature!
Avatar for Hakka Labs hakka_labs
0
3.2k
K-means Clustering to Understand Your Users
Avatar for Hakka Labs hakka_labs
0
2k
Building Amy: The Email-based Virtual Assistant by x.ai
Avatar for Hakka Labs hakka_labs
0
5k
Deep Learning and NLP Applications
Avatar for Hakka Labs hakka_labs
3
13k
Go and the Gophers
Avatar for Hakka Labs hakka_labs
2
11k

Other Decks in Programming

See All in Programming
AI & Enginnering
Avatar for codelynx codelynx
0
110
HTTPプロトコル正しく理解していますか？ 〜かわいい猫と共に学ぼう。ฅ^•ω•^ฅ ﾆｬ〜
Avatar for ♛Heitor Hirose hekuchan
2
690
Fluid Templating in TYPO3 14
Avatar for Simon Praetorius s2b
0
130
FOSDEM 2026: STUNMESH-go: Building P2P WireGuard Mesh Without Self-Hosted Infrastructure
Avatar for Date Huang tjjh89017
0
170
ぼくの開発環境2026
Avatar for yuzneri yuzneri
0
230
CSC307 Lecture 02
Avatar for Javier Gonzalez-Sanchez javiergs
PRO
1
780
責任感のあるCloudWatchアラームを設計しよう
Avatar for アキキー akihisaikeda
3
180
なぜSQLはAIぽく見えるのか/why does SQL look AI like
Avatar for florets1 florets1
0
460
Best-Practices-for-Cortex-Analyst-and-AI-Agent
Avatar for ryotaro.ikeda@finatext.com ryotaroikeda
1
110
izumin5210のプロポーザルのネタ探し #tskaigi_msup
Avatar for izumin5210 izumin5210
1
130
360° Signals in Angular: Signal Forms with SignalStore & Resources @ngLondon 01/2026
Avatar for Manfred Steyer manfredsteyer
PRO
0
130
Patterns of Patterns
Avatar for Denys Poltorak denyspoltorak
0
1.4k

Featured

See All Featured
Why Your Marketing Sucks and What You Can Do About It - Sophie Logan
Avatar for Sophie Logan marketingsoph
0
75
How People are Using Generative and Agentic AI to Supercharge Their Products, Projects, Services and Value Streams Today
Avatar for Helen Beal helenjbeal
1
120
SEOcharity - Dark patterns in SEO and UX: How to avoid them and build a more ethical web
Avatar for Sara Fernández Carmona sarafernandez
0
120
End of SEO as We Know It (SMX Advanced Version)
Avatar for Michael King ipullrank
3
3.9k
Measuring & Analyzing Core Web Vitals
Avatar for Philip Tellis bluesmoon
9
750
Data-driven link building: lessons from a $708K investment (BrightonSEO talk)
Avatar for Szymon szymonslowik
1
910
How To Speak Unicorn (iThemes Webinar)
Avatar for Michelle Schulp Hunt marktimemedia
1
380
Designing Powerful Visuals for Engaging Learning
Avatar for Mike Taylor tmiket
0
230
Groundhog Day: Seeking Process in Gaming for Health
Avatar for Sebastian Deterding codingconduct
0
93
Pawsitive SEO: Lessons from My Dog (and Many Mistakes) on Thriving as a Consultant in the Age of AI
Avatar for David Carrasco davidcarrasco
0
67
DBのスキルで生き残る技術 - AI時代におけるテーブル設計の勘所
Avatar for soudai sone soudai
PRO
62
50k
[SF Ruby Conf 2025] Rails X
Avatar for Vladimir Dementyev palkan
1
750

Transcript

  1. None

  2. ESP Evolving Sustainable {data} Pipelines Anna Smith - @OMGannaks 29

    January 2015

  3. A bit about RTR A bit about our analytics Some

    evolution stuff A bit about Luigi The future bits

  4. A bit about RTR A bit about our analytics Some

    evolution stuff A bit about Luigi The future bits
  5. None

  6. 1. 2. 3.

  7. WHERE THE MAGIC HAPPENS

  8. A bit about RTR A bit about our analytics Some

    evolution stuff A bit about Luigi The future bits

  9. DEMAND website user interaction transactions SUPPLY warehouse inventory allocation

  10. DEMAND website user interaction transactions SUPPLY warehouse inventory allocation INTERACTION

    reservation calendar
  11. None
  12. None
  13. None

  14. A bit about RTR A bit about our analytics Some

    evolution stuff A bit about Luigi The future bits

  15. PHASE 0

  16. None

  17. YUP

  18. KEEP IT TOGETHER

  19. PHASE 1 primordial soup

  20. STABILITY

  21. PHASE 2 process

  22. orderwarehouse.job

  23. $ ./runjob.py orderwarehouse.job $ ./runjob.py orderwarehouse.job --show $ ./runjob.py orderwarehouse.job

    --only 2

  24. runjob.py

  25. ADAPTING ensuring data quality

  26. PHASE 3 exposing weaknesses

  27. A bit about RTR A bit about our analytics Some

    evolution stuff A bit about Luigi The future bits

  28. dependency manager

  29. None
  30. None

  31. A bit about RTR A bit about our analytics Some

    evolution stuff A bit about Luigi The future bits

  32. PHASE 4 ownership

  33. RELIABILITY

  34. COMMUNICATION

  35. THE FUTURE @OMGannaks [email protected]