Upgrade to PRO for Only $50/Year—Limited-Time Offer! 🔥

Evolving Sustainable Data Pipelines

Avatar for Hakka Labs Hakka Labs
February 13, 2015
Programming
0
3.5k

Evolving Sustainable Data Pipelines

Full post here:

Avatar for Hakka Labs

Hakka Labs

February 13, 2015
Tweet

More Decks by Hakka Labs

See All by Hakka Labs
New Workflows for Building Data Pipelines
Avatar for Hakka Labs hakka_labs
0
2.9k
Collaborative Topic Models for Users and Texts
Avatar for Hakka Labs hakka_labs
0
2.8k
Groupcache with Evan Owen
Avatar for Hakka Labs hakka_labs
2
5.4k
Testing Android at Spotify
Avatar for Hakka Labs hakka_labs
1
4.5k
It's Not a Bug, It's a Feature!
Avatar for Hakka Labs hakka_labs
0
3.2k
K-means Clustering to Understand Your Users
Avatar for Hakka Labs hakka_labs
0
2k
Building Amy: The Email-based Virtual Assistant by x.ai
Avatar for Hakka Labs hakka_labs
0
5k
Deep Learning and NLP Applications
Avatar for Hakka Labs hakka_labs
3
13k
Go and the Gophers
Avatar for Hakka Labs hakka_labs
2
11k

Other Decks in Programming

See All in Programming
AIエージェントを活かすPM術 AI駆動開発の現場から
Avatar for gyuta gyuta
0
440
Cap'n Webについて
Avatar for Yusuke Wada yusukebe
0
140
AI時代を生き抜く 新卒エンジニアの生きる道
Avatar for coconala_engineer coconala_engineer
1
350
Pythonではじめるオープンデータ分析〜書籍の紹介と書籍で紹介しきれなかった事例の紹介〜
Avatar for Ryo YOSHI welliving
1
290
20251212 AI 時代的 Legacy Code 營救術 2025 WebConf
Avatar for mouson(墨嗓) mouson
0
200
リリース時」テストから「デイリー実行」へ！開発マネージャが取り組んだ、レガシー自動テストのモダン化戦略
Avatar for goataka (GOAMI Takaaki) goataka
0
130
Microservices rules: What good looks like
Avatar for Chris Richardson cer
PRO
0
1.5k
Rediscover the Console - SymfonyCon Amsterdam 2025
Avatar for Robin Chalas chalasr
2
180
FluorTracer / RayTracingCamp11
Avatar for kugi kugimasa
0
240
Giselleで作るAI QAアシスタント 〜 Pull Requestレビューに継続的QAを
Avatar for Tadashi Shigeoka codenote
0
250
TUIライブラリつくってみた / i-just-make-TUI-library
Avatar for kazto kazto
1
400
The Past, Present, and Future of Enterprise Java
Avatar for ivargrimstad ivargrimstad
0
200

Featured

See All Featured
Imperfection Machines: The Place of Print at Facebook
Avatar for Scott Boms scottboms
269
13k
Building Adaptive Systems
Avatar for Chris Keathley keathley
44
2.9k
Sharpening the Axe: The Primacy of Toolmaking
Avatar for Bryan Cantrill bcantrill
46
2.6k
Large-scale JavaScript Application Architecture
Avatar for Addy Osmani addyosmani
515
110k
Typedesign – Prime Four
Avatar for Hannes Fritz hannesfritz
42
2.9k
Balancing Empowerment & Direction
Avatar for Lara Hogan lara
5
810
Kristin Tynski - Automating Marketing Tasks With AI
Avatar for Tech SEO Connect techseoconnect
PRO
0
110
Building a Modern Day 
E-commerce SEO Strategy
Avatar for Aleyda Solis aleyda
45
8.3k
Easily Structure & Communicate Ideas using Wireframe
Avatar for Afnizar Nur Ghifari afnizarnur
194
17k
More Than Pixels: Becoming A User Experience Designer
Avatar for Michelle Schulp Hunt marktimemedia
2
250
Fashionably flexible responsive web design (full day workshop)
Avatar for Andy Clarke malarkey
407
66k
Embracing the Ebb and Flow
Avatar for Simon Collison colly
88
4.9k

Transcript

  1. None

  2. ESP Evolving Sustainable {data} Pipelines Anna Smith - @OMGannaks 29

    January 2015

  3. A bit about RTR A bit about our analytics Some

    evolution stuff A bit about Luigi The future bits

  4. A bit about RTR A bit about our analytics Some

    evolution stuff A bit about Luigi The future bits
  5. None

  6. 1. 2. 3.

  7. WHERE THE MAGIC HAPPENS

  8. A bit about RTR A bit about our analytics Some

    evolution stuff A bit about Luigi The future bits

  9. DEMAND website user interaction transactions SUPPLY warehouse inventory allocation

  10. DEMAND website user interaction transactions SUPPLY warehouse inventory allocation INTERACTION

    reservation calendar
  11. None
  12. None
  13. None

  14. A bit about RTR A bit about our analytics Some

    evolution stuff A bit about Luigi The future bits

  15. PHASE 0

  16. None

  17. YUP

  18. KEEP IT TOGETHER

  19. PHASE 1 primordial soup

  20. STABILITY

  21. PHASE 2 process

  22. orderwarehouse.job

  23. $ ./runjob.py orderwarehouse.job $ ./runjob.py orderwarehouse.job --show $ ./runjob.py orderwarehouse.job

    --only 2

  24. runjob.py

  25. ADAPTING ensuring data quality

  26. PHASE 3 exposing weaknesses

  27. A bit about RTR A bit about our analytics Some

    evolution stuff A bit about Luigi The future bits

  28. dependency manager

  29. None
  30. None

  31. A bit about RTR A bit about our analytics Some

    evolution stuff A bit about Luigi The future bits

  32. PHASE 4 ownership

  33. RELIABILITY

  34. COMMUNICATION

  35. THE FUTURE @OMGannaks [email protected]