Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Evolving Sustainable Data Pipelines

Sponsored · Ship Features Fearlessly Turn features on and off without deploys. Used by thousands of Ruby developers. →
Avatar for Hakka Labs Hakka Labs
February 13, 2015
Programming
0
3.5k

Evolving Sustainable Data Pipelines

Full post here:

Avatar for Hakka Labs

Hakka Labs

February 13, 2015
Tweet

More Decks by Hakka Labs

See All by Hakka Labs
New Workflows for Building Data Pipelines
Avatar for Hakka Labs hakka_labs
0
2.9k
Collaborative Topic Models for Users and Texts
Avatar for Hakka Labs hakka_labs
0
2.8k
Groupcache with Evan Owen
Avatar for Hakka Labs hakka_labs
2
5.4k
Testing Android at Spotify
Avatar for Hakka Labs hakka_labs
1
4.5k
It's Not a Bug, It's a Feature!
Avatar for Hakka Labs hakka_labs
0
3.2k
K-means Clustering to Understand Your Users
Avatar for Hakka Labs hakka_labs
0
2k
Building Amy: The Email-based Virtual Assistant by x.ai
Avatar for Hakka Labs hakka_labs
0
5k
Deep Learning and NLP Applications
Avatar for Hakka Labs hakka_labs
3
13k
Go and the Gophers
Avatar for Hakka Labs hakka_labs
2
11k

Other Decks in Programming

See All in Programming
20260228_JAWS_Beginner_Kansai
Avatar for Takuya Yonezawa takuyay0ne
5
530
ロボットのための工場に灯りは要らない
Avatar for watany watany
10
2.9k
Go 1.26でのsliceのメモリアロケーション最適化 / Go 1.26 リリースパーティ #go126party
Avatar for mazrean mazrean
1
400
Cyrius ーLinux非依存にコンテナをネイティブ実行する専用OSー
Avatar for n4mlz n4mlz
0
150
Ruby and LLM Ecosystem 2nd
Avatar for Koichi ITO koic
1
810
AI 開発合宿を通して得た学び
Avatar for ニフティ株式会社 niftycorp
PRO
0
120
Claude Codeセッション現状確認 2026福岡 / fukuoka-aicoding-00-beacon
Avatar for monochromegane monochromegane
4
420
エンジニアの「手元の自動化」を加速するn8n 2026.02.27
Avatar for 西尾義英 symy2co
0
160
守る「だけ」の優しいEMを抜けて、 事業とチームを両方見る視点を身につけた話
Avatar for Mitsui maroon8021
3
950
AI時代でも変わらない技術コミュニティの力～10年続く“ゆるい”つながりが生み出す価値
Avatar for Takehata Naoto n_takehata
2
750
AI Assistants for Your Angular Solutions
Avatar for Manfred Steyer manfredsteyer
PRO
0
140
AI時代のシステム設計：ドメインモデルで変更しやすさを守る設計戦略
Avatar for 増田 亨 masuda220
PRO
5
1k

Featured

See All Featured
How to Align SEO within the Product Triangle To Get 
Buy-In & Support - #RIMC
Avatar for Aleyda Solis aleyda
1
1.4k
Un-Boring Meetings
Avatar for Sebastian Deterding codingconduct
0
220
Building an army of robots
Avatar for Kyle Neath kneath
306
46k
From Legacy to Launchpad: Building Startup-Ready Communities
Avatar for Dug Song dugsong
0
180
Product Roadmaps are Hard
Avatar for C. Todd Lombardo iamctodd
PRO
55
12k
Tips & Tricks on How to Get Your First Job In Tech
Avatar for Honza Javorek honzajavorek
0
450
Sharpening the Axe: The Primacy of Toolmaking
Avatar for Bryan Cantrill bcantrill
46
2.7k
Lightning talk: Run Django tests with GitHub Actions
Avatar for Sarah Abderemane sabderemane
0
150
The Organizational Zoo: Understanding Human Behavior Agility Through Metaphoric Constructive Conversations (based on the works of Arthur Shelley, Ph.D)
Avatar for Dr. Kim W Petersen kimpetersen
PRO
0
270
Designing for humans not robots
Avatar for Tammie Lister tammielis
254
26k
Building a Scalable Design System with Sketch
Avatar for Laura Van Doore lauravandoore
463
34k
Why Your Marketing Sucks and What You Can Do About It - Sophie Logan
Avatar for Sophie Logan marketingsoph
0
110

Transcript

  1. None

  2. ESP Evolving Sustainable {data} Pipelines Anna Smith - @OMGannaks 29

    January 2015

  3. A bit about RTR A bit about our analytics Some

    evolution stuff A bit about Luigi The future bits

  4. A bit about RTR A bit about our analytics Some

    evolution stuff A bit about Luigi The future bits
  5. None

  6. 1. 2. 3.

  7. WHERE THE MAGIC HAPPENS

  8. A bit about RTR A bit about our analytics Some

    evolution stuff A bit about Luigi The future bits

  9. DEMAND website user interaction transactions SUPPLY warehouse inventory allocation

  10. DEMAND website user interaction transactions SUPPLY warehouse inventory allocation INTERACTION

    reservation calendar
  11. None
  12. None
  13. None

  14. A bit about RTR A bit about our analytics Some

    evolution stuff A bit about Luigi The future bits

  15. PHASE 0

  16. None

  17. YUP

  18. KEEP IT TOGETHER

  19. PHASE 1 primordial soup

  20. STABILITY

  21. PHASE 2 process

  22. orderwarehouse.job

  23. $ ./runjob.py orderwarehouse.job $ ./runjob.py orderwarehouse.job --show $ ./runjob.py orderwarehouse.job

    --only 2

  24. runjob.py

  25. ADAPTING ensuring data quality

  26. PHASE 3 exposing weaknesses

  27. A bit about RTR A bit about our analytics Some

    evolution stuff A bit about Luigi The future bits

  28. dependency manager

  29. None
  30. None

  31. A bit about RTR A bit about our analytics Some

    evolution stuff A bit about Luigi The future bits

  32. PHASE 4 ownership

  33. RELIABILITY

  34. COMMUNICATION

  35. THE FUTURE @OMGannaks [email protected]