Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Evolving Sustainable Data Pipelines

Sponsored · Your Podcast. Everywhere. Effortlessly. Share. Educate. Inspire. Entertain. You do you. We'll handle the rest. →
Avatar for Hakka Labs Hakka Labs
February 13, 2015
Programming
3.5k
0

Evolving Sustainable Data Pipelines

Full post here:

Avatar for Hakka Labs

Hakka Labs

February 13, 2015

More Decks by Hakka Labs

See All by Hakka Labs
New Workflows for Building Data Pipelines
Avatar for Hakka Labs hakka_labs
0
2.9k
Collaborative Topic Models for Users and Texts
Avatar for Hakka Labs hakka_labs
0
2.8k
Groupcache with Evan Owen
Avatar for Hakka Labs hakka_labs
2
5.4k
Testing Android at Spotify
Avatar for Hakka Labs hakka_labs
1
4.5k
It's Not a Bug, It's a Feature!
Avatar for Hakka Labs hakka_labs
0
3.2k
K-means Clustering to Understand Your Users
Avatar for Hakka Labs hakka_labs
0
2k
Building Amy: The Email-based Virtual Assistant by x.ai
Avatar for Hakka Labs hakka_labs
0
5k
Deep Learning and NLP Applications
Avatar for Hakka Labs hakka_labs
3
13k
Go and the Gophers
Avatar for Hakka Labs hakka_labs
2
11k

Other Decks in Programming

See All in Programming
クラウドネイティブなエンジニアに向ける Raycastの魅力と実際の活用事例
Avatar for Nealle nealle
2
190
Kubernetes上でAgentを動かすための最新動向と押さえるべき概念まとめ
Avatar for Sota Makino sotamaki0421
3
500
The Monolith Strikes Back: Why AI Agents ❤️ Rails Monoliths
Avatar for Rodrigo Serradura serradura
0
330
ソフトウェア設計の結合バランス #phperkaigi
Avatar for Takuma Kajikawa kajitack
0
120
2026_04_15_量子計算をパズルとして解く
Avatar for HideakiTakechi hideakitakechi
0
110
ドメインイベントでビジネスロジックを解きほぐす #phpcon_odawara
Avatar for Takuma Kajikawa kajitack
3
770
アクセシビリティ試験の"その後"を仕組み化する
Avatar for yuuumin yuuumiravy
0
150
mruby on C#: From VM Implementation to Game Scripting (RubyKaigi 2026)
Avatar for hadashiA hadashia
2
430
Swift Concurrency Type System
Avatar for Yasuhiro Inami inamiy
0
530
KagglerがMixSeekを触ってみた
Avatar for Morim morim
0
390
Kingdom of the Machine
Avatar for yui-knk yui_knk
2
290
10年分の技術的負債、完済へ ― Claude Code主導のAI駆動開発でスポーツブルを丸ごとリプレイスした話
Avatar for nero15 takuya_houshima
0
2.5k

Featured

See All Featured
The Art of Delivering Value - GDevCon NA Keynote
Avatar for David Neal reverentgeek
16
1.9k
I Don’t Have Time: Getting Over the Fear to Launch Your Podcast
Avatar for Joe Casabona jcasabona
34
2.7k
Getting science done with accelerated Python computing platforms
Avatar for Jacob Tomlinson jacobtomlinson
2
180
世界の人気アプリ100個を分析して見えたペイウォール設計の心得
Avatar for Akihiro Kokubo akihiro_kokubo
PRO
69
39k
Collaborative Software Design: How to facilitate domain modelling decisions
Avatar for Kenny Baas-Schwegler baasie
0
200
Discover your Explorer Soul
Avatar for Emna emna__ayadi
2
1.1k
How to Get Subject Matter Experts Bought In and Actively Contributing to SEO & PR Initiatives.
Avatar for Liv Day livdayseo
0
100
16th Malabo Montpellier Forum Presentation
Avatar for AKADEMIYA2063 akademiya2063
PRO
0
100
Visual Storytelling: How to be a Superhuman Communicator
Avatar for David Neal reverentgeek
2
510
Large-scale JavaScript Application Architecture
Avatar for Addy Osmani addyosmani
515
110k
StorybookのUI Testing Handbookを読んだ
Avatar for Shingo Yamazaki zakiyama
31
6.7k
How to Grow Your eCommerce with AI & Automation
Avatar for Katarina Dahlin katarinadahlin
PRO
1
170

Transcript

  1. None

  2. ESP Evolving Sustainable {data} Pipelines Anna Smith - @OMGannaks 29

    January 2015

  3. A bit about RTR A bit about our analytics Some

    evolution stuff A bit about Luigi The future bits

  4. A bit about RTR A bit about our analytics Some

    evolution stuff A bit about Luigi The future bits
  5. None

  6. 1. 2. 3.

  7. WHERE THE MAGIC HAPPENS

  8. A bit about RTR A bit about our analytics Some

    evolution stuff A bit about Luigi The future bits

  9. DEMAND website user interaction transactions SUPPLY warehouse inventory allocation

  10. DEMAND website user interaction transactions SUPPLY warehouse inventory allocation INTERACTION

    reservation calendar
  11. None
  12. None
  13. None

  14. A bit about RTR A bit about our analytics Some

    evolution stuff A bit about Luigi The future bits

  15. PHASE 0

  16. None

  17. YUP

  18. KEEP IT TOGETHER

  19. PHASE 1 primordial soup

  20. STABILITY

  21. PHASE 2 process

  22. orderwarehouse.job

  23. $ ./runjob.py orderwarehouse.job $ ./runjob.py orderwarehouse.job --show $ ./runjob.py orderwarehouse.job

    --only 2

  24. runjob.py

  25. ADAPTING ensuring data quality

  26. PHASE 3 exposing weaknesses

  27. A bit about RTR A bit about our analytics Some

    evolution stuff A bit about Luigi The future bits

  28. dependency manager

  29. None
  30. None

  31. A bit about RTR A bit about our analytics Some

    evolution stuff A bit about Luigi The future bits

  32. PHASE 4 ownership

  33. RELIABILITY

  34. COMMUNICATION

  35. THE FUTURE @OMGannaks [email protected]