Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Evolving Sustainable Data Pipelines

Sponsored · SiteGround - Reliable hosting with speed, security, and support you can count on. →
Avatar for Hakka Labs Hakka Labs
February 13, 2015
Programming
3.5k
0

Evolving Sustainable Data Pipelines

Full post here:

Avatar for Hakka Labs

Hakka Labs

February 13, 2015

More Decks by Hakka Labs

See All by Hakka Labs
New Workflows for Building Data Pipelines
Avatar for Hakka Labs hakka_labs
0
2.9k
Collaborative Topic Models for Users and Texts
Avatar for Hakka Labs hakka_labs
0
2.8k
Groupcache with Evan Owen
Avatar for Hakka Labs hakka_labs
2
5.4k
Testing Android at Spotify
Avatar for Hakka Labs hakka_labs
1
4.5k
It's Not a Bug, It's a Feature!
Avatar for Hakka Labs hakka_labs
0
3.2k
K-means Clustering to Understand Your Users
Avatar for Hakka Labs hakka_labs
0
2k
Building Amy: The Email-based Virtual Assistant by x.ai
Avatar for Hakka Labs hakka_labs
0
5k
Deep Learning and NLP Applications
Avatar for Hakka Labs hakka_labs
3
13k
Go and the Gophers
Avatar for Hakka Labs hakka_labs
2
11k

Other Decks in Programming

See All in Programming
PicoRuby for IoT: Connecting to the Cloud with MQTT
Avatar for Y_uuu yuuu
2
770
Agent Skills を社内で育てる仕組み作り
Avatar for jackchuka jackchuka
1
1.8k
The Past, Present, and Future of Enterprise Java
Avatar for ivargrimstad ivargrimstad
0
240
t *testing.T は どこからやってくるの？
Avatar for Kotaro Otaka otakakot
1
920
ハーネスエンジニアリングにどう向き合うか 〜ルールファイルを超えて開発プロセスを設計する〜 / How to approach harness engineering
Avatar for r-kagaya rkaga
28
19k
Building on Bluesky's AT Protocol with Ruby
Avatar for Kuba Suder mackuba
0
110
Kingdom of the Machine
Avatar for yui-knk yui_knk
2
1.5k
AIを導入する前にやるべきこと
Avatar for Negima negima
2
340
属人化しないコード品質の作り方_2026.04.07.pdf
Avatar for Masaki Murano muraaano
0
330
運転動画を検索可能にする〜Cosmos-Embed1とDatabricks Vector Searchで〜/cosmos-embed1-databricks-vector-search
Avatar for 開発室Graph studio_graph
1
680
WebAssembly を読み込むベストプラクティス 2026年春版 / Best Practices for Loading WebAssembly (Spring 2026)
Avatar for petamoriken / 森建 petamoriken
5
1.1k
cloudnative conference 2026 flyle
Avatar for azihsoyn azihsoyn
0
160

Featured

See All Featured
Digital Ethics as a Driver of Design Innovation
Avatar for Per Axbom axbom
PRO
1
280
Evolving SEO for Evolving Search Engines
Avatar for Ryan Jones ryanjones
0
190
CSS Pre-Processors: Stylus, Less & Sass
Avatar for Bermon Painter bermonpainter
360
30k
How to Get Subject Matter Experts Bought In and Actively Contributing to SEO & PR Initiatives.
Avatar for Liv Day livdayseo
0
110
How to audit for AI Accessibility on your Front & Back End
Avatar for davetheseo davetheseo
0
360
Into the Great Unknown - MozCon
Avatar for Noah Learner thekraken
41
2.5k
ラッコキーワード サービス紹介資料
Avatar for ラッコ株式会社 rakko
1
3.3M
The Director’s Chair: Orchestrating AI for Truly Effective Learning
Avatar for Mike Taylor tmiket
1
160
Distributed Sagas: A Protocol for Coordinating Microservices
Avatar for Caitie McCaffrey caitiem20
333
22k
Mind Mapping
Avatar for Hélio Medeiros helmedeiros
PRO
1
190
End of SEO as We Know It (SMX Advanced Version)
Avatar for Michael King ipullrank
3
4.2k
A Guide to Academic Writing Using Generative AI - A Workshop
Avatar for Kenji Saito ks91
PRO
1
300

Transcript

  1. None

  2. ESP Evolving Sustainable {data} Pipelines Anna Smith - @OMGannaks 29

    January 2015

  3. A bit about RTR A bit about our analytics Some

    evolution stuff A bit about Luigi The future bits

  4. A bit about RTR A bit about our analytics Some

    evolution stuff A bit about Luigi The future bits
  5. None

  6. 1. 2. 3.

  7. WHERE THE MAGIC HAPPENS

  8. A bit about RTR A bit about our analytics Some

    evolution stuff A bit about Luigi The future bits

  9. DEMAND website user interaction transactions SUPPLY warehouse inventory allocation

  10. DEMAND website user interaction transactions SUPPLY warehouse inventory allocation INTERACTION

    reservation calendar
  11. None
  12. None
  13. None

  14. A bit about RTR A bit about our analytics Some

    evolution stuff A bit about Luigi The future bits

  15. PHASE 0

  16. None

  17. YUP

  18. KEEP IT TOGETHER

  19. PHASE 1 primordial soup

  20. STABILITY

  21. PHASE 2 process

  22. orderwarehouse.job

  23. $ ./runjob.py orderwarehouse.job $ ./runjob.py orderwarehouse.job --show $ ./runjob.py orderwarehouse.job

    --only 2

  24. runjob.py

  25. ADAPTING ensuring data quality

  26. PHASE 3 exposing weaknesses

  27. A bit about RTR A bit about our analytics Some

    evolution stuff A bit about Luigi The future bits

  28. dependency manager

  29. None
  30. None

  31. A bit about RTR A bit about our analytics Some

    evolution stuff A bit about Luigi The future bits

  32. PHASE 4 ownership

  33. RELIABILITY

  34. COMMUNICATION

  35. THE FUTURE @OMGannaks [email protected]