Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Evolving Sustainable Data Pipelines

Avatar for Hakka Labs Hakka Labs
February 13, 2015
Programming
0
3.5k

Evolving Sustainable Data Pipelines

Full post here:

Avatar for Hakka Labs

Hakka Labs

February 13, 2015
Tweet

More Decks by Hakka Labs

See All by Hakka Labs
New Workflows for Building Data Pipelines
Avatar for Hakka Labs hakka_labs
0
2.9k
Collaborative Topic Models for Users and Texts
Avatar for Hakka Labs hakka_labs
0
2.8k
Groupcache with Evan Owen
Avatar for Hakka Labs hakka_labs
2
5.4k
Testing Android at Spotify
Avatar for Hakka Labs hakka_labs
1
4.5k
It's Not a Bug, It's a Feature!
Avatar for Hakka Labs hakka_labs
0
3.2k
K-means Clustering to Understand Your Users
Avatar for Hakka Labs hakka_labs
0
2k
Building Amy: The Email-based Virtual Assistant by x.ai
Avatar for Hakka Labs hakka_labs
0
5k
Deep Learning and NLP Applications
Avatar for Hakka Labs hakka_labs
3
13k
Go and the Gophers
Avatar for Hakka Labs hakka_labs
2
11k

Other Decks in Programming

See All in Programming
TypeScriptで設計する 堅牢さとUXを両立した非同期ワークフローの実現
Avatar for Matsumoto Moeka moeka__c
5
2.5k
無秩序からの脱却 / Emergence from chaos
Avatar for nrs nrslib
1
9.4k
Herb to ReActionView: A New Foundation for the View Layer @ San Francisco Ruby Conference 2025
Avatar for Marco Roth marcoroth
0
200
Nitro v3
Avatar for kazupon kazupon
2
330
Vueで学ぶデータ構造入門 リンクリストとキューでリアクティビティを捉える / Vue Data Structures: Linked Lists and Queues for Reactivity
Avatar for konkarin konkarin
1
350
スタートアップを支える技術戦略と組織づくり
Avatar for pospome pospome
8
12k
データファイルをAWSのDWHサービスに格納する / 20251115jawsug-tochigi
Avatar for kasacchiful kasacchiful
2
100
AI時代もSEOを頑張っている話
Avatar for しらはま shirahama_x
0
180
モデル駆動設計をやってみよう Modeling Forum2025ワークショップ/Let’s Try Model-Driven Design
Avatar for haru860 haru860
0
190
AIを駆使して新しい技術を効率的に理解する方法
Avatar for nogu nogu66
1
660
Full-Cycle Reactivity in Angular: SignalStore mit Signal Forms und Resources
Avatar for Manfred Steyer manfredsteyer
PRO
0
130
TVerのWeb内製化 - 開発スピードと品質を両立させるまでの道のり
Avatar for TVer Inc. techtver
PRO
3
1.2k

Featured

See All Featured
BBQ
Avatar for Matthew Crist matthewcrist
89
9.9k
The Web Performance Landscape in 2024 [PerfNow 2024]
Avatar for Tammy Everts tammyeverts
11
940
A better future with KSS
Avatar for Kyle Neath kneath
239
18k
Navigating Team Friction
Avatar for Lara Hogan lara
190
16k
Music & Morning Musume
Avatar for Bryan Veloso bryan
46
7k
Optimizing for Happiness
Avatar for Tom Preston-Werner mojombo
379
70k
Site-Speed That Sticks
Avatar for Harry Roberts csswizardry
13
970
Scaling GitHub
Avatar for Zach Holman holman
464
140k
Balancing Empowerment & Direction
Avatar for Lara Hogan lara
5
760
How Fast Is Fast Enough? [PerfNow 2025]
Avatar for Tammy Everts tammyeverts
3
350
Easily Structure & Communicate Ideas using Wireframe
Avatar for Afnizar Nur Ghifari afnizarnur
194
17k
Exploring the Power of Turbo Streams & Action Cable | RailsConf2023
Avatar for Kevin Liebholz kevinliebholz
36
6.1k

Transcript

  1. None

  2. ESP Evolving Sustainable {data} Pipelines Anna Smith - @OMGannaks 29

    January 2015

  3. A bit about RTR A bit about our analytics Some

    evolution stuff A bit about Luigi The future bits

  4. A bit about RTR A bit about our analytics Some

    evolution stuff A bit about Luigi The future bits
  5. None

  6. 1. 2. 3.

  7. WHERE THE MAGIC HAPPENS

  8. A bit about RTR A bit about our analytics Some

    evolution stuff A bit about Luigi The future bits

  9. DEMAND website user interaction transactions SUPPLY warehouse inventory allocation

  10. DEMAND website user interaction transactions SUPPLY warehouse inventory allocation INTERACTION

    reservation calendar
  11. None
  12. None
  13. None

  14. A bit about RTR A bit about our analytics Some

    evolution stuff A bit about Luigi The future bits

  15. PHASE 0

  16. None

  17. YUP

  18. KEEP IT TOGETHER

  19. PHASE 1 primordial soup

  20. STABILITY

  21. PHASE 2 process

  22. orderwarehouse.job

  23. $ ./runjob.py orderwarehouse.job $ ./runjob.py orderwarehouse.job --show $ ./runjob.py orderwarehouse.job

    --only 2

  24. runjob.py

  25. ADAPTING ensuring data quality

  26. PHASE 3 exposing weaknesses

  27. A bit about RTR A bit about our analytics Some

    evolution stuff A bit about Luigi The future bits

  28. dependency manager

  29. None
  30. None

  31. A bit about RTR A bit about our analytics Some

    evolution stuff A bit about Luigi The future bits

  32. PHASE 4 ownership

  33. RELIABILITY

  34. COMMUNICATION

  35. THE FUTURE @OMGannaks [email protected]