Upgrade to Pro — share decks privately, control downloads, hide ads and more …
Sign up for free

Evolving Sustainable Data Pipelines

Avatar for Hakka Labs Hakka Labs
February 13, 2015
Programming
3.5k
0

Evolving Sustainable Data Pipelines

Full post here:

Avatar for Hakka Labs

Hakka Labs

February 13, 2015

More Decks by Hakka Labs

See All by Hakka Labs
New Workflows for Building Data Pipelines
Avatar for Hakka Labs hakka_labs
0
2.9k
Collaborative Topic Models for Users and Texts
Avatar for Hakka Labs hakka_labs
0
2.8k
Groupcache with Evan Owen
Avatar for Hakka Labs hakka_labs
2
5.4k
Testing Android at Spotify
Avatar for Hakka Labs hakka_labs
1
4.5k
It's Not a Bug, It's a Feature!
Avatar for Hakka Labs hakka_labs
0
3.2k
K-means Clustering to Understand Your Users
Avatar for Hakka Labs hakka_labs
0
2k
Building Amy: The Email-based Virtual Assistant by x.ai
Avatar for Hakka Labs hakka_labs
0
5k
Deep Learning and NLP Applications
Avatar for Hakka Labs hakka_labs
3
13k
Go and the Gophers
Avatar for Hakka Labs hakka_labs
2
11k

Other Decks in Programming

See All in Programming
数年滞っていたダークモード対応をおよそ2週間で完了させる
Avatar for Kazuki Chigita chigichan24
0
720
Everything will be SERVERLESS — 信じて運用した10年の経験値 / Everything Will be Serverless — Lessons Learned from 10 Years of Operational Experience
Avatar for shiro seike seike460
PRO
1
110
ゲームコントローラやキーボードのファームウェアをSwiftで書く
Avatar for Kishikawa Katsumi kishikawakatsumi
1
240
From 6 People Classroom Meetup to 100 People Regional Conference / FOSS4G Hiroshima 2026
Avatar for YFurukawa furukawayasuto
0
210
すこし踏み込む CancellationToken
Avatar for tkym htkym
0
310
Vue Fes Japan 2026 タイムテーブル徹底解説
Avatar for 448jp | OKI Yoshiya 448jp
1
230
GKE で Pod の見方を変えたら、スケールアウト時の挙動を真に捉えられた話
Avatar for Sota Takaki stkk
0
130
AIは賢い。でも実行環境は？ CLIおじさんがAI時代に伝えたいこと ~ CLIおじさんがAI時代に伝えたいこと ~
Avatar for curekoshimizu curekoshimizu
1
240
テストを司るデーモンに会いに行く 〜隔離した仮想マシンでテストを通すまで〜
Avatar for Hidemune Takahashi h1d3mun3
1
430
AIエージェント時代のコードレビューを設計する
Avatar for nogu nogu66
6
2.7k
市販E-Readerを乗っ取れ 〜Embedded Swiftで電子ペーパーガジェットを制御する〜
Avatar for Trickart trickart
0
190
SONY CISC-NEWS NWS-1750 + NWB-225 フレームバッファの NetBSD/news68k ドライバ実装 / OSC2026Hiroshima
Avatar for Izumi Tsutsui tsutsui
0
120

Featured

See All Featured
Effective software design: The role of men in debugging patriarchy in IT @ Voxxed Days AMS
Avatar for Kenny Baas-Schwegler baasie
1
530
Claude Code のすすめ
Avatar for schroneko schroneko
67
230k
Amusing Abliteration
Avatar for ianozsvald ianozsvald
1
300
Ethics towards AI in product and experience design
Avatar for Skipper Chong Warson skipperchong
2
380
世界の人気アプリ100個を分析して見えたペイウォール設計の心得
Avatar for Akihiro Kokubo akihiro_kokubo
PRO
74
42k
The Curious Case for Waylosing
Avatar for Cassini Nazir cassininazir
1
510
Exploring the Power of Turbo Streams & Action Cable | RailsConf2023
Avatar for Kevin Liebholz kevinliebholz
37
6.6k
Abbi's Birthday
Avatar for Violet Bock coloredviolet
4
10k
RailsConf & Balkan Ruby 2019: The Past, Present, and Future of Rails at GitHub
Avatar for Eileen M. Uchitelle eileencodes
141
35k
Design of three-dimensional binary manipulators for pick-and-place task avoiding obstacles (IECON2024)
Avatar for konakalab konakalab
0
590
Impact Scores and Hybrid Strategies: The future of link building
Avatar for Tamara Novitovic tamaranovitovic
0
440
Odyssey Design
Avatar for Ryan Kendrick rkendrick25
PRO
2
810

Transcript

  1. None

  2. ESP Evolving Sustainable {data} Pipelines Anna Smith - @OMGannaks 29

    January 2015

  3. A bit about RTR A bit about our analytics Some

    evolution stuff A bit about Luigi The future bits

  4. A bit about RTR A bit about our analytics Some

    evolution stuff A bit about Luigi The future bits
  5. None

  6. 1. 2. 3.

  7. WHERE THE MAGIC HAPPENS

  8. A bit about RTR A bit about our analytics Some

    evolution stuff A bit about Luigi The future bits

  9. DEMAND website user interaction transactions SUPPLY warehouse inventory allocation

  10. DEMAND website user interaction transactions SUPPLY warehouse inventory allocation INTERACTION

    reservation calendar
  11. None
  12. None
  13. None

  14. A bit about RTR A bit about our analytics Some

    evolution stuff A bit about Luigi The future bits

  15. PHASE 0

  16. None

  17. YUP

  18. KEEP IT TOGETHER

  19. PHASE 1 primordial soup

  20. STABILITY

  21. PHASE 2 process

  22. orderwarehouse.job

  23. $ ./runjob.py orderwarehouse.job $ ./runjob.py orderwarehouse.job --show $ ./runjob.py orderwarehouse.job

    --only 2

  24. runjob.py

  25. ADAPTING ensuring data quality

  26. PHASE 3 exposing weaknesses

  27. A bit about RTR A bit about our analytics Some

    evolution stuff A bit about Luigi The future bits

  28. dependency manager

  29. None
  30. None

  31. A bit about RTR A bit about our analytics Some

    evolution stuff A bit about Luigi The future bits

  32. PHASE 4 ownership

  33. RELIABILITY

  34. COMMUNICATION

  35. THE FUTURE @OMGannaks [email protected]