Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Evolving Sustainable Data Pipelines

Avatar for Hakka Labs Hakka Labs
February 13, 2015
Programming
0
3.5k

Evolving Sustainable Data Pipelines

Full post here:

Avatar for Hakka Labs

Hakka Labs

February 13, 2015
Tweet

More Decks by Hakka Labs

See All by Hakka Labs
New Workflows for Building Data Pipelines
Avatar for Hakka Labs hakka_labs
0
2.9k
Collaborative Topic Models for Users and Texts
Avatar for Hakka Labs hakka_labs
0
2.8k
Groupcache with Evan Owen
Avatar for Hakka Labs hakka_labs
2
5.4k
Testing Android at Spotify
Avatar for Hakka Labs hakka_labs
1
4.5k
It's Not a Bug, It's a Feature!
Avatar for Hakka Labs hakka_labs
0
3.2k
K-means Clustering to Understand Your Users
Avatar for Hakka Labs hakka_labs
0
2k
Building Amy: The Email-based Virtual Assistant by x.ai
Avatar for Hakka Labs hakka_labs
0
5k
Deep Learning and NLP Applications
Avatar for Hakka Labs hakka_labs
3
13k
Go and the Gophers
Avatar for Hakka Labs hakka_labs
2
11k

Other Decks in Programming

See All in Programming
AIのバカさ加減に怒る前にやっておくこと
Avatar for k2moons blueeventhorizon
0
110
AkarengaLT vol.38
Avatar for hashimoto-kei hashimoto_kei
1
130
Foundation Modelsを実装日本語学習アプリを作ってみた！
Avatar for 蛋白（たんぱく・TANPACT） hypebeans
0
130
pnpm に provenance のダウングレード を検出する PR を出してみた
Avatar for mattsuu ryo_manba
1
160
Ktorで簡単AIアプリケーション
Avatar for Keita Tsukamoto tsukakei
0
120
PHPに関数型の魂を宿す〜PHP 8.5 で実現する堅牢なコードとは〜 #phpcon_hiroshima / phpcon-hiroshima-2025
Avatar for shogogg shogogg
1
340
他言語経験者が Golangci-lint を最初のコーディングメンターにした話 / How Golangci-lint Became My First Coding Mentor: A Story from a Polyglot Programmer
Avatar for uMa uma31
0
460
Migration to Signals, Resource API, and NgRx Signal Store
Avatar for Manfred Steyer manfredsteyer
PRO
0
130
なんでRustの環境構築してないのにRust製のツールが動くの？ / Why Do Rust-Based Tools Run Without a Rust Environment?
Avatar for TOMIKAWA Sotaro ssssota
14
47k
バッチ処理を「状態の記録」から「事実の記録」へ
Avatar for HideyukiKitao panda728
PRO
0
190
マイベストのシンプルなデータ基盤の話 - Googleスイートとのつき合い方 / mybest-simple-data-architecture-google-nized
Avatar for snhryt snhryt
0
100
Server Side Kotlin Meetup vol.16: 内部動作を理解して ハイパフォーマンスなサーバサイド Kotlin アプリケーションを書こう
Avatar for ternbusty ternbusty
3
260

Featured

See All Featured
Keith and Marios Guide to Fast Websites
Avatar for Keith Pitt keithpitt
411
23k
The Success of Rails: Ensuring Growth for the Next 100 Years
Avatar for Eileen M. Uchitelle eileencodes
46
7.7k
Music & Morning Musume
Avatar for Bryan Veloso bryan
46
6.9k
A designer walks into a library…
Avatar for Paul Jervis Heath pauljervisheath
209
24k
Design and Strategy: How to Deal with People Who Don’t "Get" Design
Avatar for Morgane Peng morganepeng
132
19k
KATA
Avatar for Megan Lloyd mclloyd
PRO
32
15k
Rails Girls Zürich Keynote
Avatar for Gregor Martynus gr2m
95
14k
Reflections from 52 weeks, 52 projects
Avatar for Jefferson Lam jeffersonlam
355
21k
The Illustrated Children's Guide to Kubernetes
Avatar for Chris Short chrisshort
49
51k
Faster Mobile Websites
Avatar for Dean Hume deanohume
310
31k
Navigating Team Friction
Avatar for Lara Hogan lara
190
15k
Typedesign – Prime Four
Avatar for Hannes Fritz hannesfritz
42
2.8k

Transcript

  1. None

  2. ESP Evolving Sustainable {data} Pipelines Anna Smith - @OMGannaks 29

    January 2015

  3. A bit about RTR A bit about our analytics Some

    evolution stuff A bit about Luigi The future bits

  4. A bit about RTR A bit about our analytics Some

    evolution stuff A bit about Luigi The future bits
  5. None

  6. 1. 2. 3.

  7. WHERE THE MAGIC HAPPENS

  8. A bit about RTR A bit about our analytics Some

    evolution stuff A bit about Luigi The future bits

  9. DEMAND website user interaction transactions SUPPLY warehouse inventory allocation

  10. DEMAND website user interaction transactions SUPPLY warehouse inventory allocation INTERACTION

    reservation calendar
  11. None
  12. None
  13. None

  14. A bit about RTR A bit about our analytics Some

    evolution stuff A bit about Luigi The future bits

  15. PHASE 0

  16. None

  17. YUP

  18. KEEP IT TOGETHER

  19. PHASE 1 primordial soup

  20. STABILITY

  21. PHASE 2 process

  22. orderwarehouse.job

  23. $ ./runjob.py orderwarehouse.job $ ./runjob.py orderwarehouse.job --show $ ./runjob.py orderwarehouse.job

    --only 2

  24. runjob.py

  25. ADAPTING ensuring data quality

  26. PHASE 3 exposing weaknesses

  27. A bit about RTR A bit about our analytics Some

    evolution stuff A bit about Luigi The future bits

  28. dependency manager

  29. None
  30. None

  31. A bit about RTR A bit about our analytics Some

    evolution stuff A bit about Luigi The future bits

  32. PHASE 4 ownership

  33. RELIABILITY

  34. COMMUNICATION

  35. THE FUTURE @OMGannaks [email protected]