Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Evolving Sustainable Data Pipelines

Avatar for Hakka Labs Hakka Labs
February 13, 2015
Programming
0
3.5k

Evolving Sustainable Data Pipelines

Full post here:

Avatar for Hakka Labs

Hakka Labs

February 13, 2015
Tweet

More Decks by Hakka Labs

See All by Hakka Labs
New Workflows for Building Data Pipelines
Avatar for Hakka Labs hakka_labs
0
2.9k
Collaborative Topic Models for Users and Texts
Avatar for Hakka Labs hakka_labs
0
2.8k
Groupcache with Evan Owen
Avatar for Hakka Labs hakka_labs
2
5.4k
Testing Android at Spotify
Avatar for Hakka Labs hakka_labs
1
4.5k
It's Not a Bug, It's a Feature!
Avatar for Hakka Labs hakka_labs
0
3.2k
K-means Clustering to Understand Your Users
Avatar for Hakka Labs hakka_labs
0
2k
Building Amy: The Email-based Virtual Assistant by x.ai
Avatar for Hakka Labs hakka_labs
0
5k
Deep Learning and NLP Applications
Avatar for Hakka Labs hakka_labs
3
13k
Go and the Gophers
Avatar for Hakka Labs hakka_labs
2
11k

Other Decks in Programming

See All in Programming
ソフトウェアテスト徹底指南書の紹介
Avatar for Hiroki Iseri goyoki
1
150
Deep Dive into Kotlin Flow
Avatar for Matsuda Jumpei jmatsu
1
300
Processing Gem ベースの、2D レトロゲームエンジンの開発
Avatar for tokujiros tokujiros
2
120
Testing Trophyは叫ばない
Avatar for toms toms74209200
0
840
Rancher と Terraform
Avatar for Ryoma Fujiwara fufuhu
2
240
「待たせ上手」なスケルトンスクリーン、 そのUXの裏側
Avatar for teamLab teamlab
PRO
0
470
FindyにおけるTakumi活用と脆弱性管理のこれから
Avatar for adachi.ryo rvirus0817
0
480
rage against annotate_predecessor
Avatar for Junichi Kobayashi junk0612
0
160
Zendeskのチケットを Amazon Bedrockで 解析した
Avatar for ryokosuge ryokosuge
3
290
もうちょっといいRubyプロファイラを作りたい (2025)
Avatar for osyoyu osyoyu
0
400
サーバーサイドのビルド時間87倍高速化
Avatar for PLAID Tech plaidtech
PRO
0
720
ユーザーも開発者も悩ませない TV アプリ開発 ~Compose の内部実装から学ぶフォーカス制御~
Avatar for Daichi Takeya taked137
0
140

Featured

See All Featured
Scaling GitHub
Avatar for Zach Holman holman
463
140k
4 Signs Your Business is Dying
Avatar for Josh Pigford shpigford
184
22k
Responsive Adventures: Dirty Tricks From The Dark Corners of Front-End
Avatar for Vitaly Friedman smashingmag
252
21k
XXLCSS - How to scale CSS and keep your sanity
Avatar for Zaharenia Atzitzikaki sugarenia
248
1.3M
Speed Design
Avatar for Sergey Chernyshev sergeychernyshev
32
1.1k
Code Reviewing Like a Champion
Avatar for maltzj maltzj
525
40k
GraphQLとの向き合い方2022年版
Avatar for Yosuke Kurami quramy
49
14k
Build The Right Thing And Hit Your Dates
Avatar for Maggie C. maggiecrowley
37
2.8k
jQuery: Nuts, Bolts and Bling
Avatar for Doug Neiner dougneiner
64
7.9k
Let's Do A Bunch of Simple Stuff to Make Websites Faster
Avatar for Chris Coyier chriscoyier
507
140k
Java REST API Framework Comparison - PWX 2021
Avatar for Matt Raible mraible
33
8.8k
VelocityConf: Rendering Performance Case Studies
Avatar for Addy Osmani addyosmani
332
24k

Transcript

  1. None

  2. ESP Evolving Sustainable {data} Pipelines Anna Smith - @OMGannaks 29

    January 2015

  3. A bit about RTR A bit about our analytics Some

    evolution stuff A bit about Luigi The future bits

  4. A bit about RTR A bit about our analytics Some

    evolution stuff A bit about Luigi The future bits
  5. None

  6. 1. 2. 3.

  7. WHERE THE MAGIC HAPPENS

  8. A bit about RTR A bit about our analytics Some

    evolution stuff A bit about Luigi The future bits

  9. DEMAND website user interaction transactions SUPPLY warehouse inventory allocation

  10. DEMAND website user interaction transactions SUPPLY warehouse inventory allocation INTERACTION

    reservation calendar
  11. None
  12. None
  13. None

  14. A bit about RTR A bit about our analytics Some

    evolution stuff A bit about Luigi The future bits

  15. PHASE 0

  16. None

  17. YUP

  18. KEEP IT TOGETHER

  19. PHASE 1 primordial soup

  20. STABILITY

  21. PHASE 2 process

  22. orderwarehouse.job

  23. $ ./runjob.py orderwarehouse.job $ ./runjob.py orderwarehouse.job --show $ ./runjob.py orderwarehouse.job

    --only 2

  24. runjob.py

  25. ADAPTING ensuring data quality

  26. PHASE 3 exposing weaknesses

  27. A bit about RTR A bit about our analytics Some

    evolution stuff A bit about Luigi The future bits

  28. dependency manager

  29. None
  30. None

  31. A bit about RTR A bit about our analytics Some

    evolution stuff A bit about Luigi The future bits

  32. PHASE 4 ownership

  33. RELIABILITY

  34. COMMUNICATION

  35. THE FUTURE @OMGannaks [email protected]