Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Evolving Sustainable Data Pipelines

Avatar for Hakka Labs Hakka Labs
February 13, 2015
Programming
3.5k
0

Evolving Sustainable Data Pipelines

Full post here:

Avatar for Hakka Labs

Hakka Labs

February 13, 2015

More Decks by Hakka Labs

See All by Hakka Labs
New Workflows for Building Data Pipelines
Avatar for Hakka Labs hakka_labs
0
2.9k
Collaborative Topic Models for Users and Texts
Avatar for Hakka Labs hakka_labs
0
2.8k
Groupcache with Evan Owen
Avatar for Hakka Labs hakka_labs
2
5.4k
Testing Android at Spotify
Avatar for Hakka Labs hakka_labs
1
4.5k
It's Not a Bug, It's a Feature!
Avatar for Hakka Labs hakka_labs
0
3.2k
K-means Clustering to Understand Your Users
Avatar for Hakka Labs hakka_labs
0
2k
Building Amy: The Email-based Virtual Assistant by x.ai
Avatar for Hakka Labs hakka_labs
0
5k
Deep Learning and NLP Applications
Avatar for Hakka Labs hakka_labs
3
13k
Go and the Gophers
Avatar for Hakka Labs hakka_labs
2
11k

Other Decks in Programming

See All in Programming
【QA Test Talk Vol.8】AI-DLC による Whole Team Approach の加速
Avatar for PKSHA Technology（パークシャテクノロジー） pkshadeck
PRO
0
120
Google Apps Script で Ruby を動かす
Avatar for Shogo Kawahara kawahara
0
140
【やさしく解説 設計編・中級 #1】一つの車に、運転手は一人　～ある倉庫システムの事例から～
Avatar for HideyukiKitao panda728
PRO
0
210
「人を評価する AI」の 設計と実装 
Avatar for 奈良亮耶 ryoyanara
0
180
ドリフトを絶対に許さない（？）CDK運用 / CDK Ops with Zero Tolerance for Drifts (?)
Avatar for アキキー | Akihisa Ikeda akihisaikeda
1
180
JAWS-UG横浜 #102 AWSサ終供養LT会 成仏できない AWS サービスたち 〜本日、三体供養します〜
Avatar for maroon1st maroon1st
0
340
The Past, Present, and Future of Enterprise Java
Avatar for ivargrimstad ivargrimstad
0
550
壊れたパーサから始める関数型設計と構成的なパーサ #fp_matsuri
Avatar for ライガー raiga0310
2
450
【やさしく解説 設計編・中級 #6】良いアーキテクチャとは ～ 一本の登り道の、行き先 ～
Avatar for HideyukiKitao panda728
PRO
0
210
PostgreSQL 18で考えるUUID主キー
Avatar for Kazuhiro Seo kazuhiro1982
0
460
React本体のコードリーディング
Avatar for high-g high_g_engineer
1
140
仕様駆動開発の消費期限
Avatar for watany watany
17
7.4k

Featured

See All Featured
Neural Spatial Audio Processing for Sound Field Analysis and Control
Avatar for NII S. Koyama's Lab skoyamalab
0
400
Typedesign – Prime Four
Avatar for Hannes Fritz hannesfritz
42
3.1k
The browser strikes back
Avatar for Jono Alderson jonoalderson
0
1.5k
Visualizing Your Data: Incorporating Mongo into Loggly Infrastructure
Avatar for mongodb mongodb
49
10k
Design of three-dimensional binary manipulators for pick-and-place task avoiding obstacles (IECON2024)
Avatar for konakalab konakalab
0
530
Building Experiences: Design Systems, User Experience, and Full Site Editing
Avatar for Michelle Schulp Hunt marktimemedia
0
570
How to build an LLM SEO readiness audit: a practical framework
Avatar for Nick Samuel nmsamuel
1
850
Leo the Paperboy
Avatar for Maya Tellez mayatellez
8
2.1k
Documentation Writing (for coders)
Avatar for Carmen Chung carmenintech
77
5.4k
Done Done
Avatar for chrislema chrislema
186
16k
Fight the Zombie Pattern Library - RWD Summit 2016
Avatar for Marcelo Somers marcelosomers
234
17k
10 Git Anti Patterns You Should be Aware of
Avatar for Lemi Orhan Ergin lemiorhan
PRO
659
62k

Transcript

  1. None

  2. ESP Evolving Sustainable {data} Pipelines Anna Smith - @OMGannaks 29

    January 2015

  3. A bit about RTR A bit about our analytics Some

    evolution stuff A bit about Luigi The future bits

  4. A bit about RTR A bit about our analytics Some

    evolution stuff A bit about Luigi The future bits
  5. None

  6. 1. 2. 3.

  7. WHERE THE MAGIC HAPPENS

  8. A bit about RTR A bit about our analytics Some

    evolution stuff A bit about Luigi The future bits

  9. DEMAND website user interaction transactions SUPPLY warehouse inventory allocation

  10. DEMAND website user interaction transactions SUPPLY warehouse inventory allocation INTERACTION

    reservation calendar
  11. None
  12. None
  13. None

  14. A bit about RTR A bit about our analytics Some

    evolution stuff A bit about Luigi The future bits

  15. PHASE 0

  16. None

  17. YUP

  18. KEEP IT TOGETHER

  19. PHASE 1 primordial soup

  20. STABILITY

  21. PHASE 2 process

  22. orderwarehouse.job

  23. $ ./runjob.py orderwarehouse.job $ ./runjob.py orderwarehouse.job --show $ ./runjob.py orderwarehouse.job

    --only 2

  24. runjob.py

  25. ADAPTING ensuring data quality

  26. PHASE 3 exposing weaknesses

  27. A bit about RTR A bit about our analytics Some

    evolution stuff A bit about Luigi The future bits

  28. dependency manager

  29. None
  30. None

  31. A bit about RTR A bit about our analytics Some

    evolution stuff A bit about Luigi The future bits

  32. PHASE 4 ownership

  33. RELIABILITY

  34. COMMUNICATION

  35. THE FUTURE @OMGannaks [email protected]